Oct 31 04:01:35.662342 ip-10-0-134-155 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Oct 31 04:01:35.662359 ip-10-0-134-155 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Oct 31 04:01:35.662369 ip-10-0-134-155 systemd[1]: kubelet.service: Failed with result 'resources'. Oct 31 04:01:35.662725 ip-10-0-134-155 systemd[1]: Failed to start Kubernetes Kubelet. Oct 31 04:01:45.322979 ip-10-0-134-155 systemd[1]: kubelet.service: Failed to schedule restart job: Unit openvswitch.service not found. Oct 31 04:01:45.322991 ip-10-0-134-155 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 665524b70d7c415e90cccb508656d751 -- Oct 31 04:03:31.680580 ip-10-0-134-155 systemd[1]: Starting Kubernetes Kubelet... Oct 31 04:03:32.047704 ip-10-0-134-155 kubenswrapper[2577]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 04:03:32.047704 ip-10-0-134-155 kubenswrapper[2577]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 31 04:03:32.047704 ip-10-0-134-155 kubenswrapper[2577]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 04:03:32.047704 ip-10-0-134-155 kubenswrapper[2577]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 04:03:32.052290 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.052222 2577 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 04:03:32.052290 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.052289 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 04:03:32.052290 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.052292 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 04:03:32.052290 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.052296 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 04:03:32.052435 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.052299 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 04:03:32.052435 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.052302 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 04:03:32.052435 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.052304 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 04:03:32.052435 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.052307 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 04:03:32.052435 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.052309 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 04:03:32.052435 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.052320 2577 feature_gate.go:386] unrecognized feature gate: Example Oct 31 04:03:32.052435 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.052323 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 04:03:32.052435 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.052340 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 04:03:32.052435 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.052345 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 04:03:32.052435 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.052349 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 04:03:32.052435 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.052355 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 04:03:32.052435 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.052359 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 04:03:32.052435 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.052362 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 04:03:32.052435 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.052365 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 04:03:32.052435 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.052367 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 04:03:32.052435 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.052370 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 04:03:32.052435 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.052373 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 04:03:32.052435 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.052375 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 04:03:32.052435 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.052378 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 04:03:32.052867 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.052380 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 04:03:32.052867 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.052383 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 04:03:32.052867 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.052385 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 04:03:32.052867 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.052388 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 04:03:32.052867 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.052390 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 04:03:32.052867 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.052393 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 04:03:32.052867 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.052395 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 04:03:32.052867 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.052399 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 04:03:32.052867 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.052404 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 04:03:32.052867 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.052407 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 04:03:32.052867 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.052410 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 04:03:32.052867 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.052412 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 04:03:32.052867 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.052415 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 04:03:32.052867 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.052417 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 04:03:32.052867 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.052421 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 04:03:32.052867 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.052423 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 04:03:32.052867 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.052427 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 04:03:32.052867 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.052429 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 04:03:32.052867 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.052432 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 04:03:32.053316 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.052434 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 04:03:32.053316 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.052437 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 04:03:32.053316 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.052440 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 04:03:32.053316 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.052443 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 04:03:32.053316 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.052445 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 04:03:32.053316 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.052448 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 04:03:32.053316 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.052450 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 04:03:32.053316 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.052453 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 04:03:32.053316 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.052455 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 04:03:32.053316 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.052458 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 04:03:32.053316 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.052460 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 04:03:32.053316 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.052463 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 04:03:32.053316 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.052465 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 04:03:32.053316 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.052468 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 04:03:32.053316 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.052470 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 04:03:32.053316 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.052473 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 04:03:32.053316 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.052476 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 04:03:32.053316 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.052478 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 04:03:32.053316 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.052481 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 04:03:32.053316 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.052484 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 04:03:32.053792 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.052486 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 04:03:32.053792 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.052489 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 04:03:32.053792 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.052491 2577 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 04:03:32.053792 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.052493 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 04:03:32.053792 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.052496 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 04:03:32.053792 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.052498 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 04:03:32.053792 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.052501 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 04:03:32.053792 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.052506 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 04:03:32.053792 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.052508 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 04:03:32.053792 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.052511 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 04:03:32.053792 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.052514 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 04:03:32.053792 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.052517 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 04:03:32.053792 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.052521 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 04:03:32.053792 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.052523 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 04:03:32.053792 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.052526 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 04:03:32.053792 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.052528 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 04:03:32.053792 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.052531 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 04:03:32.053792 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.052534 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 04:03:32.053792 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.052536 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 04:03:32.054270 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.052539 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 04:03:32.054270 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.052542 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 04:03:32.054270 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.052545 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 04:03:32.054270 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.052547 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 04:03:32.054270 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.052550 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 04:03:32.054270 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.052552 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 04:03:32.054270 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.052555 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 04:03:32.054270 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.052557 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 04:03:32.054270 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.052560 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 04:03:32.054270 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.052562 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 04:03:32.054270 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.052566 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 04:03:32.054270 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.052570 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 04:03:32.054270 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.052573 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 04:03:32.054270 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.052576 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 04:03:32.054270 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.052578 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 04:03:32.054270 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.052581 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 04:03:32.054270 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.052583 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 04:03:32.054270 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.052585 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 04:03:32.054270 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.052589 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 04:03:32.054711 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.052592 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 04:03:32.054711 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.052595 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 04:03:32.054711 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.052597 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 04:03:32.054711 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.052600 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 04:03:32.054711 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.052602 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 04:03:32.054711 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.052605 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 04:03:32.054711 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.053448 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 04:03:32.054711 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.053455 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 04:03:32.054711 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.053458 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 04:03:32.054711 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.053461 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 04:03:32.054711 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.053464 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 04:03:32.054711 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.053467 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 04:03:32.054711 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.053469 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 04:03:32.054711 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.053472 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 04:03:32.054711 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.053475 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 04:03:32.054711 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.053477 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 04:03:32.054711 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.053480 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 04:03:32.054711 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.053482 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 04:03:32.054711 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.053485 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 04:03:32.054711 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.053487 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 04:03:32.055170 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.053490 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 04:03:32.055170 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.053492 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 04:03:32.055170 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.053495 2577 feature_gate.go:386] unrecognized feature gate: Example Oct 31 04:03:32.055170 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.053497 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 04:03:32.055170 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.053500 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 04:03:32.055170 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.053502 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 04:03:32.055170 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.053505 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 04:03:32.055170 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.053509 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 04:03:32.055170 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.053512 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 04:03:32.055170 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.053515 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 04:03:32.055170 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.053518 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 04:03:32.055170 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.053520 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 04:03:32.055170 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.053523 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 04:03:32.055170 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.053526 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 04:03:32.055170 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.053528 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 04:03:32.055170 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.053531 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 04:03:32.055170 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.053534 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 04:03:32.055170 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.053536 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 04:03:32.055170 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.053539 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 04:03:32.055170 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.053542 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 04:03:32.055725 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.053544 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 04:03:32.055725 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.053547 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 04:03:32.055725 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.053549 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 04:03:32.055725 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.053552 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 04:03:32.055725 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.053554 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 04:03:32.055725 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.053557 2577 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 04:03:32.055725 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.053559 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 04:03:32.055725 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.053561 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 04:03:32.055725 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.053564 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 04:03:32.055725 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.053567 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 04:03:32.055725 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.053569 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 04:03:32.055725 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.053571 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 04:03:32.055725 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.053574 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 04:03:32.055725 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.053576 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 04:03:32.055725 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.053578 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 04:03:32.055725 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.053580 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 04:03:32.055725 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.053584 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 04:03:32.055725 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.053587 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 04:03:32.055725 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.053597 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 04:03:32.056155 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.053600 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 04:03:32.056155 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.053602 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 04:03:32.056155 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.053605 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 04:03:32.056155 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.053607 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 04:03:32.056155 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.053610 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 04:03:32.056155 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.053613 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 04:03:32.056155 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.053616 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 04:03:32.056155 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.053618 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 04:03:32.056155 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.053621 2577 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 04:03:32.056155 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.053623 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 04:03:32.056155 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.053626 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 04:03:32.056155 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.053628 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 04:03:32.056155 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.053632 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 04:03:32.056155 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.053634 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 04:03:32.056155 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.053637 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 04:03:32.056155 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.053640 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 04:03:32.056155 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.053642 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 04:03:32.056155 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.053644 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 04:03:32.056155 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.053647 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 04:03:32.056155 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.053649 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 04:03:32.056660 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.053652 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 04:03:32.056660 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.053654 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 04:03:32.056660 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.053656 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 04:03:32.056660 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.053659 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 04:03:32.056660 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.053661 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 04:03:32.056660 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.053664 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 04:03:32.056660 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.053666 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 04:03:32.056660 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.053668 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 04:03:32.056660 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.053671 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 04:03:32.056660 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.053673 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 04:03:32.056660 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.053679 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 04:03:32.056660 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.053681 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 04:03:32.056660 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.053684 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 04:03:32.056660 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.053688 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 04:03:32.056660 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.053692 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 04:03:32.056660 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.053695 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 04:03:32.056660 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.053698 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 04:03:32.056660 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.053701 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 04:03:32.056660 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.053704 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 04:03:32.057091 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.053707 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 04:03:32.057091 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.053710 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 04:03:32.057091 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.053713 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 04:03:32.057091 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.053715 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 04:03:32.057091 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.053718 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 04:03:32.057091 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.053720 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 04:03:32.057091 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.053723 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 04:03:32.057091 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.053726 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 04:03:32.057091 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.053728 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 04:03:32.057091 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.053731 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 04:03:32.057091 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.053733 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 04:03:32.057091 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.053736 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 04:03:32.057091 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.053739 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 04:03:32.057091 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.053741 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 04:03:32.057091 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.054201 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 04:03:32.057091 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.054205 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 04:03:32.057091 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.054208 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 04:03:32.057091 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.054211 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 04:03:32.057091 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.054213 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 04:03:32.057540 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.054216 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 04:03:32.057540 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.054218 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 04:03:32.057540 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.054221 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 04:03:32.057540 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.054224 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 04:03:32.057540 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.054227 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 04:03:32.057540 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.054230 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 04:03:32.057540 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.054232 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 04:03:32.057540 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.054235 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 04:03:32.057540 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.054237 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 04:03:32.057540 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.054240 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 04:03:32.057540 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.054242 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 04:03:32.057540 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.054245 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 04:03:32.057540 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.054248 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 04:03:32.057540 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.054250 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 04:03:32.057540 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.054253 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 04:03:32.057540 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.054255 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 04:03:32.057540 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.054258 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 04:03:32.057540 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.054260 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 04:03:32.057540 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.054263 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 04:03:32.058024 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.054266 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 04:03:32.058024 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.054268 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 04:03:32.058024 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.054271 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 04:03:32.058024 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.054274 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 04:03:32.058024 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.054276 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 04:03:32.058024 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.054279 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 04:03:32.058024 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.054281 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 04:03:32.058024 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.054284 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 04:03:32.058024 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.054286 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 04:03:32.058024 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.054289 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 04:03:32.058024 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.054291 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 04:03:32.058024 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.054294 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 04:03:32.058024 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.054296 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 04:03:32.058024 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.054298 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 04:03:32.058024 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.054301 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 04:03:32.058024 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.054303 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 04:03:32.058024 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.054306 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 04:03:32.058024 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.054309 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 04:03:32.058024 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.054312 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 04:03:32.058513 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.054314 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 04:03:32.058513 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.054317 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 04:03:32.058513 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.054319 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 04:03:32.058513 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.054321 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 04:03:32.058513 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.054337 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 04:03:32.058513 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.054340 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 04:03:32.058513 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.054343 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 04:03:32.058513 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.054346 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 04:03:32.058513 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.054349 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 04:03:32.058513 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.054351 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 04:03:32.058513 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.054353 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 04:03:32.058513 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.054356 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 04:03:32.058513 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.054358 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 04:03:32.058513 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.054361 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 04:03:32.058513 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.054364 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 04:03:32.058513 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.054366 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 04:03:32.058513 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.054369 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 04:03:32.058513 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.054371 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 04:03:32.058513 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.054374 2577 feature_gate.go:386] unrecognized feature gate: Example Oct 31 04:03:32.058513 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.054376 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 04:03:32.058978 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.054380 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 04:03:32.058978 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.054384 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 04:03:32.058978 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.054388 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 04:03:32.058978 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.054390 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 04:03:32.058978 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.054393 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 04:03:32.058978 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.054396 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 04:03:32.058978 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.054398 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 04:03:32.058978 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.054401 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 04:03:32.058978 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.054403 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 04:03:32.058978 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.054406 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 04:03:32.058978 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.054408 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 04:03:32.058978 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.054411 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 04:03:32.058978 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.054413 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 04:03:32.058978 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.054415 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 04:03:32.058978 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.054419 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 04:03:32.058978 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.054421 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 04:03:32.058978 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.054423 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 04:03:32.058978 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.054426 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 04:03:32.058978 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.054428 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 04:03:32.059427 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.054431 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 04:03:32.059427 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.054433 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 04:03:32.059427 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.054436 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 04:03:32.059427 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.054438 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 04:03:32.059427 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.054441 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 04:03:32.059427 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.054443 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 04:03:32.059427 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.054445 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 04:03:32.059427 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.054447 2577 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 04:03:32.059427 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.054450 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 04:03:32.059427 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.054453 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 04:03:32.059427 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.054455 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 04:03:32.059427 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.054457 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 04:03:32.059427 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.054460 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 04:03:32.059427 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.054462 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 04:03:32.059427 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.054465 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 04:03:32.059427 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.054467 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 04:03:32.059427 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.054470 2577 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 04:03:32.059427 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.054472 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 04:03:32.059427 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.054476 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 04:03:32.059427 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.054480 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 04:03:32.059871 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.054483 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 04:03:32.059871 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.054486 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 04:03:32.059871 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.054488 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 04:03:32.059871 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.054491 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 04:03:32.059871 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.055478 2577 flags.go:64] FLAG: --address="0.0.0.0" Oct 31 04:03:32.059871 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.055487 2577 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 31 04:03:32.059871 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.055493 2577 flags.go:64] FLAG: --anonymous-auth="true" Oct 31 04:03:32.059871 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.055497 2577 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 31 04:03:32.059871 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.055501 2577 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 31 04:03:32.059871 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.055505 2577 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 31 04:03:32.059871 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.055512 2577 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 31 04:03:32.059871 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.055516 2577 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 31 04:03:32.059871 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.055519 2577 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 31 04:03:32.059871 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.055522 2577 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 31 04:03:32.059871 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.055525 2577 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 31 04:03:32.059871 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.055529 2577 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 31 04:03:32.059871 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.055532 2577 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 31 04:03:32.059871 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.055535 2577 flags.go:64] FLAG: --cgroup-root="" Oct 31 04:03:32.059871 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.055537 2577 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 31 04:03:32.059871 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.055540 2577 flags.go:64] FLAG: --client-ca-file="" Oct 31 04:03:32.059871 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.055543 2577 flags.go:64] FLAG: --cloud-provider="external" Oct 31 04:03:32.059871 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.055547 2577 flags.go:64] FLAG: --cluster-dns="[]" Oct 31 04:03:32.059871 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.055551 2577 flags.go:64] FLAG: --cluster-domain="" Oct 31 04:03:32.060414 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.055554 2577 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 31 04:03:32.060414 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.055557 2577 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 31 04:03:32.060414 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.055560 2577 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 31 04:03:32.060414 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.055565 2577 flags.go:64] FLAG: --container-log-max-files="5" Oct 31 04:03:32.060414 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.055569 2577 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 31 04:03:32.060414 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.055572 2577 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 31 04:03:32.060414 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.055575 2577 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 31 04:03:32.060414 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.055578 2577 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 31 04:03:32.060414 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.055581 2577 flags.go:64] FLAG: --contention-profiling="false" Oct 31 04:03:32.060414 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.055584 2577 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 31 04:03:32.060414 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.055587 2577 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 31 04:03:32.060414 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.055590 2577 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 31 04:03:32.060414 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.055593 2577 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 31 04:03:32.060414 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.055597 2577 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 31 04:03:32.060414 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.055600 2577 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 31 04:03:32.060414 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.055603 2577 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 31 04:03:32.060414 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.055606 2577 flags.go:64] FLAG: --enable-load-reader="false" Oct 31 04:03:32.060414 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.055609 2577 flags.go:64] FLAG: --enable-server="true" Oct 31 04:03:32.060414 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.055614 2577 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 31 04:03:32.060414 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.055618 2577 flags.go:64] FLAG: --event-burst="100" Oct 31 04:03:32.060414 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.055621 2577 flags.go:64] FLAG: --event-qps="50" Oct 31 04:03:32.060414 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.055624 2577 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 31 04:03:32.060414 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.055627 2577 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 31 04:03:32.060414 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.055630 2577 flags.go:64] FLAG: --eviction-hard="" Oct 31 04:03:32.061011 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.055634 2577 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 31 04:03:32.061011 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.055637 2577 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 31 04:03:32.061011 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.055640 2577 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 31 04:03:32.061011 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.055643 2577 flags.go:64] FLAG: --eviction-soft="" Oct 31 04:03:32.061011 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.055646 2577 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 31 04:03:32.061011 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.055649 2577 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 31 04:03:32.061011 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.055652 2577 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 31 04:03:32.061011 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.055655 2577 flags.go:64] FLAG: --experimental-mounter-path="" Oct 31 04:03:32.061011 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.055658 2577 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 31 04:03:32.061011 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.055661 2577 flags.go:64] FLAG: --fail-swap-on="true" Oct 31 04:03:32.061011 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.055664 2577 flags.go:64] FLAG: --feature-gates="" Oct 31 04:03:32.061011 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.055669 2577 flags.go:64] FLAG: --file-check-frequency="20s" Oct 31 04:03:32.061011 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.055672 2577 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 31 04:03:32.061011 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.055675 2577 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 31 04:03:32.061011 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.055678 2577 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 31 04:03:32.061011 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.055682 2577 flags.go:64] FLAG: --healthz-port="10248" Oct 31 04:03:32.061011 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.055685 2577 flags.go:64] FLAG: --help="false" Oct 31 04:03:32.061011 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.055688 2577 flags.go:64] FLAG: --hostname-override="ip-10-0-134-155.ec2.internal" Oct 31 04:03:32.061011 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.055691 2577 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 31 04:03:32.061011 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.055694 2577 flags.go:64] FLAG: --http-check-frequency="20s" Oct 31 04:03:32.061011 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.055697 2577 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 31 04:03:32.061011 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.055700 2577 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 31 04:03:32.061011 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.055703 2577 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 31 04:03:32.061551 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.055706 2577 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 31 04:03:32.061551 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.055709 2577 flags.go:64] FLAG: --image-service-endpoint="" Oct 31 04:03:32.061551 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.055712 2577 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 31 04:03:32.061551 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.055716 2577 flags.go:64] FLAG: --kube-api-burst="100" Oct 31 04:03:32.061551 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.055719 2577 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 31 04:03:32.061551 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.055722 2577 flags.go:64] FLAG: --kube-api-qps="50" Oct 31 04:03:32.061551 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.055725 2577 flags.go:64] FLAG: --kube-reserved="" Oct 31 04:03:32.061551 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.055729 2577 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 31 04:03:32.061551 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.055732 2577 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 31 04:03:32.061551 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.055735 2577 flags.go:64] FLAG: --kubelet-cgroups="" Oct 31 04:03:32.061551 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.055738 2577 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 31 04:03:32.061551 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.055741 2577 flags.go:64] FLAG: --lock-file="" Oct 31 04:03:32.061551 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.055744 2577 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 31 04:03:32.061551 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.055747 2577 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 31 04:03:32.061551 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.055750 2577 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 31 04:03:32.061551 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.055755 2577 flags.go:64] FLAG: --log-json-split-stream="false" Oct 31 04:03:32.061551 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.055758 2577 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 31 04:03:32.061551 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.055761 2577 flags.go:64] FLAG: --log-text-split-stream="false" Oct 31 04:03:32.061551 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.055764 2577 flags.go:64] FLAG: --logging-format="text" Oct 31 04:03:32.061551 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.055767 2577 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 31 04:03:32.061551 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.055772 2577 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 31 04:03:32.061551 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.055775 2577 flags.go:64] FLAG: --manifest-url="" Oct 31 04:03:32.061551 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.055777 2577 flags.go:64] FLAG: --manifest-url-header="" Oct 31 04:03:32.061551 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.055781 2577 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 31 04:03:32.061551 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.055784 2577 flags.go:64] FLAG: --max-open-files="1000000" Oct 31 04:03:32.062155 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.055788 2577 flags.go:64] FLAG: --max-pods="110" Oct 31 04:03:32.062155 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.055791 2577 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 31 04:03:32.062155 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.055794 2577 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 31 04:03:32.062155 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.055797 2577 flags.go:64] FLAG: --memory-manager-policy="None" Oct 31 04:03:32.062155 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.055800 2577 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 31 04:03:32.062155 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.055803 2577 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 31 04:03:32.062155 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.055806 2577 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 31 04:03:32.062155 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.055809 2577 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 31 04:03:32.062155 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.055815 2577 flags.go:64] FLAG: --node-status-max-images="50" Oct 31 04:03:32.062155 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.055818 2577 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 31 04:03:32.062155 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.055823 2577 flags.go:64] FLAG: --oom-score-adj="-999" Oct 31 04:03:32.062155 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.055826 2577 flags.go:64] FLAG: --pod-cidr="" Oct 31 04:03:32.062155 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.055828 2577 flags.go:64] FLAG: --pod-manifest-path="" Oct 31 04:03:32.062155 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.055831 2577 flags.go:64] FLAG: --pod-max-pids="-1" Oct 31 04:03:32.062155 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.055834 2577 flags.go:64] FLAG: --pods-per-core="0" Oct 31 04:03:32.062155 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.055837 2577 flags.go:64] FLAG: --port="10250" Oct 31 04:03:32.062155 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.055840 2577 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 31 04:03:32.062155 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.055842 2577 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-06236bc10ca20d74b" Oct 31 04:03:32.062155 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.055845 2577 flags.go:64] FLAG: --qos-reserved="" Oct 31 04:03:32.062155 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.055848 2577 flags.go:64] FLAG: --read-only-port="10255" Oct 31 04:03:32.062155 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.055851 2577 flags.go:64] FLAG: --register-node="true" Oct 31 04:03:32.062155 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.055854 2577 flags.go:64] FLAG: --register-with-taints="" Oct 31 04:03:32.062155 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.055858 2577 flags.go:64] FLAG: --registry-burst="10" Oct 31 04:03:32.062155 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.055860 2577 flags.go:64] FLAG: --registry-qps="5" Oct 31 04:03:32.062155 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.055863 2577 flags.go:64] FLAG: --reserved-cpus="" Oct 31 04:03:32.062155 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.055867 2577 flags.go:64] FLAG: --reserved-memory="" Oct 31 04:03:32.062763 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.055870 2577 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 31 04:03:32.062763 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.055874 2577 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 31 04:03:32.062763 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.055877 2577 flags.go:64] FLAG: --rotate-certificates="false" Oct 31 04:03:32.062763 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.055880 2577 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 31 04:03:32.062763 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.055883 2577 flags.go:64] FLAG: --runonce="false" Oct 31 04:03:32.062763 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.055886 2577 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 31 04:03:32.062763 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.055889 2577 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 31 04:03:32.062763 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.055892 2577 flags.go:64] FLAG: --seccomp-default="false" Oct 31 04:03:32.062763 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.055895 2577 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 31 04:03:32.062763 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.055898 2577 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 31 04:03:32.062763 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.055901 2577 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 31 04:03:32.062763 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.055904 2577 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 31 04:03:32.062763 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.055908 2577 flags.go:64] FLAG: --storage-driver-password="root" Oct 31 04:03:32.062763 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.055910 2577 flags.go:64] FLAG: --storage-driver-secure="false" Oct 31 04:03:32.062763 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.055913 2577 flags.go:64] FLAG: --storage-driver-table="stats" Oct 31 04:03:32.062763 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.055916 2577 flags.go:64] FLAG: --storage-driver-user="root" Oct 31 04:03:32.062763 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.055920 2577 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 31 04:03:32.062763 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.055923 2577 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 31 04:03:32.062763 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.055926 2577 flags.go:64] FLAG: --system-cgroups="" Oct 31 04:03:32.062763 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.055929 2577 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 31 04:03:32.062763 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.055934 2577 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 31 04:03:32.062763 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.055937 2577 flags.go:64] FLAG: --tls-cert-file="" Oct 31 04:03:32.062763 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.055940 2577 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 31 04:03:32.062763 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.055943 2577 flags.go:64] FLAG: --tls-min-version="" Oct 31 04:03:32.062763 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.055946 2577 flags.go:64] FLAG: --tls-private-key-file="" Oct 31 04:03:32.063385 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.055948 2577 flags.go:64] FLAG: --topology-manager-policy="none" Oct 31 04:03:32.063385 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.055951 2577 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 31 04:03:32.063385 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.055954 2577 flags.go:64] FLAG: --topology-manager-scope="container" Oct 31 04:03:32.063385 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.055957 2577 flags.go:64] FLAG: --v="2" Oct 31 04:03:32.063385 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.055961 2577 flags.go:64] FLAG: --version="false" Oct 31 04:03:32.063385 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.055965 2577 flags.go:64] FLAG: --vmodule="" Oct 31 04:03:32.063385 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.055969 2577 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 31 04:03:32.063385 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.055973 2577 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 31 04:03:32.063385 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056107 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 04:03:32.063385 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056111 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 04:03:32.063385 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056114 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 04:03:32.063385 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056116 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 04:03:32.063385 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056119 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 04:03:32.063385 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056121 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 04:03:32.063385 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056125 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 04:03:32.063385 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056127 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 04:03:32.063385 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056130 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 04:03:32.063385 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056132 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 04:03:32.063385 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056134 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 04:03:32.063385 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056137 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 04:03:32.063385 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056139 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 04:03:32.064120 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056142 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 04:03:32.064120 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056144 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 04:03:32.064120 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056148 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 04:03:32.064120 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056151 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 04:03:32.064120 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056155 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 04:03:32.064120 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056159 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 04:03:32.064120 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056162 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 04:03:32.064120 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056165 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 04:03:32.064120 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056168 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 04:03:32.064120 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056170 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 04:03:32.064120 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056173 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 04:03:32.064120 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056177 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 04:03:32.064120 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056180 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 04:03:32.064120 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056184 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 04:03:32.064120 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056188 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 04:03:32.064120 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056192 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 04:03:32.064120 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056195 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 04:03:32.064120 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056199 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 04:03:32.064565 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056203 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 04:03:32.064565 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056210 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 04:03:32.064565 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056215 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 04:03:32.064565 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056218 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 04:03:32.064565 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056220 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 04:03:32.064565 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056222 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 04:03:32.064565 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056225 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 04:03:32.064565 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056228 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 04:03:32.064565 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056230 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 04:03:32.064565 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056233 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 04:03:32.064565 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056235 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 04:03:32.064565 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056238 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 04:03:32.064565 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056240 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 04:03:32.064565 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056243 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 04:03:32.064565 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056246 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 04:03:32.064565 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056248 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 04:03:32.064565 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056251 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 04:03:32.064565 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056254 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 04:03:32.064565 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056256 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 04:03:32.064565 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056259 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 04:03:32.065021 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056262 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 04:03:32.065021 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056264 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 04:03:32.065021 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056266 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 04:03:32.065021 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056269 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 04:03:32.065021 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056271 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 04:03:32.065021 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056274 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 04:03:32.065021 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056277 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 04:03:32.065021 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056281 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 04:03:32.065021 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056286 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 04:03:32.065021 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056290 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 04:03:32.065021 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056292 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 04:03:32.065021 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056295 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 04:03:32.065021 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056297 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 04:03:32.065021 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056301 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 04:03:32.065021 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056304 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 04:03:32.065021 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056307 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 04:03:32.065021 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056309 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 04:03:32.065021 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056312 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 04:03:32.065021 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056314 2577 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 04:03:32.065021 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056317 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 04:03:32.065540 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056319 2577 feature_gate.go:386] unrecognized feature gate: Example Oct 31 04:03:32.065540 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056321 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 04:03:32.065540 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056324 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 04:03:32.065540 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056339 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 04:03:32.065540 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056343 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 04:03:32.065540 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056347 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 04:03:32.065540 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056352 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 04:03:32.065540 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056354 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 04:03:32.065540 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056357 2577 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 04:03:32.065540 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056359 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 04:03:32.065540 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056362 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 04:03:32.065540 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056365 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 04:03:32.065540 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056367 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 04:03:32.065540 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056370 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 04:03:32.065540 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056372 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 04:03:32.065540 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056374 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 04:03:32.065540 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056377 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 04:03:32.065540 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056379 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 04:03:32.065540 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056382 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 04:03:32.065540 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056384 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 04:03:32.066062 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056387 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 04:03:32.066062 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056389 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 04:03:32.066062 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056392 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 04:03:32.066062 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056394 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 04:03:32.066062 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056396 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 04:03:32.066062 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056400 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 04:03:32.066062 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056403 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 04:03:32.066062 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056406 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 04:03:32.066062 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056409 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 04:03:32.066062 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056411 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 04:03:32.066062 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056415 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 04:03:32.066062 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056419 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 04:03:32.066062 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056441 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 04:03:32.066062 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056466 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 04:03:32.066062 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056471 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 04:03:32.066062 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.056512 2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 04:03:32.066455 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056556 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 04:03:32.066455 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056560 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 04:03:32.066455 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056563 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 04:03:32.066455 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056566 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 04:03:32.066455 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056569 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 04:03:32.066455 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056572 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 04:03:32.066455 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056574 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 04:03:32.066455 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056577 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 04:03:32.066455 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056580 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 04:03:32.066455 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056582 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 04:03:32.066455 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056585 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 04:03:32.066455 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056588 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 04:03:32.066455 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056590 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 04:03:32.066455 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056592 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 04:03:32.066455 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056595 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 04:03:32.066455 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056597 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 04:03:32.066455 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056600 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 04:03:32.066455 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056602 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 04:03:32.066455 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056605 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 04:03:32.066869 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056608 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 04:03:32.066869 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056610 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 04:03:32.066869 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056612 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 04:03:32.066869 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056615 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 04:03:32.066869 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056618 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 04:03:32.066869 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056621 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 04:03:32.066869 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056623 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 04:03:32.066869 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056625 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 04:03:32.066869 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056628 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 04:03:32.066869 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056630 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 04:03:32.066869 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056633 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 04:03:32.066869 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056637 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 04:03:32.066869 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056640 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 04:03:32.066869 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056642 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 04:03:32.066869 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056647 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 04:03:32.066869 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056651 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 04:03:32.066869 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056654 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 04:03:32.066869 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056657 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 04:03:32.066869 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056660 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 04:03:32.067300 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056663 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 04:03:32.067300 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056665 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 04:03:32.067300 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056668 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 04:03:32.067300 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056672 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 04:03:32.067300 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056676 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 04:03:32.067300 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056679 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 04:03:32.067300 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056682 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 04:03:32.067300 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056684 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 04:03:32.067300 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056687 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 04:03:32.067300 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056689 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 04:03:32.067300 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056692 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 04:03:32.067300 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056694 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 04:03:32.067300 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056697 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 04:03:32.067300 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056700 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 04:03:32.067300 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056702 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 04:03:32.067300 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056705 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 04:03:32.067300 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056707 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 04:03:32.067300 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056709 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 04:03:32.067300 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056714 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 04:03:32.067750 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056716 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 04:03:32.067750 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056719 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 04:03:32.067750 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056721 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 04:03:32.067750 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056723 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 04:03:32.067750 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056727 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 04:03:32.067750 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056729 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 04:03:32.067750 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056733 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 04:03:32.067750 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056735 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 04:03:32.067750 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056738 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 04:03:32.067750 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056741 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 04:03:32.067750 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056743 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 04:03:32.067750 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056746 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 04:03:32.067750 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056748 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 04:03:32.067750 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056751 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 04:03:32.067750 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056753 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 04:03:32.067750 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056756 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 04:03:32.067750 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056759 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 04:03:32.067750 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056761 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 04:03:32.067750 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056764 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 04:03:32.068183 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056766 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 04:03:32.068183 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056769 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 04:03:32.068183 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056771 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 04:03:32.068183 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056774 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 04:03:32.068183 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056776 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 04:03:32.068183 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056779 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 04:03:32.068183 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056781 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 04:03:32.068183 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056785 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 04:03:32.068183 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056788 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 04:03:32.068183 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056790 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 04:03:32.068183 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056792 2577 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 04:03:32.068183 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056795 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 04:03:32.068183 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056797 2577 feature_gate.go:386] unrecognized feature gate: Example Oct 31 04:03:32.068183 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056800 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 04:03:32.068183 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056802 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 04:03:32.068183 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056805 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 04:03:32.068183 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056808 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 04:03:32.068183 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056811 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 04:03:32.068183 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056817 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 04:03:32.068183 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056819 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 04:03:32.068659 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056822 2577 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 04:03:32.068659 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056825 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 04:03:32.068659 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056827 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 04:03:32.068659 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056830 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 04:03:32.068659 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056832 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 04:03:32.068659 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056835 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 04:03:32.068659 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056837 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 04:03:32.068659 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056840 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 04:03:32.068659 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056842 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 04:03:32.068659 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.056845 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 04:03:32.068659 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.065013 2577 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 31 04:03:32.068659 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.065029 2577 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 31 04:03:32.068659 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.065808 2577 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 31 04:03:32.068659 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.065816 2577 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 31 04:03:32.068659 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.065869 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 04:03:32.068659 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.065874 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 04:03:32.068659 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.065878 2577 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 04:03:32.068659 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.065880 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 04:03:32.068659 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.065883 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 04:03:32.068659 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.065886 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 04:03:32.069118 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.065889 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 04:03:32.069118 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.065891 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 04:03:32.069118 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.065894 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 04:03:32.069118 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.065896 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 04:03:32.069118 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.065900 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 04:03:32.069118 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.065904 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 04:03:32.069118 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.065908 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 04:03:32.069118 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.065911 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 04:03:32.069118 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.065913 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 04:03:32.069118 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.065916 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 04:03:32.069118 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.065919 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 04:03:32.069118 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.065921 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 04:03:32.069118 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.065924 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 04:03:32.069118 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.065926 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 04:03:32.069118 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.065929 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 04:03:32.069118 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.065931 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 04:03:32.069118 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.065934 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 04:03:32.069118 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.065937 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 04:03:32.069550 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.065939 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 04:03:32.069550 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.065942 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 04:03:32.069550 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.065944 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 04:03:32.069550 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.065947 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 04:03:32.069550 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.065949 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 04:03:32.069550 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.065953 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 04:03:32.069550 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.065955 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 04:03:32.069550 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.065958 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 04:03:32.069550 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.065960 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 04:03:32.069550 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.065963 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 04:03:32.069550 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.065966 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 04:03:32.069550 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.065968 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 04:03:32.069550 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.065971 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 04:03:32.069550 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.065973 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 04:03:32.069550 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.065976 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 04:03:32.069550 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.065979 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 04:03:32.069550 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.065982 2577 feature_gate.go:386] unrecognized feature gate: Example Oct 31 04:03:32.069550 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.065984 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 04:03:32.069550 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.065987 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 04:03:32.069550 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.065989 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 04:03:32.070027 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.065992 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 04:03:32.070027 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.065994 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 04:03:32.070027 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.065998 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 04:03:32.070027 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066000 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 04:03:32.070027 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066002 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 04:03:32.070027 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066005 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 04:03:32.070027 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066007 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 04:03:32.070027 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066010 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 04:03:32.070027 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066012 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 04:03:32.070027 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066015 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 04:03:32.070027 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066018 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 04:03:32.070027 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066021 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 04:03:32.070027 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066025 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 04:03:32.070027 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066028 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 04:03:32.070027 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066031 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 04:03:32.070027 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066034 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 04:03:32.070027 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066037 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 04:03:32.070027 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066040 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 04:03:32.070027 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066043 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 04:03:32.070480 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066045 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 04:03:32.070480 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066048 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 04:03:32.070480 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066051 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 04:03:32.070480 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066053 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 04:03:32.070480 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066056 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 04:03:32.070480 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066059 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 04:03:32.070480 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066061 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 04:03:32.070480 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066064 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 04:03:32.070480 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066067 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 04:03:32.070480 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066069 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 04:03:32.070480 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066072 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 04:03:32.070480 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066075 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 04:03:32.070480 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066077 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 04:03:32.070480 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066080 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 04:03:32.070480 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066082 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 04:03:32.070480 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066085 2577 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 04:03:32.070480 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066088 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 04:03:32.070480 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066090 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 04:03:32.070480 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066093 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 04:03:32.070480 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066096 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 04:03:32.070962 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066098 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 04:03:32.070962 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066101 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 04:03:32.070962 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066103 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 04:03:32.070962 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066106 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 04:03:32.070962 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066109 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 04:03:32.070962 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066111 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 04:03:32.070962 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066114 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 04:03:32.070962 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066116 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 04:03:32.070962 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066119 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 04:03:32.070962 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066121 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 04:03:32.070962 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066124 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 04:03:32.070962 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066127 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 04:03:32.070962 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066130 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 04:03:32.070962 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066132 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 04:03:32.070962 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066135 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 04:03:32.070962 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066137 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 04:03:32.070962 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066140 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 04:03:32.070962 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066142 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 04:03:32.070962 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066145 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 04:03:32.070962 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066147 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 04:03:32.071468 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066150 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 04:03:32.071468 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066153 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 04:03:32.071468 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066155 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 04:03:32.071468 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.066184 2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 04:03:32.071468 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066343 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 04:03:32.071468 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066348 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 04:03:32.071468 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066351 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 04:03:32.071468 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066354 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 04:03:32.071468 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066357 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 04:03:32.071468 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066360 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 04:03:32.071468 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066363 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 04:03:32.071468 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066365 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 04:03:32.071468 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066368 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 04:03:32.071468 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066370 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 04:03:32.071468 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066373 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 04:03:32.071468 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066375 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 04:03:32.071843 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066378 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 04:03:32.071843 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066380 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 04:03:32.071843 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066383 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 04:03:32.071843 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066386 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 04:03:32.071843 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066389 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 04:03:32.071843 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066391 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 04:03:32.071843 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066394 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 04:03:32.071843 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066396 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 04:03:32.071843 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066399 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 04:03:32.071843 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066401 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 04:03:32.071843 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066404 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 04:03:32.071843 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066406 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 04:03:32.071843 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066409 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 04:03:32.071843 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066412 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 04:03:32.071843 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066414 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 04:03:32.071843 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066416 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 04:03:32.071843 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066419 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 04:03:32.071843 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066421 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 04:03:32.071843 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066424 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 04:03:32.071843 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066426 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 04:03:32.072305 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066429 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 04:03:32.072305 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066431 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 04:03:32.072305 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066434 2577 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 04:03:32.072305 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066436 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 04:03:32.072305 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066438 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 04:03:32.072305 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066441 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 04:03:32.072305 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066444 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 04:03:32.072305 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066447 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 04:03:32.072305 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066449 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 04:03:32.072305 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066452 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 04:03:32.072305 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066455 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 04:03:32.072305 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066458 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 04:03:32.072305 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066460 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 04:03:32.072305 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066465 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 04:03:32.072305 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066468 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 04:03:32.072305 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066471 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 04:03:32.072305 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066476 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 04:03:32.072305 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066479 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 04:03:32.072733 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066482 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 04:03:32.072733 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066485 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 04:03:32.072733 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066488 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 04:03:32.072733 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066490 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 04:03:32.072733 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066493 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 04:03:32.072733 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066495 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 04:03:32.072733 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066498 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 04:03:32.072733 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066501 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 04:03:32.072733 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066503 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 04:03:32.072733 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066505 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 04:03:32.072733 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066508 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 04:03:32.072733 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066510 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 04:03:32.072733 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066513 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 04:03:32.072733 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066515 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 04:03:32.072733 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066518 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 04:03:32.072733 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066520 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 04:03:32.072733 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066523 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 04:03:32.072733 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066525 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 04:03:32.072733 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066528 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 04:03:32.073174 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066530 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 04:03:32.073174 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066532 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 04:03:32.073174 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066535 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 04:03:32.073174 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066538 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 04:03:32.073174 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066540 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 04:03:32.073174 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066543 2577 feature_gate.go:386] unrecognized feature gate: Example Oct 31 04:03:32.073174 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066545 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 04:03:32.073174 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066548 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 04:03:32.073174 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066550 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 04:03:32.073174 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066552 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 04:03:32.073174 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066555 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 04:03:32.073174 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066558 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 04:03:32.073174 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066560 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 04:03:32.073174 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066563 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 04:03:32.073174 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066565 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 04:03:32.073174 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066568 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 04:03:32.073174 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066570 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 04:03:32.073174 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066572 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 04:03:32.073174 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066575 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 04:03:32.073174 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066577 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 04:03:32.073771 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066580 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 04:03:32.073771 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066582 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 04:03:32.073771 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066585 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 04:03:32.073771 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066587 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 04:03:32.073771 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066590 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 04:03:32.073771 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066592 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 04:03:32.073771 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066594 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 04:03:32.073771 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066597 2577 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 04:03:32.073771 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066599 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 04:03:32.073771 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066602 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 04:03:32.073771 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066604 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 04:03:32.073771 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066607 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 04:03:32.073771 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066609 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 04:03:32.073771 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066612 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 04:03:32.073771 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066614 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 04:03:32.073771 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066617 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 04:03:32.073771 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066620 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 04:03:32.074180 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.066642 2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 04:03:32.074180 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066673 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 04:03:32.074180 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066677 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 04:03:32.074180 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066680 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 04:03:32.074180 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066684 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 04:03:32.074180 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066686 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 04:03:32.074180 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066689 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 04:03:32.074180 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066692 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 04:03:32.074180 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066694 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 04:03:32.074180 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066697 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 04:03:32.074180 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066700 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 04:03:32.074180 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066702 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 04:03:32.074180 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066705 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 04:03:32.074180 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066708 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 04:03:32.074180 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066710 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 04:03:32.074180 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066712 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 04:03:32.074564 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066715 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 04:03:32.074564 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066717 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 04:03:32.074564 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066720 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 04:03:32.074564 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066722 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 04:03:32.074564 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066725 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 04:03:32.074564 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066727 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 04:03:32.074564 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066729 2577 feature_gate.go:386] unrecognized feature gate: Example Oct 31 04:03:32.074564 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066732 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 04:03:32.074564 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066735 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 04:03:32.074564 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066737 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 04:03:32.074564 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066739 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 04:03:32.074564 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066742 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 04:03:32.074564 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066744 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 04:03:32.074564 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066747 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 04:03:32.074564 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066749 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 04:03:32.074564 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066752 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 04:03:32.074564 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066755 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 04:03:32.074564 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066757 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 04:03:32.074564 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066760 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 04:03:32.074564 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066762 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 04:03:32.074997 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066765 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 04:03:32.074997 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066767 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 04:03:32.074997 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066770 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 04:03:32.074997 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066772 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 04:03:32.074997 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066775 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 04:03:32.074997 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066777 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 04:03:32.074997 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066780 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 04:03:32.074997 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066782 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 04:03:32.074997 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066785 2577 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 04:03:32.074997 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066787 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 04:03:32.074997 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066789 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 04:03:32.074997 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066792 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 04:03:32.074997 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066794 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 04:03:32.074997 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066797 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 04:03:32.074997 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066800 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 04:03:32.074997 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066802 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 04:03:32.074997 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066804 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 04:03:32.074997 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066807 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 04:03:32.074997 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066809 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 04:03:32.074997 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066811 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 04:03:32.075469 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066814 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 04:03:32.075469 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066816 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 04:03:32.075469 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066831 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 04:03:32.075469 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066835 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 04:03:32.075469 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066838 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 04:03:32.075469 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066841 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 04:03:32.075469 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066843 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 04:03:32.075469 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066846 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 04:03:32.075469 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066849 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 04:03:32.075469 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066852 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 04:03:32.075469 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066855 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 04:03:32.075469 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066857 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 04:03:32.075469 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066860 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 04:03:32.075469 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066862 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 04:03:32.075469 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066865 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 04:03:32.075469 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066868 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 04:03:32.075469 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066870 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 04:03:32.075469 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066873 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 04:03:32.075469 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066875 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 04:03:32.075916 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066878 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 04:03:32.075916 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066881 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 04:03:32.075916 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066883 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 04:03:32.075916 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066886 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 04:03:32.075916 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066889 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 04:03:32.075916 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066891 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 04:03:32.075916 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066894 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 04:03:32.075916 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066896 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 04:03:32.075916 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066898 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 04:03:32.075916 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066901 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 04:03:32.075916 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066904 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 04:03:32.075916 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066907 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 04:03:32.075916 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066909 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 04:03:32.075916 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066911 2577 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 04:03:32.075916 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066914 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 04:03:32.075916 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066916 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 04:03:32.075916 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066919 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 04:03:32.075916 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066922 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 04:03:32.075916 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066924 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 04:03:32.076510 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066926 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 04:03:32.076510 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066931 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 04:03:32.076510 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066933 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 04:03:32.076510 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066936 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 04:03:32.076510 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066939 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 04:03:32.076510 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066943 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 04:03:32.076510 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066948 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 04:03:32.076510 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066951 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 04:03:32.076510 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066955 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 04:03:32.076510 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066959 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 04:03:32.076510 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066962 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 04:03:32.076510 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066965 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 04:03:32.076510 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:32.066968 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 04:03:32.076510 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.067020 2577 server.go:957] "Client rotation is on, will bootstrap in background" Oct 31 04:03:32.076510 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.071049 2577 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 31 04:03:32.076510 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.072169 2577 server.go:1014] "Starting client certificate rotation" Oct 31 04:03:32.076510 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.072261 2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 31 04:03:32.076510 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.072293 2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 31 04:03:32.077024 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.075148 2577 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 04:03:32.077024 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.076245 2577 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 04:03:32.087996 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.087981 2577 log.go:25] "Validated CRI v1 runtime API" Oct 31 04:03:32.092026 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.092012 2577 log.go:25] "Validated CRI v1 image API" Oct 31 04:03:32.092866 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.092841 2577 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 31 04:03:32.097276 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.097257 2577 fs.go:135] Filesystem UUIDs: map[55f77d54-5511-4a13-b35b-75d0d3b4bf71:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 850455b8-6530-41bf-bdbc-00680ae5e0be:/dev/nvme0n1p3] Oct 31 04:03:32.097370 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.097274 2577 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 31 04:03:32.102488 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.102353 2577 manager.go:217] Machine: {Timestamp:2026-10-31 04:03:32.100968042 +0000 UTC m=+0.316020339 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3199814 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec22d5b32bd50a250ea3be956df55a11 SystemUUID:ec22d5b3-2bd5-0a25-0ea3-be956df55a11 BootID:665524b7-0d7c-415e-90cc-cb508656d751 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:c9:f3:37:cd Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:c9:f3:37:cd Speed:0 Mtu:9001} {Name:ovs-system MacAddress:36:74:41:0b:f7:0e Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 31 04:03:32.103278 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.103268 2577 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 31 04:03:32.103363 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.103351 2577 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 31 04:03:32.106286 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.106267 2577 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 31 04:03:32.106427 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.106286 2577 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-155.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 31 04:03:32.106468 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.106429 2577 topology_manager.go:143] "Creating topology manager with none policy" Oct 31 04:03:32.106468 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.106438 2577 container_manager_linux.go:311] "Creating device plugin manager" Oct 31 04:03:32.106522 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.106472 2577 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 04:03:32.106522 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.106488 2577 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 04:03:32.107720 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.107711 2577 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 31 04:03:32.112379 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.112367 2577 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 31 04:03:32.112500 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.112490 2577 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 31 04:03:32.113369 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.113360 2577 kubelet.go:493] "Attempting to sync node with API server" Oct 31 04:03:32.113406 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.113372 2577 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 31 04:03:32.113406 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.113381 2577 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 31 04:03:32.113406 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.113391 2577 kubelet.go:405] "Adding apiserver pod source" Oct 31 04:03:32.113481 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.113409 2577 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 31 04:03:32.113784 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.113761 2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 04:03:32.117935 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.117911 2577 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 31 04:03:32.125096 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.125080 2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 04:03:32.126119 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.126105 2577 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 31 04:03:32.126186 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.126143 2577 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 31 04:03:32.126356 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.126342 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 31 04:03:32.126414 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.126367 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 31 04:03:32.126414 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.126379 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 31 04:03:32.126414 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.126388 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 31 04:03:32.126414 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.126396 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 31 04:03:32.126414 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.126406 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 31 04:03:32.127164 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.127155 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 31 04:03:32.127223 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.127167 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 31 04:03:32.127223 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.127179 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 31 04:03:32.127223 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.127186 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 31 04:03:32.127223 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.127199 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 31 04:03:32.127223 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.127214 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 31 04:03:32.128910 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.128898 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 31 04:03:32.128972 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.128916 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 31 04:03:32.132533 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.132518 2577 server.go:1263] "Started kubelet" Oct 31 04:03:32.132626 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.132587 2577 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 31 04:03:32.132709 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.132627 2577 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 31 04:03:32.132749 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.132730 2577 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 31 04:03:32.133165 ip-10-0-134-155 systemd[1]: Started Kubernetes Kubelet. Oct 31 04:03:32.134212 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.134195 2577 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 31 04:03:32.135815 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.135795 2577 server.go:317] "Adding debug handlers to kubelet server" Oct 31 04:03:32.139813 ip-10-0-134-155 kubenswrapper[2577]: E1031 04:03:32.139793 2577 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 31 04:03:32.143292 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.143276 2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 31 04:03:32.143879 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.143860 2577 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 31 04:03:32.143963 ip-10-0-134-155 kubenswrapper[2577]: E1031 04:03:32.143926 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-155.ec2.internal\" not found" Oct 31 04:03:32.144553 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.144536 2577 factory.go:55] Registering systemd factory Oct 31 04:03:32.144553 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.144551 2577 factory.go:223] Registration of the systemd container factory successfully Oct 31 04:03:32.144682 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.144642 2577 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 31 04:03:32.144682 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.144656 2577 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 31 04:03:32.144682 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.144660 2577 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 31 04:03:32.144802 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.144709 2577 reconstruct.go:97] "Volume reconstruction finished" Oct 31 04:03:32.144802 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.144716 2577 reconciler.go:29] "Reconciler: start to sync state" Oct 31 04:03:32.144802 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.144745 2577 factory.go:153] Registering CRI-O factory Oct 31 04:03:32.144802 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.144756 2577 factory.go:223] Registration of the crio container factory successfully Oct 31 04:03:32.144802 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.144797 2577 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 31 04:03:32.145024 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.144810 2577 factory.go:103] Registering Raw factory Oct 31 04:03:32.145024 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.144820 2577 manager.go:1199] Started watching for new ooms in manager Oct 31 04:03:32.145902 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.145889 2577 manager.go:319] Starting recovery of all containers Oct 31 04:03:32.159705 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.159687 2577 manager.go:324] Recovery completed Oct 31 04:03:32.163578 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.163565 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 04:03:32.165356 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.165336 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-155.ec2.internal" event="NodeHasSufficientMemory" Oct 31 04:03:32.165444 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.165364 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-155.ec2.internal" event="NodeHasNoDiskPressure" Oct 31 04:03:32.165444 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.165378 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-155.ec2.internal" event="NodeHasSufficientPID" Oct 31 04:03:32.165899 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.165884 2577 cpu_manager.go:226] "Starting" policy="none" Oct 31 04:03:32.165955 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.165898 2577 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 31 04:03:32.165955 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.165911 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 04:03:32.165955 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.165917 2577 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 31 04:03:32.166580 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.166563 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-155.ec2.internal" event="NodeHasSufficientMemory" Oct 31 04:03:32.166658 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.166590 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-155.ec2.internal" event="NodeHasNoDiskPressure" Oct 31 04:03:32.166658 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.166600 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-155.ec2.internal" event="NodeHasSufficientPID" Oct 31 04:03:32.168089 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.168076 2577 policy_none.go:50] "Start" Oct 31 04:03:32.168133 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.168094 2577 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 31 04:03:32.168133 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.168107 2577 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 31 04:03:32.170129 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.170111 2577 policy_none.go:44] "Start" Oct 31 04:03:32.170245 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.170113 2577 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-134-155.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 31 04:03:32.170356 ip-10-0-134-155 kubenswrapper[2577]: E1031 04:03:32.170107 2577 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-134-155.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 31 04:03:32.171245 ip-10-0-134-155 kubenswrapper[2577]: E1031 04:03:32.170095 2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-155.ec2.internal.18e381741172842a default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-155.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-134-155.ec2.internal,},FirstTimestamp:2026-10-31 04:03:32.132496426 +0000 UTC m=+0.347548722,LastTimestamp:2026-10-31 04:03:32.132496426 +0000 UTC m=+0.347548722,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-155.ec2.internal,}" Oct 31 04:03:32.172022 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.172004 2577 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 31 04:03:32.172088 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.172037 2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-fw9nj" Oct 31 04:03:32.179028 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.179000 2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-fw9nj" Oct 31 04:03:32.185151 ip-10-0-134-155 kubenswrapper[2577]: E1031 04:03:32.185123 2577 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Oct 31 04:03:32.185237 ip-10-0-134-155 kubenswrapper[2577]: E1031 04:03:32.185150 2577 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-134-155.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Oct 31 04:03:32.185237 ip-10-0-134-155 kubenswrapper[2577]: E1031 04:03:32.185141 2577 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Oct 31 04:03:32.185300 ip-10-0-134-155 kubenswrapper[2577]: E1031 04:03:32.185178 2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-155.ec2.internal.18e381741367df0e default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-155.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-134-155.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-134-155.ec2.internal,},FirstTimestamp:2026-10-31 04:03:32.16535323 +0000 UTC m=+0.380405528,LastTimestamp:2026-10-31 04:03:32.16535323 +0000 UTC m=+0.380405528,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-155.ec2.internal,}" Oct 31 04:03:32.213912 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.213876 2577 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 31 04:03:32.216483 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.216470 2577 manager.go:341] "Starting Device Plugin manager" Oct 31 04:03:32.216556 ip-10-0-134-155 kubenswrapper[2577]: E1031 04:03:32.216498 2577 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 31 04:03:32.216556 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.216509 2577 server.go:87] "Starting device plugin registration server" Oct 31 04:03:32.216753 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.216740 2577 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 31 04:03:32.216810 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.216753 2577 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 31 04:03:32.217030 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.216908 2577 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 31 04:03:32.217030 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.216984 2577 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 31 04:03:32.217030 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.216992 2577 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 31 04:03:32.217566 ip-10-0-134-155 kubenswrapper[2577]: E1031 04:03:32.217549 2577 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 31 04:03:32.217659 ip-10-0-134-155 kubenswrapper[2577]: E1031 04:03:32.217601 2577 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-134-155.ec2.internal\" not found" Oct 31 04:03:32.271679 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.271655 2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 31 04:03:32.273603 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.272744 2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 31 04:03:32.273603 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.272764 2577 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 31 04:03:32.273603 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.272782 2577 kubelet.go:2521] "Starting kubelet main sync loop" Oct 31 04:03:32.273603 ip-10-0-134-155 kubenswrapper[2577]: E1031 04:03:32.272818 2577 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Oct 31 04:03:32.287156 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.287140 2577 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 04:03:32.317562 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.317516 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 04:03:32.318235 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.318219 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-155.ec2.internal" event="NodeHasSufficientMemory" Oct 31 04:03:32.318321 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.318248 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-155.ec2.internal" event="NodeHasNoDiskPressure" Oct 31 04:03:32.318321 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.318262 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-155.ec2.internal" event="NodeHasSufficientPID" Oct 31 04:03:32.318321 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.318290 2577 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-134-155.ec2.internal" Oct 31 04:03:32.335007 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.334987 2577 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-134-155.ec2.internal" Oct 31 04:03:32.335085 ip-10-0-134-155 kubenswrapper[2577]: E1031 04:03:32.335009 2577 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-134-155.ec2.internal\": node \"ip-10-0-134-155.ec2.internal\" not found" Oct 31 04:03:32.373280 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.373258 2577 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-134-155.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-155.ec2.internal"] Oct 31 04:03:32.373384 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.373309 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 04:03:32.374106 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.374094 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-155.ec2.internal" event="NodeHasSufficientMemory" Oct 31 04:03:32.374166 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.374116 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-155.ec2.internal" event="NodeHasNoDiskPressure" Oct 31 04:03:32.374166 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.374126 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-155.ec2.internal" event="NodeHasSufficientPID" Oct 31 04:03:32.376226 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.376215 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 04:03:32.376381 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.376360 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-155.ec2.internal" Oct 31 04:03:32.376445 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.376392 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 04:03:32.377203 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.377185 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-155.ec2.internal" event="NodeHasSufficientMemory" Oct 31 04:03:32.377203 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.377199 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-155.ec2.internal" event="NodeHasSufficientMemory" Oct 31 04:03:32.377316 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.377209 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-155.ec2.internal" event="NodeHasNoDiskPressure" Oct 31 04:03:32.377316 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.377221 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-155.ec2.internal" event="NodeHasNoDiskPressure" Oct 31 04:03:32.377316 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.377237 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-155.ec2.internal" event="NodeHasSufficientPID" Oct 31 04:03:32.377316 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.377222 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-155.ec2.internal" event="NodeHasSufficientPID" Oct 31 04:03:32.377485 ip-10-0-134-155 kubenswrapper[2577]: E1031 04:03:32.377471 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-155.ec2.internal\" not found" Oct 31 04:03:32.379364 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.379344 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-155.ec2.internal" Oct 31 04:03:32.379409 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.379375 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 04:03:32.380065 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.380051 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-155.ec2.internal" event="NodeHasSufficientMemory" Oct 31 04:03:32.380139 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.380069 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-155.ec2.internal" event="NodeHasNoDiskPressure" Oct 31 04:03:32.380139 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.380079 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-155.ec2.internal" event="NodeHasSufficientPID" Oct 31 04:03:32.406826 ip-10-0-134-155 kubenswrapper[2577]: E1031 04:03:32.406809 2577 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-155.ec2.internal\" not found" node="ip-10-0-134-155.ec2.internal" Oct 31 04:03:32.412973 ip-10-0-134-155 kubenswrapper[2577]: E1031 04:03:32.412953 2577 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-155.ec2.internal\" not found" node="ip-10-0-134-155.ec2.internal" Oct 31 04:03:32.478197 ip-10-0-134-155 kubenswrapper[2577]: E1031 04:03:32.478182 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-155.ec2.internal\" not found" Oct 31 04:03:32.545770 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.545755 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b3351600635c8d30ebae275373d4b16d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-155.ec2.internal\" (UID: \"b3351600635c8d30ebae275373d4b16d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-155.ec2.internal" Oct 31 04:03:32.545840 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.545779 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b3351600635c8d30ebae275373d4b16d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-155.ec2.internal\" (UID: \"b3351600635c8d30ebae275373d4b16d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-155.ec2.internal" Oct 31 04:03:32.545840 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.545798 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/db7fb1e6e2bd3665becb16d5fafc3d7e-config\") pod \"kube-apiserver-proxy-ip-10-0-134-155.ec2.internal\" (UID: \"db7fb1e6e2bd3665becb16d5fafc3d7e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-155.ec2.internal" Oct 31 04:03:32.579274 ip-10-0-134-155 kubenswrapper[2577]: E1031 04:03:32.579234 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-155.ec2.internal\" not found" Oct 31 04:03:32.646845 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.646827 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/db7fb1e6e2bd3665becb16d5fafc3d7e-config\") pod \"kube-apiserver-proxy-ip-10-0-134-155.ec2.internal\" (UID: \"db7fb1e6e2bd3665becb16d5fafc3d7e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-155.ec2.internal" Oct 31 04:03:32.646901 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.646842 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/db7fb1e6e2bd3665becb16d5fafc3d7e-config\") pod \"kube-apiserver-proxy-ip-10-0-134-155.ec2.internal\" (UID: \"db7fb1e6e2bd3665becb16d5fafc3d7e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-155.ec2.internal" Oct 31 04:03:32.646901 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.646889 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b3351600635c8d30ebae275373d4b16d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-155.ec2.internal\" (UID: \"b3351600635c8d30ebae275373d4b16d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-155.ec2.internal" Oct 31 04:03:32.646963 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.646905 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b3351600635c8d30ebae275373d4b16d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-155.ec2.internal\" (UID: \"b3351600635c8d30ebae275373d4b16d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-155.ec2.internal" Oct 31 04:03:32.646963 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.646919 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b3351600635c8d30ebae275373d4b16d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-155.ec2.internal\" (UID: \"b3351600635c8d30ebae275373d4b16d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-155.ec2.internal" Oct 31 04:03:32.646963 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.646941 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b3351600635c8d30ebae275373d4b16d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-155.ec2.internal\" (UID: \"b3351600635c8d30ebae275373d4b16d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-155.ec2.internal" Oct 31 04:03:32.679902 ip-10-0-134-155 kubenswrapper[2577]: E1031 04:03:32.679886 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-155.ec2.internal\" not found" Oct 31 04:03:32.710078 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.710063 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-155.ec2.internal" Oct 31 04:03:32.716659 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:32.716549 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-155.ec2.internal" Oct 31 04:03:32.780767 ip-10-0-134-155 kubenswrapper[2577]: E1031 04:03:32.780747 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-155.ec2.internal\" not found" Oct 31 04:03:32.881216 ip-10-0-134-155 kubenswrapper[2577]: E1031 04:03:32.881173 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-155.ec2.internal\" not found" Oct 31 04:03:32.981693 ip-10-0-134-155 kubenswrapper[2577]: E1031 04:03:32.981671 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-155.ec2.internal\" not found" Oct 31 04:03:33.072157 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:33.072137 2577 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 31 04:03:33.072555 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:33.072267 2577 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 31 04:03:33.082430 ip-10-0-134-155 kubenswrapper[2577]: E1031 04:03:33.082413 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-155.ec2.internal\" not found" Oct 31 04:03:33.171743 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:33.171710 2577 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 04:03:33.182683 ip-10-0-134-155 kubenswrapper[2577]: E1031 04:03:33.182659 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-155.ec2.internal\" not found" Oct 31 04:03:33.186077 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:33.185412 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-30 03:58:33 +0000 UTC" deadline="2028-05-06 15:45:13.690657377 +0000 UTC" Oct 31 04:03:33.186077 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:33.185445 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13283h41m40.505215536s" Oct 31 04:03:33.210266 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:33.210250 2577 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 04:03:33.276028 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:33.276002 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb7fb1e6e2bd3665becb16d5fafc3d7e.slice/crio-8022cc9c6ccbd005a0ecdab2ba41a52a5e4c05e4b51c3c7f012c5cd630f6341b WatchSource:0}: Error finding container 8022cc9c6ccbd005a0ecdab2ba41a52a5e4c05e4b51c3c7f012c5cd630f6341b: Status 404 returned error can't find the container with id 8022cc9c6ccbd005a0ecdab2ba41a52a5e4c05e4b51c3c7f012c5cd630f6341b Oct 31 04:03:33.277113 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:33.277091 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3351600635c8d30ebae275373d4b16d.slice/crio-1ddb915eccc0737c404aca07c5758b96a9f0fdd59c1cf88f059dae0e94b737d9 WatchSource:0}: Error finding container 1ddb915eccc0737c404aca07c5758b96a9f0fdd59c1cf88f059dae0e94b737d9: Status 404 returned error can't find the container with id 1ddb915eccc0737c404aca07c5758b96a9f0fdd59c1cf88f059dae0e94b737d9 Oct 31 04:03:33.280969 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:33.280954 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 04:03:33.283051 ip-10-0-134-155 kubenswrapper[2577]: E1031 04:03:33.283031 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-155.ec2.internal\" not found" Oct 31 04:03:33.355984 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:33.355930 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 04:03:33.356836 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:33.356820 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-155.ec2.internal" event="NodeHasSufficientMemory" Oct 31 04:03:33.356910 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:33.356850 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-155.ec2.internal" event="NodeHasNoDiskPressure" Oct 31 04:03:33.356910 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:33.356861 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-155.ec2.internal" event="NodeHasSufficientPID" Oct 31 04:03:33.383518 ip-10-0-134-155 kubenswrapper[2577]: E1031 04:03:33.383495 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-155.ec2.internal\" not found" Oct 31 04:03:33.485360 ip-10-0-134-155 kubenswrapper[2577]: E1031 04:03:33.485303 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-155.ec2.internal\" not found" Oct 31 04:03:33.546767 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:33.546747 2577 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 04:03:33.644917 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:33.644891 2577 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-155.ec2.internal" Oct 31 04:03:33.667654 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:33.667632 2577 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 04:03:33.668625 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:33.668613 2577 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-155.ec2.internal" Oct 31 04:03:33.691558 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:33.691538 2577 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 04:03:34.114519 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.114490 2577 apiserver.go:52] "Watching apiserver" Oct 31 04:03:34.143879 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.143854 2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 31 04:03:34.144680 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.144657 2577 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 31 04:03:34.145120 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.145091 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-77z7p","openshift-image-registry/node-ca-xmbxc","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-155.ec2.internal","openshift-multus/multus-additional-cni-plugins-kmh48","openshift-cluster-node-tuning-operator/tuned-b59p9","openshift-network-diagnostics/network-check-target-w9sdm","openshift-ovn-kubernetes/ovnkube-node-zbh9t","kube-system/global-pull-secret-syncer-tlcrp","kube-system/konnectivity-agent-ntbm9","openshift-multus/multus-zmrmk","openshift-multus/network-metrics-daemon-gs42n","openshift-network-operator/iptables-alerter-6q4xv","kube-system/kube-apiserver-proxy-ip-10-0-134-155.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zw4rh"] Oct 31 04:03:34.147900 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.147878 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-zbh9t" Oct 31 04:03:34.150298 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.150277 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gs42n" Oct 31 04:03:34.150423 ip-10-0-134-155 kubenswrapper[2577]: E1031 04:03:34.150366 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gs42n" podUID="e7f09238-2bbf-4587-9989-49419d90dabf" Oct 31 04:03:34.153253 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.153231 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zw4rh" Oct 31 04:03:34.156432 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.156399 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-kmh48" Oct 31 04:03:34.159049 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.158493 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-b59p9" Oct 31 04:03:34.162141 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.162123 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w9sdm" Oct 31 04:03:34.162268 ip-10-0-134-155 kubenswrapper[2577]: E1031 04:03:34.162231 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w9sdm" podUID="f70f5eab-445f-430e-bfb1-92e4d620b90d" Oct 31 04:03:34.163957 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.163713 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 31 04:03:34.163957 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.163866 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 31 04:03:34.163957 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.163920 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 31 04:03:34.163957 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.163935 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 31 04:03:34.164281 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.164263 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-ntbm9" Oct 31 04:03:34.169171 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.169060 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tlcrp" Oct 31 04:03:34.169171 ip-10-0-134-155 kubenswrapper[2577]: E1031 04:03:34.169130 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tlcrp" podUID="bc9d2733-89fa-4079-9dc1-01983df773cb" Oct 31 04:03:34.171159 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.171136 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 31 04:03:34.171260 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.171162 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 31 04:03:34.171747 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.171724 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 31 04:03:34.171880 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.171822 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 31 04:03:34.171961 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.171949 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 31 04:03:34.172557 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.172538 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-ctbtg\"" Oct 31 04:03:34.172646 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.172553 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-xmbxc" Oct 31 04:03:34.175052 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.174662 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-zmrmk" Oct 31 04:03:34.175052 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.175025 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 31 04:03:34.175203 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.175058 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 31 04:03:34.175203 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.175078 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-kfpck\"" Oct 31 04:03:34.175203 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.175119 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-pjn2p\"" Oct 31 04:03:34.175203 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.175134 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 31 04:03:34.175378 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.175209 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 31 04:03:34.177698 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.177680 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-77z7p" Oct 31 04:03:34.180267 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.180245 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-6q4xv" Oct 31 04:03:34.204657 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.204637 2577 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 31 04:03:34.204760 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.204700 2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-6jvwm" Oct 31 04:03:34.220576 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.220553 2577 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 04:03:34.246947 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.246930 2577 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 31 04:03:34.255684 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.255659 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxgq7\" (UniqueName: \"kubernetes.io/projected/1f2543cc-b528-41be-92e0-9d654405a723-kube-api-access-fxgq7\") pod \"node-resolver-77z7p\" (UID: \"1f2543cc-b528-41be-92e0-9d654405a723\") " pod="openshift-dns/node-resolver-77z7p" Oct 31 04:03:34.255798 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.255701 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f7a16817-8ea2-4649-9e42-3d340e1b9703-ovnkube-script-lib\") pod \"ovnkube-node-zbh9t\" (UID: \"f7a16817-8ea2-4649-9e42-3d340e1b9703\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbh9t" Oct 31 04:03:34.255798 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.255749 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwltl\" (UniqueName: \"kubernetes.io/projected/c581e590-87e5-4cd7-8d84-eabb4ee21fd1-kube-api-access-wwltl\") pod \"aws-ebs-csi-driver-node-zw4rh\" (UID: \"c581e590-87e5-4cd7-8d84-eabb4ee21fd1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zw4rh" Oct 31 04:03:34.255888 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.255821 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bc9d2733-89fa-4079-9dc1-01983df773cb-original-pull-secret\") pod \"global-pull-secret-syncer-tlcrp\" (UID: \"bc9d2733-89fa-4079-9dc1-01983df773cb\") " pod="kube-system/global-pull-secret-syncer-tlcrp" Oct 31 04:03:34.255888 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.255861 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1f2543cc-b528-41be-92e0-9d654405a723-hosts-file\") pod \"node-resolver-77z7p\" (UID: \"1f2543cc-b528-41be-92e0-9d654405a723\") " pod="openshift-dns/node-resolver-77z7p" Oct 31 04:03:34.255993 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.255915 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f7a16817-8ea2-4649-9e42-3d340e1b9703-host-run-netns\") pod \"ovnkube-node-zbh9t\" (UID: \"f7a16817-8ea2-4649-9e42-3d340e1b9703\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbh9t" Oct 31 04:03:34.255993 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.255943 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f7a16817-8ea2-4649-9e42-3d340e1b9703-run-openvswitch\") pod \"ovnkube-node-zbh9t\" (UID: \"f7a16817-8ea2-4649-9e42-3d340e1b9703\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbh9t" Oct 31 04:03:34.255993 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.255968 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f7a16817-8ea2-4649-9e42-3d340e1b9703-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zbh9t\" (UID: \"f7a16817-8ea2-4649-9e42-3d340e1b9703\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbh9t" Oct 31 04:03:34.256112 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.256005 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n54k7\" (UniqueName: \"kubernetes.io/projected/d4eabffb-abf1-4409-94b5-c0489ab37853-kube-api-access-n54k7\") pod \"tuned-b59p9\" (UID: \"d4eabffb-abf1-4409-94b5-c0489ab37853\") " pod="openshift-cluster-node-tuning-operator/tuned-b59p9" Oct 31 04:03:34.256112 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.256077 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/153e979c-3c5d-4c4f-b838-fc26db0b5cac-cnibin\") pod \"multus-zmrmk\" (UID: \"153e979c-3c5d-4c4f-b838-fc26db0b5cac\") " pod="openshift-multus/multus-zmrmk" Oct 31 04:03:34.256112 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.256101 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/153e979c-3c5d-4c4f-b838-fc26db0b5cac-multus-daemon-config\") pod \"multus-zmrmk\" (UID: \"153e979c-3c5d-4c4f-b838-fc26db0b5cac\") " pod="openshift-multus/multus-zmrmk" Oct 31 04:03:34.256259 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.256126 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/153e979c-3c5d-4c4f-b838-fc26db0b5cac-etc-kubernetes\") pod \"multus-zmrmk\" (UID: \"153e979c-3c5d-4c4f-b838-fc26db0b5cac\") " pod="openshift-multus/multus-zmrmk" Oct 31 04:03:34.256259 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.256151 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f7a16817-8ea2-4649-9e42-3d340e1b9703-etc-openvswitch\") pod \"ovnkube-node-zbh9t\" (UID: \"f7a16817-8ea2-4649-9e42-3d340e1b9703\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbh9t" Oct 31 04:03:34.256259 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.256176 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f7a16817-8ea2-4649-9e42-3d340e1b9703-log-socket\") pod \"ovnkube-node-zbh9t\" (UID: \"f7a16817-8ea2-4649-9e42-3d340e1b9703\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbh9t" Oct 31 04:03:34.256259 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.256201 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkkp9\" (UniqueName: \"kubernetes.io/projected/f7a16817-8ea2-4649-9e42-3d340e1b9703-kube-api-access-dkkp9\") pod \"ovnkube-node-zbh9t\" (UID: \"f7a16817-8ea2-4649-9e42-3d340e1b9703\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbh9t" Oct 31 04:03:34.256444 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.256267 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7f09238-2bbf-4587-9989-49419d90dabf-metrics-certs\") pod \"network-metrics-daemon-gs42n\" (UID: \"e7f09238-2bbf-4587-9989-49419d90dabf\") " pod="openshift-multus/network-metrics-daemon-gs42n" Oct 31 04:03:34.256444 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.256303 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f2543cc-b528-41be-92e0-9d654405a723-tmp-dir\") pod \"node-resolver-77z7p\" (UID: \"1f2543cc-b528-41be-92e0-9d654405a723\") " pod="openshift-dns/node-resolver-77z7p" Oct 31 04:03:34.256444 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.256326 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d4eabffb-abf1-4409-94b5-c0489ab37853-etc-tuned\") pod \"tuned-b59p9\" (UID: \"d4eabffb-abf1-4409-94b5-c0489ab37853\") " pod="openshift-cluster-node-tuning-operator/tuned-b59p9" Oct 31 04:03:34.256444 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.256357 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f12d4ae1-e673-4a9e-bd43-92d6e9d5a023-konnectivity-ca\") pod \"konnectivity-agent-ntbm9\" (UID: \"f12d4ae1-e673-4a9e-bd43-92d6e9d5a023\") " pod="kube-system/konnectivity-agent-ntbm9" Oct 31 04:03:34.256444 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.256401 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bc9d2733-89fa-4079-9dc1-01983df773cb-dbus\") pod \"global-pull-secret-syncer-tlcrp\" (UID: \"bc9d2733-89fa-4079-9dc1-01983df773cb\") " pod="kube-system/global-pull-secret-syncer-tlcrp" Oct 31 04:03:34.256444 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.256435 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/153e979c-3c5d-4c4f-b838-fc26db0b5cac-multus-conf-dir\") pod \"multus-zmrmk\" (UID: \"153e979c-3c5d-4c4f-b838-fc26db0b5cac\") " pod="openshift-multus/multus-zmrmk" Oct 31 04:03:34.256681 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.256460 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f7a16817-8ea2-4649-9e42-3d340e1b9703-host-kubelet\") pod \"ovnkube-node-zbh9t\" (UID: \"f7a16817-8ea2-4649-9e42-3d340e1b9703\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbh9t" Oct 31 04:03:34.256681 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.256493 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f7a16817-8ea2-4649-9e42-3d340e1b9703-host-run-ovn-kubernetes\") pod \"ovnkube-node-zbh9t\" (UID: \"f7a16817-8ea2-4649-9e42-3d340e1b9703\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbh9t" Oct 31 04:03:34.256681 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.256517 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c581e590-87e5-4cd7-8d84-eabb4ee21fd1-registration-dir\") pod \"aws-ebs-csi-driver-node-zw4rh\" (UID: \"c581e590-87e5-4cd7-8d84-eabb4ee21fd1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zw4rh" Oct 31 04:03:34.256681 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.256541 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d4eabffb-abf1-4409-94b5-c0489ab37853-var-lib-kubelet\") pod \"tuned-b59p9\" (UID: \"d4eabffb-abf1-4409-94b5-c0489ab37853\") " pod="openshift-cluster-node-tuning-operator/tuned-b59p9" Oct 31 04:03:34.256681 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.256565 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/153e979c-3c5d-4c4f-b838-fc26db0b5cac-cni-binary-copy\") pod \"multus-zmrmk\" (UID: \"153e979c-3c5d-4c4f-b838-fc26db0b5cac\") " pod="openshift-multus/multus-zmrmk" Oct 31 04:03:34.256681 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.256600 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/153e979c-3c5d-4c4f-b838-fc26db0b5cac-host-var-lib-cni-bin\") pod \"multus-zmrmk\" (UID: \"153e979c-3c5d-4c4f-b838-fc26db0b5cac\") " pod="openshift-multus/multus-zmrmk" Oct 31 04:03:34.256681 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.256630 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/153e979c-3c5d-4c4f-b838-fc26db0b5cac-host-run-multus-certs\") pod \"multus-zmrmk\" (UID: \"153e979c-3c5d-4c4f-b838-fc26db0b5cac\") " pod="openshift-multus/multus-zmrmk" Oct 31 04:03:34.256934 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.256768 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f7a16817-8ea2-4649-9e42-3d340e1b9703-run-ovn\") pod \"ovnkube-node-zbh9t\" (UID: \"f7a16817-8ea2-4649-9e42-3d340e1b9703\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbh9t" Oct 31 04:03:34.256934 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.256804 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d4eabffb-abf1-4409-94b5-c0489ab37853-etc-systemd\") pod \"tuned-b59p9\" (UID: \"d4eabffb-abf1-4409-94b5-c0489ab37853\") " pod="openshift-cluster-node-tuning-operator/tuned-b59p9" Oct 31 04:03:34.256934 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.256837 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a90157bb-46c0-4afa-82b1-9f6819518744-host\") pod \"node-ca-xmbxc\" (UID: \"a90157bb-46c0-4afa-82b1-9f6819518744\") " pod="openshift-image-registry/node-ca-xmbxc" Oct 31 04:03:34.256934 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.256865 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c581e590-87e5-4cd7-8d84-eabb4ee21fd1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zw4rh\" (UID: \"c581e590-87e5-4cd7-8d84-eabb4ee21fd1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zw4rh" Oct 31 04:03:34.256934 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.256911 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c581e590-87e5-4cd7-8d84-eabb4ee21fd1-sys-fs\") pod \"aws-ebs-csi-driver-node-zw4rh\" (UID: \"c581e590-87e5-4cd7-8d84-eabb4ee21fd1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zw4rh" Oct 31 04:03:34.257133 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.256935 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d4eabffb-abf1-4409-94b5-c0489ab37853-etc-kubernetes\") pod \"tuned-b59p9\" (UID: \"d4eabffb-abf1-4409-94b5-c0489ab37853\") " pod="openshift-cluster-node-tuning-operator/tuned-b59p9" Oct 31 04:03:34.257133 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.256991 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d4eabffb-abf1-4409-94b5-c0489ab37853-lib-modules\") pod \"tuned-b59p9\" (UID: \"d4eabffb-abf1-4409-94b5-c0489ab37853\") " pod="openshift-cluster-node-tuning-operator/tuned-b59p9" Oct 31 04:03:34.257133 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.257015 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d4eabffb-abf1-4409-94b5-c0489ab37853-tmp\") pod \"tuned-b59p9\" (UID: \"d4eabffb-abf1-4409-94b5-c0489ab37853\") " pod="openshift-cluster-node-tuning-operator/tuned-b59p9" Oct 31 04:03:34.257133 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.257052 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/153e979c-3c5d-4c4f-b838-fc26db0b5cac-system-cni-dir\") pod \"multus-zmrmk\" (UID: \"153e979c-3c5d-4c4f-b838-fc26db0b5cac\") " pod="openshift-multus/multus-zmrmk" Oct 31 04:03:34.257133 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.257075 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/153e979c-3c5d-4c4f-b838-fc26db0b5cac-host-run-k8s-cni-cncf-io\") pod \"multus-zmrmk\" (UID: \"153e979c-3c5d-4c4f-b838-fc26db0b5cac\") " pod="openshift-multus/multus-zmrmk" Oct 31 04:03:34.257133 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.257112 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bc9d2733-89fa-4079-9dc1-01983df773cb-kubelet-config\") pod \"global-pull-secret-syncer-tlcrp\" (UID: \"bc9d2733-89fa-4079-9dc1-01983df773cb\") " pod="kube-system/global-pull-secret-syncer-tlcrp" Oct 31 04:03:34.257372 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.257152 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d4eabffb-abf1-4409-94b5-c0489ab37853-etc-sysconfig\") pod \"tuned-b59p9\" (UID: \"d4eabffb-abf1-4409-94b5-c0489ab37853\") " pod="openshift-cluster-node-tuning-operator/tuned-b59p9" Oct 31 04:03:34.257372 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.257180 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d4eabffb-abf1-4409-94b5-c0489ab37853-etc-sysctl-d\") pod \"tuned-b59p9\" (UID: \"d4eabffb-abf1-4409-94b5-c0489ab37853\") " pod="openshift-cluster-node-tuning-operator/tuned-b59p9" Oct 31 04:03:34.257372 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.257204 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/79ae1195-4ff1-46da-b1e2-83042dba6caa-cnibin\") pod \"multus-additional-cni-plugins-kmh48\" (UID: \"79ae1195-4ff1-46da-b1e2-83042dba6caa\") " pod="openshift-multus/multus-additional-cni-plugins-kmh48" Oct 31 04:03:34.257372 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.257253 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/79ae1195-4ff1-46da-b1e2-83042dba6caa-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kmh48\" (UID: \"79ae1195-4ff1-46da-b1e2-83042dba6caa\") " pod="openshift-multus/multus-additional-cni-plugins-kmh48" Oct 31 04:03:34.257372 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.257276 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d4eabffb-abf1-4409-94b5-c0489ab37853-sys\") pod \"tuned-b59p9\" (UID: \"d4eabffb-abf1-4409-94b5-c0489ab37853\") " pod="openshift-cluster-node-tuning-operator/tuned-b59p9" Oct 31 04:03:34.257372 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.257298 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d4eabffb-abf1-4409-94b5-c0489ab37853-host\") pod \"tuned-b59p9\" (UID: \"d4eabffb-abf1-4409-94b5-c0489ab37853\") " pod="openshift-cluster-node-tuning-operator/tuned-b59p9" Oct 31 04:03:34.257372 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.257324 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f7a16817-8ea2-4649-9e42-3d340e1b9703-ovnkube-config\") pod \"ovnkube-node-zbh9t\" (UID: \"f7a16817-8ea2-4649-9e42-3d340e1b9703\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbh9t" Oct 31 04:03:34.257372 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.257357 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f7a16817-8ea2-4649-9e42-3d340e1b9703-env-overrides\") pod \"ovnkube-node-zbh9t\" (UID: \"f7a16817-8ea2-4649-9e42-3d340e1b9703\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbh9t" Oct 31 04:03:34.257726 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.257382 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9r7fx\" (UniqueName: \"kubernetes.io/projected/e7f09238-2bbf-4587-9989-49419d90dabf-kube-api-access-9r7fx\") pod \"network-metrics-daemon-gs42n\" (UID: \"e7f09238-2bbf-4587-9989-49419d90dabf\") " pod="openshift-multus/network-metrics-daemon-gs42n" Oct 31 04:03:34.257726 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.257422 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-786gp\" (UniqueName: \"kubernetes.io/projected/a90157bb-46c0-4afa-82b1-9f6819518744-kube-api-access-786gp\") pod \"node-ca-xmbxc\" (UID: \"a90157bb-46c0-4afa-82b1-9f6819518744\") " pod="openshift-image-registry/node-ca-xmbxc" Oct 31 04:03:34.257726 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.257446 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vp84q\" (UniqueName: \"kubernetes.io/projected/153e979c-3c5d-4c4f-b838-fc26db0b5cac-kube-api-access-vp84q\") pod \"multus-zmrmk\" (UID: \"153e979c-3c5d-4c4f-b838-fc26db0b5cac\") " pod="openshift-multus/multus-zmrmk" Oct 31 04:03:34.257726 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.257471 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f7a16817-8ea2-4649-9e42-3d340e1b9703-host-slash\") pod \"ovnkube-node-zbh9t\" (UID: \"f7a16817-8ea2-4649-9e42-3d340e1b9703\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbh9t" Oct 31 04:03:34.257726 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.257678 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c581e590-87e5-4cd7-8d84-eabb4ee21fd1-socket-dir\") pod \"aws-ebs-csi-driver-node-zw4rh\" (UID: \"c581e590-87e5-4cd7-8d84-eabb4ee21fd1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zw4rh" Oct 31 04:03:34.257726 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.257709 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jq5vs\" (UniqueName: \"kubernetes.io/projected/f70f5eab-445f-430e-bfb1-92e4d620b90d-kube-api-access-jq5vs\") pod \"network-check-target-w9sdm\" (UID: \"f70f5eab-445f-430e-bfb1-92e4d620b90d\") " pod="openshift-network-diagnostics/network-check-target-w9sdm" Oct 31 04:03:34.257938 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.257750 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/79ae1195-4ff1-46da-b1e2-83042dba6caa-system-cni-dir\") pod \"multus-additional-cni-plugins-kmh48\" (UID: \"79ae1195-4ff1-46da-b1e2-83042dba6caa\") " pod="openshift-multus/multus-additional-cni-plugins-kmh48" Oct 31 04:03:34.257938 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.257779 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a90157bb-46c0-4afa-82b1-9f6819518744-serviceca\") pod \"node-ca-xmbxc\" (UID: \"a90157bb-46c0-4afa-82b1-9f6819518744\") " pod="openshift-image-registry/node-ca-xmbxc" Oct 31 04:03:34.257938 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.257808 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/153e979c-3c5d-4c4f-b838-fc26db0b5cac-multus-cni-dir\") pod \"multus-zmrmk\" (UID: \"153e979c-3c5d-4c4f-b838-fc26db0b5cac\") " pod="openshift-multus/multus-zmrmk" Oct 31 04:03:34.257938 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.257832 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/153e979c-3c5d-4c4f-b838-fc26db0b5cac-multus-socket-dir-parent\") pod \"multus-zmrmk\" (UID: \"153e979c-3c5d-4c4f-b838-fc26db0b5cac\") " pod="openshift-multus/multus-zmrmk" Oct 31 04:03:34.257938 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.257856 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f7a16817-8ea2-4649-9e42-3d340e1b9703-systemd-units\") pod \"ovnkube-node-zbh9t\" (UID: \"f7a16817-8ea2-4649-9e42-3d340e1b9703\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbh9t" Oct 31 04:03:34.257938 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.257889 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f7a16817-8ea2-4649-9e42-3d340e1b9703-host-cni-bin\") pod \"ovnkube-node-zbh9t\" (UID: \"f7a16817-8ea2-4649-9e42-3d340e1b9703\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbh9t" Oct 31 04:03:34.257938 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.257920 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c581e590-87e5-4cd7-8d84-eabb4ee21fd1-device-dir\") pod \"aws-ebs-csi-driver-node-zw4rh\" (UID: \"c581e590-87e5-4cd7-8d84-eabb4ee21fd1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zw4rh" Oct 31 04:03:34.258200 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.257943 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d4eabffb-abf1-4409-94b5-c0489ab37853-run\") pod \"tuned-b59p9\" (UID: \"d4eabffb-abf1-4409-94b5-c0489ab37853\") " pod="openshift-cluster-node-tuning-operator/tuned-b59p9" Oct 31 04:03:34.258200 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.257970 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/79ae1195-4ff1-46da-b1e2-83042dba6caa-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kmh48\" (UID: \"79ae1195-4ff1-46da-b1e2-83042dba6caa\") " pod="openshift-multus/multus-additional-cni-plugins-kmh48" Oct 31 04:03:34.258200 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.258011 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/153e979c-3c5d-4c4f-b838-fc26db0b5cac-host-var-lib-cni-multus\") pod \"multus-zmrmk\" (UID: \"153e979c-3c5d-4c4f-b838-fc26db0b5cac\") " pod="openshift-multus/multus-zmrmk" Oct 31 04:03:34.258200 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.258036 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/153e979c-3c5d-4c4f-b838-fc26db0b5cac-host-var-lib-kubelet\") pod \"multus-zmrmk\" (UID: \"153e979c-3c5d-4c4f-b838-fc26db0b5cac\") " pod="openshift-multus/multus-zmrmk" Oct 31 04:03:34.258200 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.258073 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2zgx\" (UniqueName: \"kubernetes.io/projected/24e22999-2c81-4779-a901-5b9faa15c1cb-kube-api-access-v2zgx\") pod \"iptables-alerter-6q4xv\" (UID: \"24e22999-2c81-4779-a901-5b9faa15c1cb\") " pod="openshift-network-operator/iptables-alerter-6q4xv" Oct 31 04:03:34.258200 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.258097 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f7a16817-8ea2-4649-9e42-3d340e1b9703-run-systemd\") pod \"ovnkube-node-zbh9t\" (UID: \"f7a16817-8ea2-4649-9e42-3d340e1b9703\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbh9t" Oct 31 04:03:34.258200 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.258125 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f7a16817-8ea2-4649-9e42-3d340e1b9703-var-lib-openvswitch\") pod \"ovnkube-node-zbh9t\" (UID: \"f7a16817-8ea2-4649-9e42-3d340e1b9703\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbh9t" Oct 31 04:03:34.258200 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.258168 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f7a16817-8ea2-4649-9e42-3d340e1b9703-node-log\") pod \"ovnkube-node-zbh9t\" (UID: \"f7a16817-8ea2-4649-9e42-3d340e1b9703\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbh9t" Oct 31 04:03:34.258200 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.258191 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d4eabffb-abf1-4409-94b5-c0489ab37853-etc-sysctl-conf\") pod \"tuned-b59p9\" (UID: \"d4eabffb-abf1-4409-94b5-c0489ab37853\") " pod="openshift-cluster-node-tuning-operator/tuned-b59p9" Oct 31 04:03:34.258559 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.258217 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/79ae1195-4ff1-46da-b1e2-83042dba6caa-cni-binary-copy\") pod \"multus-additional-cni-plugins-kmh48\" (UID: \"79ae1195-4ff1-46da-b1e2-83042dba6caa\") " pod="openshift-multus/multus-additional-cni-plugins-kmh48" Oct 31 04:03:34.258559 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.258247 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/79ae1195-4ff1-46da-b1e2-83042dba6caa-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kmh48\" (UID: \"79ae1195-4ff1-46da-b1e2-83042dba6caa\") " pod="openshift-multus/multus-additional-cni-plugins-kmh48" Oct 31 04:03:34.258559 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.258289 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/153e979c-3c5d-4c4f-b838-fc26db0b5cac-host-run-netns\") pod \"multus-zmrmk\" (UID: \"153e979c-3c5d-4c4f-b838-fc26db0b5cac\") " pod="openshift-multus/multus-zmrmk" Oct 31 04:03:34.258559 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.258312 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/153e979c-3c5d-4c4f-b838-fc26db0b5cac-hostroot\") pod \"multus-zmrmk\" (UID: \"153e979c-3c5d-4c4f-b838-fc26db0b5cac\") " pod="openshift-multus/multus-zmrmk" Oct 31 04:03:34.258559 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.258336 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/24e22999-2c81-4779-a901-5b9faa15c1cb-iptables-alerter-script\") pod \"iptables-alerter-6q4xv\" (UID: \"24e22999-2c81-4779-a901-5b9faa15c1cb\") " pod="openshift-network-operator/iptables-alerter-6q4xv" Oct 31 04:03:34.258559 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.258365 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/24e22999-2c81-4779-a901-5b9faa15c1cb-host-slash\") pod \"iptables-alerter-6q4xv\" (UID: \"24e22999-2c81-4779-a901-5b9faa15c1cb\") " pod="openshift-network-operator/iptables-alerter-6q4xv" Oct 31 04:03:34.258559 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.258391 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d4eabffb-abf1-4409-94b5-c0489ab37853-etc-modprobe-d\") pod \"tuned-b59p9\" (UID: \"d4eabffb-abf1-4409-94b5-c0489ab37853\") " pod="openshift-cluster-node-tuning-operator/tuned-b59p9" Oct 31 04:03:34.258559 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.258430 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f12d4ae1-e673-4a9e-bd43-92d6e9d5a023-agent-certs\") pod \"konnectivity-agent-ntbm9\" (UID: \"f12d4ae1-e673-4a9e-bd43-92d6e9d5a023\") " pod="kube-system/konnectivity-agent-ntbm9" Oct 31 04:03:34.258559 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.258456 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgdpm\" (UniqueName: \"kubernetes.io/projected/79ae1195-4ff1-46da-b1e2-83042dba6caa-kube-api-access-wgdpm\") pod \"multus-additional-cni-plugins-kmh48\" (UID: \"79ae1195-4ff1-46da-b1e2-83042dba6caa\") " pod="openshift-multus/multus-additional-cni-plugins-kmh48" Oct 31 04:03:34.258559 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.258483 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f7a16817-8ea2-4649-9e42-3d340e1b9703-host-cni-netd\") pod \"ovnkube-node-zbh9t\" (UID: \"f7a16817-8ea2-4649-9e42-3d340e1b9703\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbh9t" Oct 31 04:03:34.258559 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.258515 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f7a16817-8ea2-4649-9e42-3d340e1b9703-ovn-node-metrics-cert\") pod \"ovnkube-node-zbh9t\" (UID: \"f7a16817-8ea2-4649-9e42-3d340e1b9703\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbh9t" Oct 31 04:03:34.258559 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.258543 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c581e590-87e5-4cd7-8d84-eabb4ee21fd1-etc-selinux\") pod \"aws-ebs-csi-driver-node-zw4rh\" (UID: \"c581e590-87e5-4cd7-8d84-eabb4ee21fd1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zw4rh" Oct 31 04:03:34.272999 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.272669 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 31 04:03:34.272999 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.272731 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 31 04:03:34.272999 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.272869 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 31 04:03:34.279715 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.273620 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 31 04:03:34.279715 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.273710 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 31 04:03:34.279715 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.273761 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 31 04:03:34.279715 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.273894 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 31 04:03:34.279715 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.273939 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 31 04:03:34.279715 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.274300 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 31 04:03:34.279715 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.274609 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 31 04:03:34.279715 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.275051 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 31 04:03:34.279715 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.275220 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 31 04:03:34.279715 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.279157 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 31 04:03:34.281052 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.280444 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-vc7dp\"" Oct 31 04:03:34.281052 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.280484 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-9gl9k\"" Oct 31 04:03:34.281052 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.280546 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-ldktz\"" Oct 31 04:03:34.281052 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.280670 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-xzg5s\"" Oct 31 04:03:34.281052 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.280853 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-brbhc\"" Oct 31 04:03:34.281052 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.280997 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-45qpm\"" Oct 31 04:03:34.281418 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.281110 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 31 04:03:34.281418 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.281098 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-155.ec2.internal" event={"ID":"db7fb1e6e2bd3665becb16d5fafc3d7e","Type":"ContainerStarted","Data":"8022cc9c6ccbd005a0ecdab2ba41a52a5e4c05e4b51c3c7f012c5cd630f6341b"} Oct 31 04:03:34.282648 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.282626 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-155.ec2.internal" event={"ID":"b3351600635c8d30ebae275373d4b16d","Type":"ContainerStarted","Data":"1ddb915eccc0737c404aca07c5758b96a9f0fdd59c1cf88f059dae0e94b737d9"} Oct 31 04:03:34.359388 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.359358 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/79ae1195-4ff1-46da-b1e2-83042dba6caa-cnibin\") pod \"multus-additional-cni-plugins-kmh48\" (UID: \"79ae1195-4ff1-46da-b1e2-83042dba6caa\") " pod="openshift-multus/multus-additional-cni-plugins-kmh48" Oct 31 04:03:34.359524 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.359398 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/79ae1195-4ff1-46da-b1e2-83042dba6caa-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kmh48\" (UID: \"79ae1195-4ff1-46da-b1e2-83042dba6caa\") " pod="openshift-multus/multus-additional-cni-plugins-kmh48" Oct 31 04:03:34.359524 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.359428 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d4eabffb-abf1-4409-94b5-c0489ab37853-sys\") pod \"tuned-b59p9\" (UID: \"d4eabffb-abf1-4409-94b5-c0489ab37853\") " pod="openshift-cluster-node-tuning-operator/tuned-b59p9" Oct 31 04:03:34.359524 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.359453 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d4eabffb-abf1-4409-94b5-c0489ab37853-host\") pod \"tuned-b59p9\" (UID: \"d4eabffb-abf1-4409-94b5-c0489ab37853\") " pod="openshift-cluster-node-tuning-operator/tuned-b59p9" Oct 31 04:03:34.359524 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.359478 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f7a16817-8ea2-4649-9e42-3d340e1b9703-ovnkube-config\") pod \"ovnkube-node-zbh9t\" (UID: \"f7a16817-8ea2-4649-9e42-3d340e1b9703\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbh9t" Oct 31 04:03:34.359524 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.359500 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f7a16817-8ea2-4649-9e42-3d340e1b9703-env-overrides\") pod \"ovnkube-node-zbh9t\" (UID: \"f7a16817-8ea2-4649-9e42-3d340e1b9703\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbh9t" Oct 31 04:03:34.359524 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.359524 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9r7fx\" (UniqueName: \"kubernetes.io/projected/e7f09238-2bbf-4587-9989-49419d90dabf-kube-api-access-9r7fx\") pod \"network-metrics-daemon-gs42n\" (UID: \"e7f09238-2bbf-4587-9989-49419d90dabf\") " pod="openshift-multus/network-metrics-daemon-gs42n" Oct 31 04:03:34.359794 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.359548 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-786gp\" (UniqueName: \"kubernetes.io/projected/a90157bb-46c0-4afa-82b1-9f6819518744-kube-api-access-786gp\") pod \"node-ca-xmbxc\" (UID: \"a90157bb-46c0-4afa-82b1-9f6819518744\") " pod="openshift-image-registry/node-ca-xmbxc" Oct 31 04:03:34.359794 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.359572 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vp84q\" (UniqueName: \"kubernetes.io/projected/153e979c-3c5d-4c4f-b838-fc26db0b5cac-kube-api-access-vp84q\") pod \"multus-zmrmk\" (UID: \"153e979c-3c5d-4c4f-b838-fc26db0b5cac\") " pod="openshift-multus/multus-zmrmk" Oct 31 04:03:34.359794 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.359602 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f7a16817-8ea2-4649-9e42-3d340e1b9703-host-slash\") pod \"ovnkube-node-zbh9t\" (UID: \"f7a16817-8ea2-4649-9e42-3d340e1b9703\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbh9t" Oct 31 04:03:34.359794 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.359626 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c581e590-87e5-4cd7-8d84-eabb4ee21fd1-socket-dir\") pod \"aws-ebs-csi-driver-node-zw4rh\" (UID: \"c581e590-87e5-4cd7-8d84-eabb4ee21fd1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zw4rh" Oct 31 04:03:34.359794 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.359651 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jq5vs\" (UniqueName: \"kubernetes.io/projected/f70f5eab-445f-430e-bfb1-92e4d620b90d-kube-api-access-jq5vs\") pod \"network-check-target-w9sdm\" (UID: \"f70f5eab-445f-430e-bfb1-92e4d620b90d\") " pod="openshift-network-diagnostics/network-check-target-w9sdm" Oct 31 04:03:34.359794 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.359674 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/79ae1195-4ff1-46da-b1e2-83042dba6caa-system-cni-dir\") pod \"multus-additional-cni-plugins-kmh48\" (UID: \"79ae1195-4ff1-46da-b1e2-83042dba6caa\") " pod="openshift-multus/multus-additional-cni-plugins-kmh48" Oct 31 04:03:34.359794 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.359696 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a90157bb-46c0-4afa-82b1-9f6819518744-serviceca\") pod \"node-ca-xmbxc\" (UID: \"a90157bb-46c0-4afa-82b1-9f6819518744\") " pod="openshift-image-registry/node-ca-xmbxc" Oct 31 04:03:34.359794 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.359719 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/153e979c-3c5d-4c4f-b838-fc26db0b5cac-multus-cni-dir\") pod \"multus-zmrmk\" (UID: \"153e979c-3c5d-4c4f-b838-fc26db0b5cac\") " pod="openshift-multus/multus-zmrmk" Oct 31 04:03:34.359794 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.359742 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/153e979c-3c5d-4c4f-b838-fc26db0b5cac-multus-socket-dir-parent\") pod \"multus-zmrmk\" (UID: \"153e979c-3c5d-4c4f-b838-fc26db0b5cac\") " pod="openshift-multus/multus-zmrmk" Oct 31 04:03:34.359794 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.359767 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f7a16817-8ea2-4649-9e42-3d340e1b9703-systemd-units\") pod \"ovnkube-node-zbh9t\" (UID: \"f7a16817-8ea2-4649-9e42-3d340e1b9703\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbh9t" Oct 31 04:03:34.359794 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.359791 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f7a16817-8ea2-4649-9e42-3d340e1b9703-host-cni-bin\") pod \"ovnkube-node-zbh9t\" (UID: \"f7a16817-8ea2-4649-9e42-3d340e1b9703\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbh9t" Oct 31 04:03:34.360259 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.359816 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c581e590-87e5-4cd7-8d84-eabb4ee21fd1-device-dir\") pod \"aws-ebs-csi-driver-node-zw4rh\" (UID: \"c581e590-87e5-4cd7-8d84-eabb4ee21fd1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zw4rh" Oct 31 04:03:34.360259 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.359838 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d4eabffb-abf1-4409-94b5-c0489ab37853-run\") pod \"tuned-b59p9\" (UID: \"d4eabffb-abf1-4409-94b5-c0489ab37853\") " pod="openshift-cluster-node-tuning-operator/tuned-b59p9" Oct 31 04:03:34.360259 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.359864 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/79ae1195-4ff1-46da-b1e2-83042dba6caa-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kmh48\" (UID: \"79ae1195-4ff1-46da-b1e2-83042dba6caa\") " pod="openshift-multus/multus-additional-cni-plugins-kmh48" Oct 31 04:03:34.360259 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.359891 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/153e979c-3c5d-4c4f-b838-fc26db0b5cac-host-var-lib-cni-multus\") pod \"multus-zmrmk\" (UID: \"153e979c-3c5d-4c4f-b838-fc26db0b5cac\") " pod="openshift-multus/multus-zmrmk" Oct 31 04:03:34.360259 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.359915 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/153e979c-3c5d-4c4f-b838-fc26db0b5cac-host-var-lib-kubelet\") pod \"multus-zmrmk\" (UID: \"153e979c-3c5d-4c4f-b838-fc26db0b5cac\") " pod="openshift-multus/multus-zmrmk" Oct 31 04:03:34.360259 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.359941 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2zgx\" (UniqueName: \"kubernetes.io/projected/24e22999-2c81-4779-a901-5b9faa15c1cb-kube-api-access-v2zgx\") pod \"iptables-alerter-6q4xv\" (UID: \"24e22999-2c81-4779-a901-5b9faa15c1cb\") " pod="openshift-network-operator/iptables-alerter-6q4xv" Oct 31 04:03:34.360259 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.359964 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f7a16817-8ea2-4649-9e42-3d340e1b9703-run-systemd\") pod \"ovnkube-node-zbh9t\" (UID: \"f7a16817-8ea2-4649-9e42-3d340e1b9703\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbh9t" Oct 31 04:03:34.360259 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.359989 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f7a16817-8ea2-4649-9e42-3d340e1b9703-var-lib-openvswitch\") pod \"ovnkube-node-zbh9t\" (UID: \"f7a16817-8ea2-4649-9e42-3d340e1b9703\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbh9t" Oct 31 04:03:34.360259 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.360013 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f7a16817-8ea2-4649-9e42-3d340e1b9703-node-log\") pod \"ovnkube-node-zbh9t\" (UID: \"f7a16817-8ea2-4649-9e42-3d340e1b9703\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbh9t" Oct 31 04:03:34.360259 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.360036 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d4eabffb-abf1-4409-94b5-c0489ab37853-etc-sysctl-conf\") pod \"tuned-b59p9\" (UID: \"d4eabffb-abf1-4409-94b5-c0489ab37853\") " pod="openshift-cluster-node-tuning-operator/tuned-b59p9" Oct 31 04:03:34.360259 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.360060 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/79ae1195-4ff1-46da-b1e2-83042dba6caa-cni-binary-copy\") pod \"multus-additional-cni-plugins-kmh48\" (UID: \"79ae1195-4ff1-46da-b1e2-83042dba6caa\") " pod="openshift-multus/multus-additional-cni-plugins-kmh48" Oct 31 04:03:34.360259 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.360086 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/79ae1195-4ff1-46da-b1e2-83042dba6caa-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kmh48\" (UID: \"79ae1195-4ff1-46da-b1e2-83042dba6caa\") " pod="openshift-multus/multus-additional-cni-plugins-kmh48" Oct 31 04:03:34.360259 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.360111 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/153e979c-3c5d-4c4f-b838-fc26db0b5cac-host-run-netns\") pod \"multus-zmrmk\" (UID: \"153e979c-3c5d-4c4f-b838-fc26db0b5cac\") " pod="openshift-multus/multus-zmrmk" Oct 31 04:03:34.360259 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.360142 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/153e979c-3c5d-4c4f-b838-fc26db0b5cac-hostroot\") pod \"multus-zmrmk\" (UID: \"153e979c-3c5d-4c4f-b838-fc26db0b5cac\") " pod="openshift-multus/multus-zmrmk" Oct 31 04:03:34.360259 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.360167 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/24e22999-2c81-4779-a901-5b9faa15c1cb-iptables-alerter-script\") pod \"iptables-alerter-6q4xv\" (UID: \"24e22999-2c81-4779-a901-5b9faa15c1cb\") " pod="openshift-network-operator/iptables-alerter-6q4xv" Oct 31 04:03:34.360259 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.360193 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/24e22999-2c81-4779-a901-5b9faa15c1cb-host-slash\") pod \"iptables-alerter-6q4xv\" (UID: \"24e22999-2c81-4779-a901-5b9faa15c1cb\") " pod="openshift-network-operator/iptables-alerter-6q4xv" Oct 31 04:03:34.360259 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.360217 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d4eabffb-abf1-4409-94b5-c0489ab37853-etc-modprobe-d\") pod \"tuned-b59p9\" (UID: \"d4eabffb-abf1-4409-94b5-c0489ab37853\") " pod="openshift-cluster-node-tuning-operator/tuned-b59p9" Oct 31 04:03:34.360982 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.360241 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f12d4ae1-e673-4a9e-bd43-92d6e9d5a023-agent-certs\") pod \"konnectivity-agent-ntbm9\" (UID: \"f12d4ae1-e673-4a9e-bd43-92d6e9d5a023\") " pod="kube-system/konnectivity-agent-ntbm9" Oct 31 04:03:34.360982 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.360266 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgdpm\" (UniqueName: \"kubernetes.io/projected/79ae1195-4ff1-46da-b1e2-83042dba6caa-kube-api-access-wgdpm\") pod \"multus-additional-cni-plugins-kmh48\" (UID: \"79ae1195-4ff1-46da-b1e2-83042dba6caa\") " pod="openshift-multus/multus-additional-cni-plugins-kmh48" Oct 31 04:03:34.360982 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.360291 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f7a16817-8ea2-4649-9e42-3d340e1b9703-host-cni-netd\") pod \"ovnkube-node-zbh9t\" (UID: \"f7a16817-8ea2-4649-9e42-3d340e1b9703\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbh9t" Oct 31 04:03:34.360982 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.360314 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f7a16817-8ea2-4649-9e42-3d340e1b9703-ovn-node-metrics-cert\") pod \"ovnkube-node-zbh9t\" (UID: \"f7a16817-8ea2-4649-9e42-3d340e1b9703\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbh9t" Oct 31 04:03:34.360982 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.360341 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c581e590-87e5-4cd7-8d84-eabb4ee21fd1-etc-selinux\") pod \"aws-ebs-csi-driver-node-zw4rh\" (UID: \"c581e590-87e5-4cd7-8d84-eabb4ee21fd1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zw4rh" Oct 31 04:03:34.360982 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.360367 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxgq7\" (UniqueName: \"kubernetes.io/projected/1f2543cc-b528-41be-92e0-9d654405a723-kube-api-access-fxgq7\") pod \"node-resolver-77z7p\" (UID: \"1f2543cc-b528-41be-92e0-9d654405a723\") " pod="openshift-dns/node-resolver-77z7p" Oct 31 04:03:34.360982 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.360393 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f7a16817-8ea2-4649-9e42-3d340e1b9703-ovnkube-script-lib\") pod \"ovnkube-node-zbh9t\" (UID: \"f7a16817-8ea2-4649-9e42-3d340e1b9703\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbh9t" Oct 31 04:03:34.360982 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.360435 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwltl\" (UniqueName: \"kubernetes.io/projected/c581e590-87e5-4cd7-8d84-eabb4ee21fd1-kube-api-access-wwltl\") pod \"aws-ebs-csi-driver-node-zw4rh\" (UID: \"c581e590-87e5-4cd7-8d84-eabb4ee21fd1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zw4rh" Oct 31 04:03:34.360982 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.360461 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bc9d2733-89fa-4079-9dc1-01983df773cb-original-pull-secret\") pod \"global-pull-secret-syncer-tlcrp\" (UID: \"bc9d2733-89fa-4079-9dc1-01983df773cb\") " pod="kube-system/global-pull-secret-syncer-tlcrp" Oct 31 04:03:34.360982 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.360486 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1f2543cc-b528-41be-92e0-9d654405a723-hosts-file\") pod \"node-resolver-77z7p\" (UID: \"1f2543cc-b528-41be-92e0-9d654405a723\") " pod="openshift-dns/node-resolver-77z7p" Oct 31 04:03:34.360982 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.360512 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f7a16817-8ea2-4649-9e42-3d340e1b9703-host-run-netns\") pod \"ovnkube-node-zbh9t\" (UID: \"f7a16817-8ea2-4649-9e42-3d340e1b9703\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbh9t" Oct 31 04:03:34.360982 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.360543 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f7a16817-8ea2-4649-9e42-3d340e1b9703-run-openvswitch\") pod \"ovnkube-node-zbh9t\" (UID: \"f7a16817-8ea2-4649-9e42-3d340e1b9703\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbh9t" Oct 31 04:03:34.360982 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.360571 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f7a16817-8ea2-4649-9e42-3d340e1b9703-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zbh9t\" (UID: \"f7a16817-8ea2-4649-9e42-3d340e1b9703\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbh9t" Oct 31 04:03:34.360982 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.360598 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n54k7\" (UniqueName: \"kubernetes.io/projected/d4eabffb-abf1-4409-94b5-c0489ab37853-kube-api-access-n54k7\") pod \"tuned-b59p9\" (UID: \"d4eabffb-abf1-4409-94b5-c0489ab37853\") " pod="openshift-cluster-node-tuning-operator/tuned-b59p9" Oct 31 04:03:34.360982 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.360625 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/153e979c-3c5d-4c4f-b838-fc26db0b5cac-cnibin\") pod \"multus-zmrmk\" (UID: \"153e979c-3c5d-4c4f-b838-fc26db0b5cac\") " pod="openshift-multus/multus-zmrmk" Oct 31 04:03:34.360982 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.360650 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/153e979c-3c5d-4c4f-b838-fc26db0b5cac-multus-daemon-config\") pod \"multus-zmrmk\" (UID: \"153e979c-3c5d-4c4f-b838-fc26db0b5cac\") " pod="openshift-multus/multus-zmrmk" Oct 31 04:03:34.360982 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.360674 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/153e979c-3c5d-4c4f-b838-fc26db0b5cac-etc-kubernetes\") pod \"multus-zmrmk\" (UID: \"153e979c-3c5d-4c4f-b838-fc26db0b5cac\") " pod="openshift-multus/multus-zmrmk" Oct 31 04:03:34.361597 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.360702 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f7a16817-8ea2-4649-9e42-3d340e1b9703-etc-openvswitch\") pod \"ovnkube-node-zbh9t\" (UID: \"f7a16817-8ea2-4649-9e42-3d340e1b9703\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbh9t" Oct 31 04:03:34.361597 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.360727 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f7a16817-8ea2-4649-9e42-3d340e1b9703-log-socket\") pod \"ovnkube-node-zbh9t\" (UID: \"f7a16817-8ea2-4649-9e42-3d340e1b9703\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbh9t" Oct 31 04:03:34.361597 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.360752 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkkp9\" (UniqueName: \"kubernetes.io/projected/f7a16817-8ea2-4649-9e42-3d340e1b9703-kube-api-access-dkkp9\") pod \"ovnkube-node-zbh9t\" (UID: \"f7a16817-8ea2-4649-9e42-3d340e1b9703\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbh9t" Oct 31 04:03:34.361597 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.360777 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7f09238-2bbf-4587-9989-49419d90dabf-metrics-certs\") pod \"network-metrics-daemon-gs42n\" (UID: \"e7f09238-2bbf-4587-9989-49419d90dabf\") " pod="openshift-multus/network-metrics-daemon-gs42n" Oct 31 04:03:34.361597 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.360801 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f2543cc-b528-41be-92e0-9d654405a723-tmp-dir\") pod \"node-resolver-77z7p\" (UID: \"1f2543cc-b528-41be-92e0-9d654405a723\") " pod="openshift-dns/node-resolver-77z7p" Oct 31 04:03:34.361597 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.360826 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d4eabffb-abf1-4409-94b5-c0489ab37853-etc-tuned\") pod \"tuned-b59p9\" (UID: \"d4eabffb-abf1-4409-94b5-c0489ab37853\") " pod="openshift-cluster-node-tuning-operator/tuned-b59p9" Oct 31 04:03:34.361597 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.360848 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f12d4ae1-e673-4a9e-bd43-92d6e9d5a023-konnectivity-ca\") pod \"konnectivity-agent-ntbm9\" (UID: \"f12d4ae1-e673-4a9e-bd43-92d6e9d5a023\") " pod="kube-system/konnectivity-agent-ntbm9" Oct 31 04:03:34.361597 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.360871 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bc9d2733-89fa-4079-9dc1-01983df773cb-dbus\") pod \"global-pull-secret-syncer-tlcrp\" (UID: \"bc9d2733-89fa-4079-9dc1-01983df773cb\") " pod="kube-system/global-pull-secret-syncer-tlcrp" Oct 31 04:03:34.361597 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.360887 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/153e979c-3c5d-4c4f-b838-fc26db0b5cac-multus-conf-dir\") pod \"multus-zmrmk\" (UID: \"153e979c-3c5d-4c4f-b838-fc26db0b5cac\") " pod="openshift-multus/multus-zmrmk" Oct 31 04:03:34.361597 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.360903 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f7a16817-8ea2-4649-9e42-3d340e1b9703-host-kubelet\") pod \"ovnkube-node-zbh9t\" (UID: \"f7a16817-8ea2-4649-9e42-3d340e1b9703\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbh9t" Oct 31 04:03:34.361597 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.360919 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f7a16817-8ea2-4649-9e42-3d340e1b9703-host-run-ovn-kubernetes\") pod \"ovnkube-node-zbh9t\" (UID: \"f7a16817-8ea2-4649-9e42-3d340e1b9703\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbh9t" Oct 31 04:03:34.361597 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.360937 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c581e590-87e5-4cd7-8d84-eabb4ee21fd1-registration-dir\") pod \"aws-ebs-csi-driver-node-zw4rh\" (UID: \"c581e590-87e5-4cd7-8d84-eabb4ee21fd1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zw4rh" Oct 31 04:03:34.361597 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.360960 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d4eabffb-abf1-4409-94b5-c0489ab37853-var-lib-kubelet\") pod \"tuned-b59p9\" (UID: \"d4eabffb-abf1-4409-94b5-c0489ab37853\") " pod="openshift-cluster-node-tuning-operator/tuned-b59p9" Oct 31 04:03:34.361597 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.360984 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/153e979c-3c5d-4c4f-b838-fc26db0b5cac-cni-binary-copy\") pod \"multus-zmrmk\" (UID: \"153e979c-3c5d-4c4f-b838-fc26db0b5cac\") " pod="openshift-multus/multus-zmrmk" Oct 31 04:03:34.361597 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.361002 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/153e979c-3c5d-4c4f-b838-fc26db0b5cac-host-var-lib-cni-bin\") pod \"multus-zmrmk\" (UID: \"153e979c-3c5d-4c4f-b838-fc26db0b5cac\") " pod="openshift-multus/multus-zmrmk" Oct 31 04:03:34.361597 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.361021 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/153e979c-3c5d-4c4f-b838-fc26db0b5cac-host-run-multus-certs\") pod \"multus-zmrmk\" (UID: \"153e979c-3c5d-4c4f-b838-fc26db0b5cac\") " pod="openshift-multus/multus-zmrmk" Oct 31 04:03:34.361597 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.361049 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f7a16817-8ea2-4649-9e42-3d340e1b9703-run-ovn\") pod \"ovnkube-node-zbh9t\" (UID: \"f7a16817-8ea2-4649-9e42-3d340e1b9703\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbh9t" Oct 31 04:03:34.362203 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.361076 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d4eabffb-abf1-4409-94b5-c0489ab37853-etc-systemd\") pod \"tuned-b59p9\" (UID: \"d4eabffb-abf1-4409-94b5-c0489ab37853\") " pod="openshift-cluster-node-tuning-operator/tuned-b59p9" Oct 31 04:03:34.362203 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.361101 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a90157bb-46c0-4afa-82b1-9f6819518744-host\") pod \"node-ca-xmbxc\" (UID: \"a90157bb-46c0-4afa-82b1-9f6819518744\") " pod="openshift-image-registry/node-ca-xmbxc" Oct 31 04:03:34.362203 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.361122 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c581e590-87e5-4cd7-8d84-eabb4ee21fd1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zw4rh\" (UID: \"c581e590-87e5-4cd7-8d84-eabb4ee21fd1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zw4rh" Oct 31 04:03:34.362203 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.361144 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c581e590-87e5-4cd7-8d84-eabb4ee21fd1-sys-fs\") pod \"aws-ebs-csi-driver-node-zw4rh\" (UID: \"c581e590-87e5-4cd7-8d84-eabb4ee21fd1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zw4rh" Oct 31 04:03:34.362203 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.361164 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d4eabffb-abf1-4409-94b5-c0489ab37853-etc-kubernetes\") pod \"tuned-b59p9\" (UID: \"d4eabffb-abf1-4409-94b5-c0489ab37853\") " pod="openshift-cluster-node-tuning-operator/tuned-b59p9" Oct 31 04:03:34.362203 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.361185 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d4eabffb-abf1-4409-94b5-c0489ab37853-lib-modules\") pod \"tuned-b59p9\" (UID: \"d4eabffb-abf1-4409-94b5-c0489ab37853\") " pod="openshift-cluster-node-tuning-operator/tuned-b59p9" Oct 31 04:03:34.362203 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.361201 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d4eabffb-abf1-4409-94b5-c0489ab37853-tmp\") pod \"tuned-b59p9\" (UID: \"d4eabffb-abf1-4409-94b5-c0489ab37853\") " pod="openshift-cluster-node-tuning-operator/tuned-b59p9" Oct 31 04:03:34.362203 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.361225 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/153e979c-3c5d-4c4f-b838-fc26db0b5cac-system-cni-dir\") pod \"multus-zmrmk\" (UID: \"153e979c-3c5d-4c4f-b838-fc26db0b5cac\") " pod="openshift-multus/multus-zmrmk" Oct 31 04:03:34.362203 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.361244 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/153e979c-3c5d-4c4f-b838-fc26db0b5cac-host-run-k8s-cni-cncf-io\") pod \"multus-zmrmk\" (UID: \"153e979c-3c5d-4c4f-b838-fc26db0b5cac\") " pod="openshift-multus/multus-zmrmk" Oct 31 04:03:34.362203 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.361261 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bc9d2733-89fa-4079-9dc1-01983df773cb-kubelet-config\") pod \"global-pull-secret-syncer-tlcrp\" (UID: \"bc9d2733-89fa-4079-9dc1-01983df773cb\") " pod="kube-system/global-pull-secret-syncer-tlcrp" Oct 31 04:03:34.362203 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.361282 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d4eabffb-abf1-4409-94b5-c0489ab37853-etc-sysconfig\") pod \"tuned-b59p9\" (UID: \"d4eabffb-abf1-4409-94b5-c0489ab37853\") " pod="openshift-cluster-node-tuning-operator/tuned-b59p9" Oct 31 04:03:34.362203 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.361302 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d4eabffb-abf1-4409-94b5-c0489ab37853-etc-sysctl-d\") pod \"tuned-b59p9\" (UID: \"d4eabffb-abf1-4409-94b5-c0489ab37853\") " pod="openshift-cluster-node-tuning-operator/tuned-b59p9" Oct 31 04:03:34.362203 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.361482 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d4eabffb-abf1-4409-94b5-c0489ab37853-etc-sysctl-d\") pod \"tuned-b59p9\" (UID: \"d4eabffb-abf1-4409-94b5-c0489ab37853\") " pod="openshift-cluster-node-tuning-operator/tuned-b59p9" Oct 31 04:03:34.362203 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.361527 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/79ae1195-4ff1-46da-b1e2-83042dba6caa-cnibin\") pod \"multus-additional-cni-plugins-kmh48\" (UID: \"79ae1195-4ff1-46da-b1e2-83042dba6caa\") " pod="openshift-multus/multus-additional-cni-plugins-kmh48" Oct 31 04:03:34.362203 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.361687 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/79ae1195-4ff1-46da-b1e2-83042dba6caa-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kmh48\" (UID: \"79ae1195-4ff1-46da-b1e2-83042dba6caa\") " pod="openshift-multus/multus-additional-cni-plugins-kmh48" Oct 31 04:03:34.362203 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.361739 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d4eabffb-abf1-4409-94b5-c0489ab37853-sys\") pod \"tuned-b59p9\" (UID: \"d4eabffb-abf1-4409-94b5-c0489ab37853\") " pod="openshift-cluster-node-tuning-operator/tuned-b59p9" Oct 31 04:03:34.362203 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.361776 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d4eabffb-abf1-4409-94b5-c0489ab37853-host\") pod \"tuned-b59p9\" (UID: \"d4eabffb-abf1-4409-94b5-c0489ab37853\") " pod="openshift-cluster-node-tuning-operator/tuned-b59p9" Oct 31 04:03:34.362928 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.362266 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f7a16817-8ea2-4649-9e42-3d340e1b9703-ovnkube-config\") pod \"ovnkube-node-zbh9t\" (UID: \"f7a16817-8ea2-4649-9e42-3d340e1b9703\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbh9t" Oct 31 04:03:34.362928 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.362663 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f7a16817-8ea2-4649-9e42-3d340e1b9703-env-overrides\") pod \"ovnkube-node-zbh9t\" (UID: \"f7a16817-8ea2-4649-9e42-3d340e1b9703\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbh9t" Oct 31 04:03:34.362928 ip-10-0-134-155 kubenswrapper[2577]: E1031 04:03:34.362754 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 04:03:34.362928 ip-10-0-134-155 kubenswrapper[2577]: E1031 04:03:34.362828 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc9d2733-89fa-4079-9dc1-01983df773cb-original-pull-secret podName:bc9d2733-89fa-4079-9dc1-01983df773cb nodeName:}" failed. No retries permitted until 2026-10-31 04:03:34.862807878 +0000 UTC m=+3.077860185 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bc9d2733-89fa-4079-9dc1-01983df773cb-original-pull-secret") pod "global-pull-secret-syncer-tlcrp" (UID: "bc9d2733-89fa-4079-9dc1-01983df773cb") : object "kube-system"/"original-pull-secret" not registered Oct 31 04:03:34.362928 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.362892 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1f2543cc-b528-41be-92e0-9d654405a723-hosts-file\") pod \"node-resolver-77z7p\" (UID: \"1f2543cc-b528-41be-92e0-9d654405a723\") " pod="openshift-dns/node-resolver-77z7p" Oct 31 04:03:34.363123 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.362938 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f7a16817-8ea2-4649-9e42-3d340e1b9703-host-run-netns\") pod \"ovnkube-node-zbh9t\" (UID: \"f7a16817-8ea2-4649-9e42-3d340e1b9703\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbh9t" Oct 31 04:03:34.363123 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.362983 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f7a16817-8ea2-4649-9e42-3d340e1b9703-run-openvswitch\") pod \"ovnkube-node-zbh9t\" (UID: \"f7a16817-8ea2-4649-9e42-3d340e1b9703\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbh9t" Oct 31 04:03:34.363123 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.363027 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f7a16817-8ea2-4649-9e42-3d340e1b9703-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zbh9t\" (UID: \"f7a16817-8ea2-4649-9e42-3d340e1b9703\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbh9t" Oct 31 04:03:34.363256 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.363187 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/153e979c-3c5d-4c4f-b838-fc26db0b5cac-cnibin\") pod \"multus-zmrmk\" (UID: \"153e979c-3c5d-4c4f-b838-fc26db0b5cac\") " pod="openshift-multus/multus-zmrmk" Oct 31 04:03:34.363313 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.363257 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f7a16817-8ea2-4649-9e42-3d340e1b9703-host-slash\") pod \"ovnkube-node-zbh9t\" (UID: \"f7a16817-8ea2-4649-9e42-3d340e1b9703\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbh9t" Oct 31 04:03:34.364236 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.363367 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c581e590-87e5-4cd7-8d84-eabb4ee21fd1-socket-dir\") pod \"aws-ebs-csi-driver-node-zw4rh\" (UID: \"c581e590-87e5-4cd7-8d84-eabb4ee21fd1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zw4rh" Oct 31 04:03:34.364236 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.363440 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f7a16817-8ea2-4649-9e42-3d340e1b9703-node-log\") pod \"ovnkube-node-zbh9t\" (UID: \"f7a16817-8ea2-4649-9e42-3d340e1b9703\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbh9t" Oct 31 04:03:34.364236 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.363545 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d4eabffb-abf1-4409-94b5-c0489ab37853-etc-sysctl-conf\") pod \"tuned-b59p9\" (UID: \"d4eabffb-abf1-4409-94b5-c0489ab37853\") " pod="openshift-cluster-node-tuning-operator/tuned-b59p9" Oct 31 04:03:34.364236 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.363755 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/153e979c-3c5d-4c4f-b838-fc26db0b5cac-multus-daemon-config\") pod \"multus-zmrmk\" (UID: \"153e979c-3c5d-4c4f-b838-fc26db0b5cac\") " pod="openshift-multus/multus-zmrmk" Oct 31 04:03:34.364236 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.363953 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/153e979c-3c5d-4c4f-b838-fc26db0b5cac-multus-cni-dir\") pod \"multus-zmrmk\" (UID: \"153e979c-3c5d-4c4f-b838-fc26db0b5cac\") " pod="openshift-multus/multus-zmrmk" Oct 31 04:03:34.364236 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.364017 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/153e979c-3c5d-4c4f-b838-fc26db0b5cac-multus-socket-dir-parent\") pod \"multus-zmrmk\" (UID: \"153e979c-3c5d-4c4f-b838-fc26db0b5cac\") " pod="openshift-multus/multus-zmrmk" Oct 31 04:03:34.364236 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.364064 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/79ae1195-4ff1-46da-b1e2-83042dba6caa-cni-binary-copy\") pod \"multus-additional-cni-plugins-kmh48\" (UID: \"79ae1195-4ff1-46da-b1e2-83042dba6caa\") " pod="openshift-multus/multus-additional-cni-plugins-kmh48" Oct 31 04:03:34.364236 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.364085 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f7a16817-8ea2-4649-9e42-3d340e1b9703-systemd-units\") pod \"ovnkube-node-zbh9t\" (UID: \"f7a16817-8ea2-4649-9e42-3d340e1b9703\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbh9t" Oct 31 04:03:34.364236 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.364130 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f7a16817-8ea2-4649-9e42-3d340e1b9703-host-cni-bin\") pod \"ovnkube-node-zbh9t\" (UID: \"f7a16817-8ea2-4649-9e42-3d340e1b9703\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbh9t" Oct 31 04:03:34.364236 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.364165 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c581e590-87e5-4cd7-8d84-eabb4ee21fd1-device-dir\") pod \"aws-ebs-csi-driver-node-zw4rh\" (UID: \"c581e590-87e5-4cd7-8d84-eabb4ee21fd1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zw4rh" Oct 31 04:03:34.364236 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.364171 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a90157bb-46c0-4afa-82b1-9f6819518744-serviceca\") pod \"node-ca-xmbxc\" (UID: \"a90157bb-46c0-4afa-82b1-9f6819518744\") " pod="openshift-image-registry/node-ca-xmbxc" Oct 31 04:03:34.364726 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.364257 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d4eabffb-abf1-4409-94b5-c0489ab37853-run\") pod \"tuned-b59p9\" (UID: \"d4eabffb-abf1-4409-94b5-c0489ab37853\") " pod="openshift-cluster-node-tuning-operator/tuned-b59p9" Oct 31 04:03:34.364726 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.364307 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/153e979c-3c5d-4c4f-b838-fc26db0b5cac-etc-kubernetes\") pod \"multus-zmrmk\" (UID: \"153e979c-3c5d-4c4f-b838-fc26db0b5cac\") " pod="openshift-multus/multus-zmrmk" Oct 31 04:03:34.364726 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.364353 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f7a16817-8ea2-4649-9e42-3d340e1b9703-etc-openvswitch\") pod \"ovnkube-node-zbh9t\" (UID: \"f7a16817-8ea2-4649-9e42-3d340e1b9703\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbh9t" Oct 31 04:03:34.364726 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.364396 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f7a16817-8ea2-4649-9e42-3d340e1b9703-log-socket\") pod \"ovnkube-node-zbh9t\" (UID: \"f7a16817-8ea2-4649-9e42-3d340e1b9703\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbh9t" Oct 31 04:03:34.364726 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.364572 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/79ae1195-4ff1-46da-b1e2-83042dba6caa-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kmh48\" (UID: \"79ae1195-4ff1-46da-b1e2-83042dba6caa\") " pod="openshift-multus/multus-additional-cni-plugins-kmh48" Oct 31 04:03:34.364726 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.364628 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/153e979c-3c5d-4c4f-b838-fc26db0b5cac-host-run-netns\") pod \"multus-zmrmk\" (UID: \"153e979c-3c5d-4c4f-b838-fc26db0b5cac\") " pod="openshift-multus/multus-zmrmk" Oct 31 04:03:34.364726 ip-10-0-134-155 kubenswrapper[2577]: E1031 04:03:34.364697 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 04:03:34.364996 ip-10-0-134-155 kubenswrapper[2577]: E1031 04:03:34.364741 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7f09238-2bbf-4587-9989-49419d90dabf-metrics-certs podName:e7f09238-2bbf-4587-9989-49419d90dabf nodeName:}" failed. No retries permitted until 2026-10-31 04:03:34.864725737 +0000 UTC m=+3.079778036 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e7f09238-2bbf-4587-9989-49419d90dabf-metrics-certs") pod "network-metrics-daemon-gs42n" (UID: "e7f09238-2bbf-4587-9989-49419d90dabf") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 04:03:34.364996 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.364795 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/153e979c-3c5d-4c4f-b838-fc26db0b5cac-host-var-lib-cni-multus\") pod \"multus-zmrmk\" (UID: \"153e979c-3c5d-4c4f-b838-fc26db0b5cac\") " pod="openshift-multus/multus-zmrmk" Oct 31 04:03:34.364996 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.364858 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f7a16817-8ea2-4649-9e42-3d340e1b9703-run-systemd\") pod \"ovnkube-node-zbh9t\" (UID: \"f7a16817-8ea2-4649-9e42-3d340e1b9703\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbh9t" Oct 31 04:03:34.364996 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.364930 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/153e979c-3c5d-4c4f-b838-fc26db0b5cac-host-var-lib-kubelet\") pod \"multus-zmrmk\" (UID: \"153e979c-3c5d-4c4f-b838-fc26db0b5cac\") " pod="openshift-multus/multus-zmrmk" Oct 31 04:03:34.364996 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.364934 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f7a16817-8ea2-4649-9e42-3d340e1b9703-var-lib-openvswitch\") pod \"ovnkube-node-zbh9t\" (UID: \"f7a16817-8ea2-4649-9e42-3d340e1b9703\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbh9t" Oct 31 04:03:34.364996 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.364957 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c581e590-87e5-4cd7-8d84-eabb4ee21fd1-registration-dir\") pod \"aws-ebs-csi-driver-node-zw4rh\" (UID: \"c581e590-87e5-4cd7-8d84-eabb4ee21fd1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zw4rh" Oct 31 04:03:34.364996 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.364991 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f7a16817-8ea2-4649-9e42-3d340e1b9703-host-cni-netd\") pod \"ovnkube-node-zbh9t\" (UID: \"f7a16817-8ea2-4649-9e42-3d340e1b9703\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbh9t" Oct 31 04:03:34.365262 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.365004 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/24e22999-2c81-4779-a901-5b9faa15c1cb-host-slash\") pod \"iptables-alerter-6q4xv\" (UID: \"24e22999-2c81-4779-a901-5b9faa15c1cb\") " pod="openshift-network-operator/iptables-alerter-6q4xv" Oct 31 04:03:34.365262 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.365016 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f7a16817-8ea2-4649-9e42-3d340e1b9703-host-kubelet\") pod \"ovnkube-node-zbh9t\" (UID: \"f7a16817-8ea2-4649-9e42-3d340e1b9703\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbh9t" Oct 31 04:03:34.365262 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.365053 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bc9d2733-89fa-4079-9dc1-01983df773cb-dbus\") pod \"global-pull-secret-syncer-tlcrp\" (UID: \"bc9d2733-89fa-4079-9dc1-01983df773cb\") " pod="kube-system/global-pull-secret-syncer-tlcrp" Oct 31 04:03:34.365262 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.365060 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f7a16817-8ea2-4649-9e42-3d340e1b9703-host-run-ovn-kubernetes\") pod \"ovnkube-node-zbh9t\" (UID: \"f7a16817-8ea2-4649-9e42-3d340e1b9703\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbh9t" Oct 31 04:03:34.365262 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.365106 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d4eabffb-abf1-4409-94b5-c0489ab37853-etc-modprobe-d\") pod \"tuned-b59p9\" (UID: \"d4eabffb-abf1-4409-94b5-c0489ab37853\") " pod="openshift-cluster-node-tuning-operator/tuned-b59p9" Oct 31 04:03:34.365262 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.365110 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/153e979c-3c5d-4c4f-b838-fc26db0b5cac-multus-conf-dir\") pod \"multus-zmrmk\" (UID: \"153e979c-3c5d-4c4f-b838-fc26db0b5cac\") " pod="openshift-multus/multus-zmrmk" Oct 31 04:03:34.365262 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.365248 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/79ae1195-4ff1-46da-b1e2-83042dba6caa-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kmh48\" (UID: \"79ae1195-4ff1-46da-b1e2-83042dba6caa\") " pod="openshift-multus/multus-additional-cni-plugins-kmh48" Oct 31 04:03:34.365592 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.365440 2577 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 31 04:03:34.365592 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.365495 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bc9d2733-89fa-4079-9dc1-01983df773cb-kubelet-config\") pod \"global-pull-secret-syncer-tlcrp\" (UID: \"bc9d2733-89fa-4079-9dc1-01983df773cb\") " pod="kube-system/global-pull-secret-syncer-tlcrp" Oct 31 04:03:34.365592 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.365509 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c581e590-87e5-4cd7-8d84-eabb4ee21fd1-etc-selinux\") pod \"aws-ebs-csi-driver-node-zw4rh\" (UID: \"c581e590-87e5-4cd7-8d84-eabb4ee21fd1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zw4rh" Oct 31 04:03:34.365788 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.365595 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/153e979c-3c5d-4c4f-b838-fc26db0b5cac-system-cni-dir\") pod \"multus-zmrmk\" (UID: \"153e979c-3c5d-4c4f-b838-fc26db0b5cac\") " pod="openshift-multus/multus-zmrmk" Oct 31 04:03:34.365788 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.365633 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d4eabffb-abf1-4409-94b5-c0489ab37853-etc-kubernetes\") pod \"tuned-b59p9\" (UID: \"d4eabffb-abf1-4409-94b5-c0489ab37853\") " pod="openshift-cluster-node-tuning-operator/tuned-b59p9" Oct 31 04:03:34.365788 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.365642 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/153e979c-3c5d-4c4f-b838-fc26db0b5cac-host-run-k8s-cni-cncf-io\") pod \"multus-zmrmk\" (UID: \"153e979c-3c5d-4c4f-b838-fc26db0b5cac\") " pod="openshift-multus/multus-zmrmk" Oct 31 04:03:34.365788 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.365683 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a90157bb-46c0-4afa-82b1-9f6819518744-host\") pod \"node-ca-xmbxc\" (UID: \"a90157bb-46c0-4afa-82b1-9f6819518744\") " pod="openshift-image-registry/node-ca-xmbxc" Oct 31 04:03:34.365788 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.365684 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f12d4ae1-e673-4a9e-bd43-92d6e9d5a023-konnectivity-ca\") pod \"konnectivity-agent-ntbm9\" (UID: \"f12d4ae1-e673-4a9e-bd43-92d6e9d5a023\") " pod="kube-system/konnectivity-agent-ntbm9" Oct 31 04:03:34.365788 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.365734 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/24e22999-2c81-4779-a901-5b9faa15c1cb-iptables-alerter-script\") pod \"iptables-alerter-6q4xv\" (UID: \"24e22999-2c81-4779-a901-5b9faa15c1cb\") " pod="openshift-network-operator/iptables-alerter-6q4xv" Oct 31 04:03:34.365788 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.365759 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c581e590-87e5-4cd7-8d84-eabb4ee21fd1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zw4rh\" (UID: \"c581e590-87e5-4cd7-8d84-eabb4ee21fd1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zw4rh" Oct 31 04:03:34.365788 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.365788 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/153e979c-3c5d-4c4f-b838-fc26db0b5cac-hostroot\") pod \"multus-zmrmk\" (UID: \"153e979c-3c5d-4c4f-b838-fc26db0b5cac\") " pod="openshift-multus/multus-zmrmk" Oct 31 04:03:34.366116 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.365797 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d4eabffb-abf1-4409-94b5-c0489ab37853-lib-modules\") pod \"tuned-b59p9\" (UID: \"d4eabffb-abf1-4409-94b5-c0489ab37853\") " pod="openshift-cluster-node-tuning-operator/tuned-b59p9" Oct 31 04:03:34.366116 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.365806 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d4eabffb-abf1-4409-94b5-c0489ab37853-etc-systemd\") pod \"tuned-b59p9\" (UID: \"d4eabffb-abf1-4409-94b5-c0489ab37853\") " pod="openshift-cluster-node-tuning-operator/tuned-b59p9" Oct 31 04:03:34.366116 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.365842 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/153e979c-3c5d-4c4f-b838-fc26db0b5cac-host-var-lib-cni-bin\") pod \"multus-zmrmk\" (UID: \"153e979c-3c5d-4c4f-b838-fc26db0b5cac\") " pod="openshift-multus/multus-zmrmk" Oct 31 04:03:34.366116 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.365874 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d4eabffb-abf1-4409-94b5-c0489ab37853-var-lib-kubelet\") pod \"tuned-b59p9\" (UID: \"d4eabffb-abf1-4409-94b5-c0489ab37853\") " pod="openshift-cluster-node-tuning-operator/tuned-b59p9" Oct 31 04:03:34.366116 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.365881 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/153e979c-3c5d-4c4f-b838-fc26db0b5cac-host-run-multus-certs\") pod \"multus-zmrmk\" (UID: \"153e979c-3c5d-4c4f-b838-fc26db0b5cac\") " pod="openshift-multus/multus-zmrmk" Oct 31 04:03:34.366116 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.365893 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f7a16817-8ea2-4649-9e42-3d340e1b9703-ovnkube-script-lib\") pod \"ovnkube-node-zbh9t\" (UID: \"f7a16817-8ea2-4649-9e42-3d340e1b9703\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbh9t" Oct 31 04:03:34.366116 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.365947 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/79ae1195-4ff1-46da-b1e2-83042dba6caa-system-cni-dir\") pod \"multus-additional-cni-plugins-kmh48\" (UID: \"79ae1195-4ff1-46da-b1e2-83042dba6caa\") " pod="openshift-multus/multus-additional-cni-plugins-kmh48" Oct 31 04:03:34.366116 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.365960 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d4eabffb-abf1-4409-94b5-c0489ab37853-etc-sysconfig\") pod \"tuned-b59p9\" (UID: \"d4eabffb-abf1-4409-94b5-c0489ab37853\") " pod="openshift-cluster-node-tuning-operator/tuned-b59p9" Oct 31 04:03:34.366116 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.365975 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f7a16817-8ea2-4649-9e42-3d340e1b9703-run-ovn\") pod \"ovnkube-node-zbh9t\" (UID: \"f7a16817-8ea2-4649-9e42-3d340e1b9703\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbh9t" Oct 31 04:03:34.366116 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.366073 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c581e590-87e5-4cd7-8d84-eabb4ee21fd1-sys-fs\") pod \"aws-ebs-csi-driver-node-zw4rh\" (UID: \"c581e590-87e5-4cd7-8d84-eabb4ee21fd1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zw4rh" Oct 31 04:03:34.366561 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.366360 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f2543cc-b528-41be-92e0-9d654405a723-tmp-dir\") pod \"node-resolver-77z7p\" (UID: \"1f2543cc-b528-41be-92e0-9d654405a723\") " pod="openshift-dns/node-resolver-77z7p" Oct 31 04:03:34.366634 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.366614 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/153e979c-3c5d-4c4f-b838-fc26db0b5cac-cni-binary-copy\") pod \"multus-zmrmk\" (UID: \"153e979c-3c5d-4c4f-b838-fc26db0b5cac\") " pod="openshift-multus/multus-zmrmk" Oct 31 04:03:34.369625 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.369602 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d4eabffb-abf1-4409-94b5-c0489ab37853-tmp\") pod \"tuned-b59p9\" (UID: \"d4eabffb-abf1-4409-94b5-c0489ab37853\") " pod="openshift-cluster-node-tuning-operator/tuned-b59p9" Oct 31 04:03:34.369756 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.369736 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f7a16817-8ea2-4649-9e42-3d340e1b9703-ovn-node-metrics-cert\") pod \"ovnkube-node-zbh9t\" (UID: \"f7a16817-8ea2-4649-9e42-3d340e1b9703\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbh9t" Oct 31 04:03:34.370482 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.370459 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d4eabffb-abf1-4409-94b5-c0489ab37853-etc-tuned\") pod \"tuned-b59p9\" (UID: \"d4eabffb-abf1-4409-94b5-c0489ab37853\") " pod="openshift-cluster-node-tuning-operator/tuned-b59p9" Oct 31 04:03:34.370738 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.370713 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f12d4ae1-e673-4a9e-bd43-92d6e9d5a023-agent-certs\") pod \"konnectivity-agent-ntbm9\" (UID: \"f12d4ae1-e673-4a9e-bd43-92d6e9d5a023\") " pod="kube-system/konnectivity-agent-ntbm9" Oct 31 04:03:34.385804 ip-10-0-134-155 kubenswrapper[2577]: E1031 04:03:34.385777 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 04:03:34.385907 ip-10-0-134-155 kubenswrapper[2577]: E1031 04:03:34.385812 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 04:03:34.385907 ip-10-0-134-155 kubenswrapper[2577]: E1031 04:03:34.385828 2577 projected.go:196] Error preparing data for projected volume kube-api-access-jq5vs for pod openshift-network-diagnostics/network-check-target-w9sdm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 04:03:34.385907 ip-10-0-134-155 kubenswrapper[2577]: E1031 04:03:34.385899 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f70f5eab-445f-430e-bfb1-92e4d620b90d-kube-api-access-jq5vs podName:f70f5eab-445f-430e-bfb1-92e4d620b90d nodeName:}" failed. No retries permitted until 2026-10-31 04:03:34.885882626 +0000 UTC m=+3.100934935 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-jq5vs" (UniqueName: "kubernetes.io/projected/f70f5eab-445f-430e-bfb1-92e4d620b90d-kube-api-access-jq5vs") pod "network-check-target-w9sdm" (UID: "f70f5eab-445f-430e-bfb1-92e4d620b90d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 04:03:34.387822 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.387793 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9r7fx\" (UniqueName: \"kubernetes.io/projected/e7f09238-2bbf-4587-9989-49419d90dabf-kube-api-access-9r7fx\") pod \"network-metrics-daemon-gs42n\" (UID: \"e7f09238-2bbf-4587-9989-49419d90dabf\") " pod="openshift-multus/network-metrics-daemon-gs42n" Oct 31 04:03:34.388481 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.388456 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-786gp\" (UniqueName: \"kubernetes.io/projected/a90157bb-46c0-4afa-82b1-9f6819518744-kube-api-access-786gp\") pod \"node-ca-xmbxc\" (UID: \"a90157bb-46c0-4afa-82b1-9f6819518744\") " pod="openshift-image-registry/node-ca-xmbxc" Oct 31 04:03:34.390025 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.390002 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkkp9\" (UniqueName: \"kubernetes.io/projected/f7a16817-8ea2-4649-9e42-3d340e1b9703-kube-api-access-dkkp9\") pod \"ovnkube-node-zbh9t\" (UID: \"f7a16817-8ea2-4649-9e42-3d340e1b9703\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbh9t" Oct 31 04:03:34.391506 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.391484 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n54k7\" (UniqueName: \"kubernetes.io/projected/d4eabffb-abf1-4409-94b5-c0489ab37853-kube-api-access-n54k7\") pod \"tuned-b59p9\" (UID: \"d4eabffb-abf1-4409-94b5-c0489ab37853\") " pod="openshift-cluster-node-tuning-operator/tuned-b59p9" Oct 31 04:03:34.391636 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.391492 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vp84q\" (UniqueName: \"kubernetes.io/projected/153e979c-3c5d-4c4f-b838-fc26db0b5cac-kube-api-access-vp84q\") pod \"multus-zmrmk\" (UID: \"153e979c-3c5d-4c4f-b838-fc26db0b5cac\") " pod="openshift-multus/multus-zmrmk" Oct 31 04:03:34.392357 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.391704 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2zgx\" (UniqueName: \"kubernetes.io/projected/24e22999-2c81-4779-a901-5b9faa15c1cb-kube-api-access-v2zgx\") pod \"iptables-alerter-6q4xv\" (UID: \"24e22999-2c81-4779-a901-5b9faa15c1cb\") " pod="openshift-network-operator/iptables-alerter-6q4xv" Oct 31 04:03:34.392357 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.391892 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgdpm\" (UniqueName: \"kubernetes.io/projected/79ae1195-4ff1-46da-b1e2-83042dba6caa-kube-api-access-wgdpm\") pod \"multus-additional-cni-plugins-kmh48\" (UID: \"79ae1195-4ff1-46da-b1e2-83042dba6caa\") " pod="openshift-multus/multus-additional-cni-plugins-kmh48" Oct 31 04:03:34.392576 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.392534 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxgq7\" (UniqueName: \"kubernetes.io/projected/1f2543cc-b528-41be-92e0-9d654405a723-kube-api-access-fxgq7\") pod \"node-resolver-77z7p\" (UID: \"1f2543cc-b528-41be-92e0-9d654405a723\") " pod="openshift-dns/node-resolver-77z7p" Oct 31 04:03:34.396017 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.395990 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwltl\" (UniqueName: \"kubernetes.io/projected/c581e590-87e5-4cd7-8d84-eabb4ee21fd1-kube-api-access-wwltl\") pod \"aws-ebs-csi-driver-node-zw4rh\" (UID: \"c581e590-87e5-4cd7-8d84-eabb4ee21fd1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zw4rh" Oct 31 04:03:34.485679 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.485648 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-zbh9t" Oct 31 04:03:34.490654 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.490631 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zw4rh" Oct 31 04:03:34.495797 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.495775 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-xmbxc" Oct 31 04:03:34.502610 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:34.502589 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7a16817_8ea2_4649_9e42_3d340e1b9703.slice/crio-dfb45165dd6714f2e2e521be1f2c11d492273d10525ed8974ea1c0601f249a0e WatchSource:0}: Error finding container dfb45165dd6714f2e2e521be1f2c11d492273d10525ed8974ea1c0601f249a0e: Status 404 returned error can't find the container with id dfb45165dd6714f2e2e521be1f2c11d492273d10525ed8974ea1c0601f249a0e Oct 31 04:03:34.505835 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.505816 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-b59p9" Oct 31 04:03:34.506148 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:34.506124 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc581e590_87e5_4cd7_8d84_eabb4ee21fd1.slice/crio-cdec06bf47f6a54fe3f2dd0b5ffb405a4dad6a7395313767f517d814dd709cc5 WatchSource:0}: Error finding container cdec06bf47f6a54fe3f2dd0b5ffb405a4dad6a7395313767f517d814dd709cc5: Status 404 returned error can't find the container with id cdec06bf47f6a54fe3f2dd0b5ffb405a4dad6a7395313767f517d814dd709cc5 Oct 31 04:03:34.508839 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.508811 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-ntbm9" Oct 31 04:03:34.509244 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:34.509223 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda90157bb_46c0_4afa_82b1_9f6819518744.slice/crio-5f9774c0d0ed14f1f2a9fe916e5c2762491290e9c194e57e3d8d9dd40b8ed118 WatchSource:0}: Error finding container 5f9774c0d0ed14f1f2a9fe916e5c2762491290e9c194e57e3d8d9dd40b8ed118: Status 404 returned error can't find the container with id 5f9774c0d0ed14f1f2a9fe916e5c2762491290e9c194e57e3d8d9dd40b8ed118 Oct 31 04:03:34.514014 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.513993 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-kmh48" Oct 31 04:03:34.516516 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.516493 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-zmrmk" Oct 31 04:03:34.518695 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:34.518671 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4eabffb_abf1_4409_94b5_c0489ab37853.slice/crio-e6156483b2bd587a6ca8d006729ce73196b8e19f995598da467ed8a38ae074cd WatchSource:0}: Error finding container e6156483b2bd587a6ca8d006729ce73196b8e19f995598da467ed8a38ae074cd: Status 404 returned error can't find the container with id e6156483b2bd587a6ca8d006729ce73196b8e19f995598da467ed8a38ae074cd Oct 31 04:03:34.521724 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.521701 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-77z7p" Oct 31 04:03:34.527831 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:34.527795 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79ae1195_4ff1_46da_b1e2_83042dba6caa.slice/crio-f16830d905f69c2c30a7e8fef47baa94b6a2db1b4880d62ab6f0b7895f17e68e WatchSource:0}: Error finding container f16830d905f69c2c30a7e8fef47baa94b6a2db1b4880d62ab6f0b7895f17e68e: Status 404 returned error can't find the container with id f16830d905f69c2c30a7e8fef47baa94b6a2db1b4880d62ab6f0b7895f17e68e Oct 31 04:03:34.528559 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.528540 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-6q4xv" Oct 31 04:03:34.529256 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:34.529191 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod153e979c_3c5d_4c4f_b838_fc26db0b5cac.slice/crio-ea0ff4d5abf545a0400b791d9d5625dc627feb17b9582ee4ecc29c8827acf93a WatchSource:0}: Error finding container ea0ff4d5abf545a0400b791d9d5625dc627feb17b9582ee4ecc29c8827acf93a: Status 404 returned error can't find the container with id ea0ff4d5abf545a0400b791d9d5625dc627feb17b9582ee4ecc29c8827acf93a Oct 31 04:03:34.538670 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:34.538638 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f2543cc_b528_41be_92e0_9d654405a723.slice/crio-a69bc663b12a32d16dcb8255d451bb0ace44a40d73d2144fdd60c44368664668 WatchSource:0}: Error finding container a69bc663b12a32d16dcb8255d451bb0ace44a40d73d2144fdd60c44368664668: Status 404 returned error can't find the container with id a69bc663b12a32d16dcb8255d451bb0ace44a40d73d2144fdd60c44368664668 Oct 31 04:03:34.541920 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:03:34.541896 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24e22999_2c81_4779_a901_5b9faa15c1cb.slice/crio-f66a0ba4c2917e14250e724b408ee86370622c0619ec3bd5ab230906b66c353a WatchSource:0}: Error finding container f66a0ba4c2917e14250e724b408ee86370622c0619ec3bd5ab230906b66c353a: Status 404 returned error can't find the container with id f66a0ba4c2917e14250e724b408ee86370622c0619ec3bd5ab230906b66c353a Oct 31 04:03:34.865568 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.865533 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bc9d2733-89fa-4079-9dc1-01983df773cb-original-pull-secret\") pod \"global-pull-secret-syncer-tlcrp\" (UID: \"bc9d2733-89fa-4079-9dc1-01983df773cb\") " pod="kube-system/global-pull-secret-syncer-tlcrp" Oct 31 04:03:34.865730 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.865576 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7f09238-2bbf-4587-9989-49419d90dabf-metrics-certs\") pod \"network-metrics-daemon-gs42n\" (UID: \"e7f09238-2bbf-4587-9989-49419d90dabf\") " pod="openshift-multus/network-metrics-daemon-gs42n" Oct 31 04:03:34.865730 ip-10-0-134-155 kubenswrapper[2577]: E1031 04:03:34.865692 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 04:03:34.865730 ip-10-0-134-155 kubenswrapper[2577]: E1031 04:03:34.865710 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 04:03:34.865887 ip-10-0-134-155 kubenswrapper[2577]: E1031 04:03:34.865760 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc9d2733-89fa-4079-9dc1-01983df773cb-original-pull-secret podName:bc9d2733-89fa-4079-9dc1-01983df773cb nodeName:}" failed. No retries permitted until 2026-10-31 04:03:35.865744103 +0000 UTC m=+4.080796389 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bc9d2733-89fa-4079-9dc1-01983df773cb-original-pull-secret") pod "global-pull-secret-syncer-tlcrp" (UID: "bc9d2733-89fa-4079-9dc1-01983df773cb") : object "kube-system"/"original-pull-secret" not registered Oct 31 04:03:34.865887 ip-10-0-134-155 kubenswrapper[2577]: E1031 04:03:34.865785 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7f09238-2bbf-4587-9989-49419d90dabf-metrics-certs podName:e7f09238-2bbf-4587-9989-49419d90dabf nodeName:}" failed. No retries permitted until 2026-10-31 04:03:35.865776842 +0000 UTC m=+4.080829127 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e7f09238-2bbf-4587-9989-49419d90dabf-metrics-certs") pod "network-metrics-daemon-gs42n" (UID: "e7f09238-2bbf-4587-9989-49419d90dabf") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 04:03:34.966508 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:34.966485 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jq5vs\" (UniqueName: \"kubernetes.io/projected/f70f5eab-445f-430e-bfb1-92e4d620b90d-kube-api-access-jq5vs\") pod \"network-check-target-w9sdm\" (UID: \"f70f5eab-445f-430e-bfb1-92e4d620b90d\") " pod="openshift-network-diagnostics/network-check-target-w9sdm" Oct 31 04:03:34.966618 ip-10-0-134-155 kubenswrapper[2577]: E1031 04:03:34.966603 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 04:03:34.966673 ip-10-0-134-155 kubenswrapper[2577]: E1031 04:03:34.966625 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 04:03:34.966673 ip-10-0-134-155 kubenswrapper[2577]: E1031 04:03:34.966635 2577 projected.go:196] Error preparing data for projected volume kube-api-access-jq5vs for pod openshift-network-diagnostics/network-check-target-w9sdm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 04:03:34.966673 ip-10-0-134-155 kubenswrapper[2577]: E1031 04:03:34.966673 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f70f5eab-445f-430e-bfb1-92e4d620b90d-kube-api-access-jq5vs podName:f70f5eab-445f-430e-bfb1-92e4d620b90d nodeName:}" failed. No retries permitted until 2026-10-31 04:03:35.966661387 +0000 UTC m=+4.181713674 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-jq5vs" (UniqueName: "kubernetes.io/projected/f70f5eab-445f-430e-bfb1-92e4d620b90d-kube-api-access-jq5vs") pod "network-check-target-w9sdm" (UID: "f70f5eab-445f-430e-bfb1-92e4d620b90d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 04:03:35.208554 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:35.208206 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 03:58:34 +0000 UTC" deadline="2028-06-05 23:00:10.46510755 +0000 UTC" Oct 31 04:03:35.208554 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:35.208490 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14010h56m35.256624011s" Oct 31 04:03:35.296541 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:35.295956 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-155.ec2.internal" event={"ID":"db7fb1e6e2bd3665becb16d5fafc3d7e","Type":"ContainerStarted","Data":"5a30c74cd15688f629eff52dc6b13ac010fb54e444f607c992e06f0fa1f8fc39"} Oct 31 04:03:35.299327 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:35.299257 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-zmrmk" event={"ID":"153e979c-3c5d-4c4f-b838-fc26db0b5cac","Type":"ContainerStarted","Data":"ea0ff4d5abf545a0400b791d9d5625dc627feb17b9582ee4ecc29c8827acf93a"} Oct 31 04:03:35.302375 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:35.302286 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kmh48" event={"ID":"79ae1195-4ff1-46da-b1e2-83042dba6caa","Type":"ContainerStarted","Data":"f16830d905f69c2c30a7e8fef47baa94b6a2db1b4880d62ab6f0b7895f17e68e"} Oct 31 04:03:35.306201 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:35.306124 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-ntbm9" event={"ID":"f12d4ae1-e673-4a9e-bd43-92d6e9d5a023","Type":"ContainerStarted","Data":"a2f924ac02df888acb1b969d58be1f782b5eae0d396b10c804368d942460fcd5"} Oct 31 04:03:35.313122 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:35.313062 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-b59p9" event={"ID":"d4eabffb-abf1-4409-94b5-c0489ab37853","Type":"ContainerStarted","Data":"e6156483b2bd587a6ca8d006729ce73196b8e19f995598da467ed8a38ae074cd"} Oct 31 04:03:35.318274 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:35.318206 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-xmbxc" event={"ID":"a90157bb-46c0-4afa-82b1-9f6819518744","Type":"ContainerStarted","Data":"5f9774c0d0ed14f1f2a9fe916e5c2762491290e9c194e57e3d8d9dd40b8ed118"} Oct 31 04:03:35.320956 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:35.320850 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zbh9t" event={"ID":"f7a16817-8ea2-4649-9e42-3d340e1b9703","Type":"ContainerStarted","Data":"dfb45165dd6714f2e2e521be1f2c11d492273d10525ed8974ea1c0601f249a0e"} Oct 31 04:03:35.335357 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:35.334463 2577 generic.go:356] "Generic (PLEG): container finished" podID="b3351600635c8d30ebae275373d4b16d" containerID="08120e1a540caf7fcb57f78e1a171909962020370b9a57ef9e4cb786e45ceee3" exitCode=0 Oct 31 04:03:35.335357 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:35.334539 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-155.ec2.internal" event={"ID":"b3351600635c8d30ebae275373d4b16d","Type":"ContainerDied","Data":"08120e1a540caf7fcb57f78e1a171909962020370b9a57ef9e4cb786e45ceee3"} Oct 31 04:03:35.350075 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:35.350049 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-6q4xv" event={"ID":"24e22999-2c81-4779-a901-5b9faa15c1cb","Type":"ContainerStarted","Data":"f66a0ba4c2917e14250e724b408ee86370622c0619ec3bd5ab230906b66c353a"} Oct 31 04:03:35.358577 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:35.358483 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-77z7p" event={"ID":"1f2543cc-b528-41be-92e0-9d654405a723","Type":"ContainerStarted","Data":"a69bc663b12a32d16dcb8255d451bb0ace44a40d73d2144fdd60c44368664668"} Oct 31 04:03:35.363132 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:35.363054 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zw4rh" event={"ID":"c581e590-87e5-4cd7-8d84-eabb4ee21fd1","Type":"ContainerStarted","Data":"cdec06bf47f6a54fe3f2dd0b5ffb405a4dad6a7395313767f517d814dd709cc5"} Oct 31 04:03:35.379769 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:35.379084 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-155.ec2.internal" podStartSLOduration=1.379067615 podStartE2EDuration="1.379067615s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 04:03:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:03:35.337664505 +0000 UTC m=+3.552716816" watchObservedRunningTime="2026-10-31 04:03:35.379067615 +0000 UTC m=+3.594119925" Oct 31 04:03:35.873262 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:35.873195 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7f09238-2bbf-4587-9989-49419d90dabf-metrics-certs\") pod \"network-metrics-daemon-gs42n\" (UID: \"e7f09238-2bbf-4587-9989-49419d90dabf\") " pod="openshift-multus/network-metrics-daemon-gs42n" Oct 31 04:03:35.873468 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:35.873340 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bc9d2733-89fa-4079-9dc1-01983df773cb-original-pull-secret\") pod \"global-pull-secret-syncer-tlcrp\" (UID: \"bc9d2733-89fa-4079-9dc1-01983df773cb\") " pod="kube-system/global-pull-secret-syncer-tlcrp" Oct 31 04:03:35.873545 ip-10-0-134-155 kubenswrapper[2577]: E1031 04:03:35.873504 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 04:03:35.873604 ip-10-0-134-155 kubenswrapper[2577]: E1031 04:03:35.873573 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc9d2733-89fa-4079-9dc1-01983df773cb-original-pull-secret podName:bc9d2733-89fa-4079-9dc1-01983df773cb nodeName:}" failed. No retries permitted until 2026-10-31 04:03:37.873554796 +0000 UTC m=+6.088607098 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bc9d2733-89fa-4079-9dc1-01983df773cb-original-pull-secret") pod "global-pull-secret-syncer-tlcrp" (UID: "bc9d2733-89fa-4079-9dc1-01983df773cb") : object "kube-system"/"original-pull-secret" not registered Oct 31 04:03:35.874011 ip-10-0-134-155 kubenswrapper[2577]: E1031 04:03:35.873990 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 04:03:35.874099 ip-10-0-134-155 kubenswrapper[2577]: E1031 04:03:35.874050 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7f09238-2bbf-4587-9989-49419d90dabf-metrics-certs podName:e7f09238-2bbf-4587-9989-49419d90dabf nodeName:}" failed. No retries permitted until 2026-10-31 04:03:37.874032927 +0000 UTC m=+6.089085230 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e7f09238-2bbf-4587-9989-49419d90dabf-metrics-certs") pod "network-metrics-daemon-gs42n" (UID: "e7f09238-2bbf-4587-9989-49419d90dabf") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 04:03:35.974013 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:35.973980 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jq5vs\" (UniqueName: \"kubernetes.io/projected/f70f5eab-445f-430e-bfb1-92e4d620b90d-kube-api-access-jq5vs\") pod \"network-check-target-w9sdm\" (UID: \"f70f5eab-445f-430e-bfb1-92e4d620b90d\") " pod="openshift-network-diagnostics/network-check-target-w9sdm" Oct 31 04:03:35.974189 ip-10-0-134-155 kubenswrapper[2577]: E1031 04:03:35.974174 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 04:03:35.974257 ip-10-0-134-155 kubenswrapper[2577]: E1031 04:03:35.974198 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 04:03:35.974257 ip-10-0-134-155 kubenswrapper[2577]: E1031 04:03:35.974213 2577 projected.go:196] Error preparing data for projected volume kube-api-access-jq5vs for pod openshift-network-diagnostics/network-check-target-w9sdm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 04:03:35.974357 ip-10-0-134-155 kubenswrapper[2577]: E1031 04:03:35.974267 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f70f5eab-445f-430e-bfb1-92e4d620b90d-kube-api-access-jq5vs podName:f70f5eab-445f-430e-bfb1-92e4d620b90d nodeName:}" failed. No retries permitted until 2026-10-31 04:03:37.974249378 +0000 UTC m=+6.189301667 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-jq5vs" (UniqueName: "kubernetes.io/projected/f70f5eab-445f-430e-bfb1-92e4d620b90d-kube-api-access-jq5vs") pod "network-check-target-w9sdm" (UID: "f70f5eab-445f-430e-bfb1-92e4d620b90d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 04:03:36.209451 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:36.209356 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 03:58:34 +0000 UTC" deadline="2028-06-02 00:10:39.853054146 +0000 UTC" Oct 31 04:03:36.209451 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:36.209395 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13916h7m3.643664126s" Oct 31 04:03:36.275808 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:36.275776 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gs42n" Oct 31 04:03:36.275976 ip-10-0-134-155 kubenswrapper[2577]: E1031 04:03:36.275937 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gs42n" podUID="e7f09238-2bbf-4587-9989-49419d90dabf" Oct 31 04:03:36.276370 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:36.276349 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w9sdm" Oct 31 04:03:36.276511 ip-10-0-134-155 kubenswrapper[2577]: E1031 04:03:36.276473 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w9sdm" podUID="f70f5eab-445f-430e-bfb1-92e4d620b90d" Oct 31 04:03:36.276589 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:36.276549 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tlcrp" Oct 31 04:03:36.276640 ip-10-0-134-155 kubenswrapper[2577]: E1031 04:03:36.276622 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tlcrp" podUID="bc9d2733-89fa-4079-9dc1-01983df773cb" Oct 31 04:03:36.377874 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:36.377842 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-155.ec2.internal" event={"ID":"b3351600635c8d30ebae275373d4b16d","Type":"ContainerStarted","Data":"50fef1270a6438617bb2c72fb215da4f2c237e4ac94be7b51b9d641177bacede"} Oct 31 04:03:36.420049 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:36.419144 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-155.ec2.internal" podStartSLOduration=2.4191278929999998 podStartE2EDuration="2.419127893s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 04:03:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:03:36.41868875 +0000 UTC m=+4.633741060" watchObservedRunningTime="2026-10-31 04:03:36.419127893 +0000 UTC m=+4.634180206" Oct 31 04:03:37.386389 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:37.386333 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kmh48" event={"ID":"79ae1195-4ff1-46da-b1e2-83042dba6caa","Type":"ContainerStarted","Data":"39af99a8302ba7f2a6db408099a24e9fae48b6152d199627fcb141f3cefff6c1"} Oct 31 04:03:37.897958 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:37.897136 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bc9d2733-89fa-4079-9dc1-01983df773cb-original-pull-secret\") pod \"global-pull-secret-syncer-tlcrp\" (UID: \"bc9d2733-89fa-4079-9dc1-01983df773cb\") " pod="kube-system/global-pull-secret-syncer-tlcrp" Oct 31 04:03:37.897958 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:37.897190 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7f09238-2bbf-4587-9989-49419d90dabf-metrics-certs\") pod \"network-metrics-daemon-gs42n\" (UID: \"e7f09238-2bbf-4587-9989-49419d90dabf\") " pod="openshift-multus/network-metrics-daemon-gs42n" Oct 31 04:03:37.897958 ip-10-0-134-155 kubenswrapper[2577]: E1031 04:03:37.897404 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 04:03:37.897958 ip-10-0-134-155 kubenswrapper[2577]: E1031 04:03:37.897473 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7f09238-2bbf-4587-9989-49419d90dabf-metrics-certs podName:e7f09238-2bbf-4587-9989-49419d90dabf nodeName:}" failed. No retries permitted until 2026-10-31 04:03:41.89745311 +0000 UTC m=+10.112505411 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e7f09238-2bbf-4587-9989-49419d90dabf-metrics-certs") pod "network-metrics-daemon-gs42n" (UID: "e7f09238-2bbf-4587-9989-49419d90dabf") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 04:03:37.897958 ip-10-0-134-155 kubenswrapper[2577]: E1031 04:03:37.897878 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 04:03:37.897958 ip-10-0-134-155 kubenswrapper[2577]: E1031 04:03:37.897932 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc9d2733-89fa-4079-9dc1-01983df773cb-original-pull-secret podName:bc9d2733-89fa-4079-9dc1-01983df773cb nodeName:}" failed. No retries permitted until 2026-10-31 04:03:41.897916203 +0000 UTC m=+10.112968492 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bc9d2733-89fa-4079-9dc1-01983df773cb-original-pull-secret") pod "global-pull-secret-syncer-tlcrp" (UID: "bc9d2733-89fa-4079-9dc1-01983df773cb") : object "kube-system"/"original-pull-secret" not registered Oct 31 04:03:37.998055 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:37.998017 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jq5vs\" (UniqueName: \"kubernetes.io/projected/f70f5eab-445f-430e-bfb1-92e4d620b90d-kube-api-access-jq5vs\") pod \"network-check-target-w9sdm\" (UID: \"f70f5eab-445f-430e-bfb1-92e4d620b90d\") " pod="openshift-network-diagnostics/network-check-target-w9sdm" Oct 31 04:03:37.998392 ip-10-0-134-155 kubenswrapper[2577]: E1031 04:03:37.998288 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 04:03:37.998392 ip-10-0-134-155 kubenswrapper[2577]: E1031 04:03:37.998310 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 04:03:37.998392 ip-10-0-134-155 kubenswrapper[2577]: E1031 04:03:37.998323 2577 projected.go:196] Error preparing data for projected volume kube-api-access-jq5vs for pod openshift-network-diagnostics/network-check-target-w9sdm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 04:03:37.998648 ip-10-0-134-155 kubenswrapper[2577]: E1031 04:03:37.998400 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f70f5eab-445f-430e-bfb1-92e4d620b90d-kube-api-access-jq5vs podName:f70f5eab-445f-430e-bfb1-92e4d620b90d nodeName:}" failed. No retries permitted until 2026-10-31 04:03:41.998364831 +0000 UTC m=+10.213417136 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-jq5vs" (UniqueName: "kubernetes.io/projected/f70f5eab-445f-430e-bfb1-92e4d620b90d-kube-api-access-jq5vs") pod "network-check-target-w9sdm" (UID: "f70f5eab-445f-430e-bfb1-92e4d620b90d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 04:03:38.279838 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:38.279763 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tlcrp" Oct 31 04:03:38.279991 ip-10-0-134-155 kubenswrapper[2577]: E1031 04:03:38.279900 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tlcrp" podUID="bc9d2733-89fa-4079-9dc1-01983df773cb" Oct 31 04:03:38.280358 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:38.280307 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gs42n" Oct 31 04:03:38.280454 ip-10-0-134-155 kubenswrapper[2577]: E1031 04:03:38.280434 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gs42n" podUID="e7f09238-2bbf-4587-9989-49419d90dabf" Oct 31 04:03:38.280512 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:38.280502 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w9sdm" Oct 31 04:03:38.280596 ip-10-0-134-155 kubenswrapper[2577]: E1031 04:03:38.280577 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w9sdm" podUID="f70f5eab-445f-430e-bfb1-92e4d620b90d" Oct 31 04:03:38.396918 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:38.396884 2577 generic.go:356] "Generic (PLEG): container finished" podID="79ae1195-4ff1-46da-b1e2-83042dba6caa" containerID="39af99a8302ba7f2a6db408099a24e9fae48b6152d199627fcb141f3cefff6c1" exitCode=0 Oct 31 04:03:38.397302 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:38.396930 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kmh48" event={"ID":"79ae1195-4ff1-46da-b1e2-83042dba6caa","Type":"ContainerDied","Data":"39af99a8302ba7f2a6db408099a24e9fae48b6152d199627fcb141f3cefff6c1"} Oct 31 04:03:39.407395 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:39.407361 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-ntbm9" event={"ID":"f12d4ae1-e673-4a9e-bd43-92d6e9d5a023","Type":"ContainerStarted","Data":"9370eb7590e109f571027fca92e767c5e3b5e6ad18cd73ab728c5f5b8c2eaae0"} Oct 31 04:03:40.004989 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:40.004921 2577 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-ntbm9" Oct 31 04:03:40.005738 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:40.005716 2577 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-ntbm9" Oct 31 04:03:40.047571 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:40.047523 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-ntbm9" podStartSLOduration=3.178817877 podStartE2EDuration="7.047504023s" totalImagesPullingTime="3.868686146s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:03:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:03:39.449609621 +0000 UTC m=+7.664661932" watchObservedRunningTime="2026-10-31 04:03:40.047504023 +0000 UTC m=+8.262556334" Oct 31 04:03:40.273675 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:40.273575 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tlcrp" Oct 31 04:03:40.273837 ip-10-0-134-155 kubenswrapper[2577]: E1031 04:03:40.273711 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tlcrp" podUID="bc9d2733-89fa-4079-9dc1-01983df773cb" Oct 31 04:03:40.274117 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:40.274097 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gs42n" Oct 31 04:03:40.274236 ip-10-0-134-155 kubenswrapper[2577]: E1031 04:03:40.274213 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gs42n" podUID="e7f09238-2bbf-4587-9989-49419d90dabf" Oct 31 04:03:40.274308 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:40.274281 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w9sdm" Oct 31 04:03:40.274841 ip-10-0-134-155 kubenswrapper[2577]: E1031 04:03:40.274784 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w9sdm" podUID="f70f5eab-445f-430e-bfb1-92e4d620b90d" Oct 31 04:03:40.414428 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:40.413681 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-xmbxc" event={"ID":"a90157bb-46c0-4afa-82b1-9f6819518744","Type":"ContainerStarted","Data":"8467b15a0c9f1a01fe2fe155be16c8df25f0e600690fa6a1c9f06d6f83c6b88b"} Oct 31 04:03:40.418118 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:40.417828 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zw4rh" event={"ID":"c581e590-87e5-4cd7-8d84-eabb4ee21fd1","Type":"ContainerStarted","Data":"3bbfb977da486017f15380ccbc400946bdc86bf78d114ec1d3e87f9f80e1f63e"} Oct 31 04:03:41.421424 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:41.420453 2577 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 31 04:03:41.673622 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:41.673563 2577 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 31 04:03:41.931234 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:41.931137 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bc9d2733-89fa-4079-9dc1-01983df773cb-original-pull-secret\") pod \"global-pull-secret-syncer-tlcrp\" (UID: \"bc9d2733-89fa-4079-9dc1-01983df773cb\") " pod="kube-system/global-pull-secret-syncer-tlcrp" Oct 31 04:03:41.931234 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:41.931201 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7f09238-2bbf-4587-9989-49419d90dabf-metrics-certs\") pod \"network-metrics-daemon-gs42n\" (UID: \"e7f09238-2bbf-4587-9989-49419d90dabf\") " pod="openshift-multus/network-metrics-daemon-gs42n" Oct 31 04:03:41.931495 ip-10-0-134-155 kubenswrapper[2577]: E1031 04:03:41.931349 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 04:03:41.931495 ip-10-0-134-155 kubenswrapper[2577]: E1031 04:03:41.931431 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7f09238-2bbf-4587-9989-49419d90dabf-metrics-certs podName:e7f09238-2bbf-4587-9989-49419d90dabf nodeName:}" failed. No retries permitted until 2026-10-31 04:03:49.931409197 +0000 UTC m=+18.146461505 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e7f09238-2bbf-4587-9989-49419d90dabf-metrics-certs") pod "network-metrics-daemon-gs42n" (UID: "e7f09238-2bbf-4587-9989-49419d90dabf") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 04:03:41.931842 ip-10-0-134-155 kubenswrapper[2577]: E1031 04:03:41.931732 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 04:03:41.931842 ip-10-0-134-155 kubenswrapper[2577]: E1031 04:03:41.931811 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc9d2733-89fa-4079-9dc1-01983df773cb-original-pull-secret podName:bc9d2733-89fa-4079-9dc1-01983df773cb nodeName:}" failed. No retries permitted until 2026-10-31 04:03:49.931792436 +0000 UTC m=+18.146844741 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bc9d2733-89fa-4079-9dc1-01983df773cb-original-pull-secret") pod "global-pull-secret-syncer-tlcrp" (UID: "bc9d2733-89fa-4079-9dc1-01983df773cb") : object "kube-system"/"original-pull-secret" not registered Oct 31 04:03:42.032958 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:42.032305 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jq5vs\" (UniqueName: \"kubernetes.io/projected/f70f5eab-445f-430e-bfb1-92e4d620b90d-kube-api-access-jq5vs\") pod \"network-check-target-w9sdm\" (UID: \"f70f5eab-445f-430e-bfb1-92e4d620b90d\") " pod="openshift-network-diagnostics/network-check-target-w9sdm" Oct 31 04:03:42.032958 ip-10-0-134-155 kubenswrapper[2577]: E1031 04:03:42.032529 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 04:03:42.032958 ip-10-0-134-155 kubenswrapper[2577]: E1031 04:03:42.032546 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 04:03:42.032958 ip-10-0-134-155 kubenswrapper[2577]: E1031 04:03:42.032556 2577 projected.go:196] Error preparing data for projected volume kube-api-access-jq5vs for pod openshift-network-diagnostics/network-check-target-w9sdm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 04:03:42.032958 ip-10-0-134-155 kubenswrapper[2577]: E1031 04:03:42.032605 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f70f5eab-445f-430e-bfb1-92e4d620b90d-kube-api-access-jq5vs podName:f70f5eab-445f-430e-bfb1-92e4d620b90d nodeName:}" failed. No retries permitted until 2026-10-31 04:03:50.032590771 +0000 UTC m=+18.247643072 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-jq5vs" (UniqueName: "kubernetes.io/projected/f70f5eab-445f-430e-bfb1-92e4d620b90d-kube-api-access-jq5vs") pod "network-check-target-w9sdm" (UID: "f70f5eab-445f-430e-bfb1-92e4d620b90d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 04:03:42.222433 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:42.222257 2577 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-31T04:03:41.673583981Z","UUID":"34e9a87d-0992-4c44-a6ab-27f68a88ecdd","Handler":null,"Name":"","Endpoint":""} Oct 31 04:03:42.224452 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:42.224428 2577 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 31 04:03:42.224615 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:42.224463 2577 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 31 04:03:42.278114 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:42.278087 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gs42n" Oct 31 04:03:42.278302 ip-10-0-134-155 kubenswrapper[2577]: E1031 04:03:42.278189 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gs42n" podUID="e7f09238-2bbf-4587-9989-49419d90dabf" Oct 31 04:03:42.278585 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:42.278537 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w9sdm" Oct 31 04:03:42.278658 ip-10-0-134-155 kubenswrapper[2577]: E1031 04:03:42.278635 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w9sdm" podUID="f70f5eab-445f-430e-bfb1-92e4d620b90d" Oct 31 04:03:42.278772 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:42.278755 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tlcrp" Oct 31 04:03:42.278935 ip-10-0-134-155 kubenswrapper[2577]: E1031 04:03:42.278881 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tlcrp" podUID="bc9d2733-89fa-4079-9dc1-01983df773cb" Oct 31 04:03:42.425730 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:42.425685 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-zmrmk" event={"ID":"153e979c-3c5d-4c4f-b838-fc26db0b5cac","Type":"ContainerStarted","Data":"4daf203b840874f000404649770317633b2eafa98d4717ad0179c56d6f74dd3d"} Oct 31 04:03:42.429235 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:42.429209 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zw4rh" event={"ID":"c581e590-87e5-4cd7-8d84-eabb4ee21fd1","Type":"ContainerStarted","Data":"7fb6ed6afc68570c2ad39e10684ae5b039af4361eb13f47cc84575f8cd8f1b05"} Oct 31 04:03:42.470914 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:42.470838 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-xmbxc" podStartSLOduration=4.5965624609999995 podStartE2EDuration="9.47081736s" totalImagesPullingTime="4.874254899s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 04:03:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:03:40.45572622 +0000 UTC m=+8.670778529" watchObservedRunningTime="2026-10-31 04:03:42.47081736 +0000 UTC m=+10.685869681" Oct 31 04:03:43.436145 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:43.435935 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-77z7p" event={"ID":"1f2543cc-b528-41be-92e0-9d654405a723","Type":"ContainerStarted","Data":"a301460a24a35e06d19b7f33713c37d99bd4dd01763d14b8e4def7503225b767"} Oct 31 04:03:43.439686 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:43.439655 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zw4rh" event={"ID":"c581e590-87e5-4cd7-8d84-eabb4ee21fd1","Type":"ContainerStarted","Data":"a9539d875cc3ab716c2c3aa1e701538e270ea4e0a28a3f93ae35c2ac5d58ca52"} Oct 31 04:03:43.442056 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:43.441964 2577 generic.go:356] "Generic (PLEG): container finished" podID="79ae1195-4ff1-46da-b1e2-83042dba6caa" containerID="b7c07b812fa6f40ca769dff85f75ecdbb93b0cfc33ac64de3c1127b2d7178162" exitCode=0 Oct 31 04:03:43.442056 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:43.442040 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kmh48" event={"ID":"79ae1195-4ff1-46da-b1e2-83042dba6caa","Type":"ContainerDied","Data":"b7c07b812fa6f40ca769dff85f75ecdbb93b0cfc33ac64de3c1127b2d7178162"} Oct 31 04:03:43.478062 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:43.477945 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-zmrmk" podStartSLOduration=2.710326525 podStartE2EDuration="10.477926615s" totalImagesPullingTime="7.76760009s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 04:03:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:03:42.471775365 +0000 UTC m=+10.686827678" watchObservedRunningTime="2026-10-31 04:03:43.477926615 +0000 UTC m=+11.692978926" Oct 31 04:03:43.478483 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:43.478443 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-77z7p" podStartSLOduration=2.27781003 podStartE2EDuration="10.478434685s" totalImagesPullingTime="8.200624655s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 04:03:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:03:43.47745978 +0000 UTC m=+11.692512091" watchObservedRunningTime="2026-10-31 04:03:43.478434685 +0000 UTC m=+11.693487000" Oct 31 04:03:43.571630 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:43.571541 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zw4rh" podStartSLOduration=2.543287911 podStartE2EDuration="10.571525057s" totalImagesPullingTime="8.028237146s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 04:03:33 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:03:43.522432449 +0000 UTC m=+11.737484761" watchObservedRunningTime="2026-10-31 04:03:43.571525057 +0000 UTC m=+11.786577370" Oct 31 04:03:44.274240 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:44.273717 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w9sdm" Oct 31 04:03:44.274240 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:44.273727 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gs42n" Oct 31 04:03:44.274240 ip-10-0-134-155 kubenswrapper[2577]: E1031 04:03:44.273858 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w9sdm" podUID="f70f5eab-445f-430e-bfb1-92e4d620b90d" Oct 31 04:03:44.274240 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:44.273871 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tlcrp" Oct 31 04:03:44.274240 ip-10-0-134-155 kubenswrapper[2577]: E1031 04:03:44.273979 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gs42n" podUID="e7f09238-2bbf-4587-9989-49419d90dabf" Oct 31 04:03:44.274240 ip-10-0-134-155 kubenswrapper[2577]: E1031 04:03:44.274076 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tlcrp" podUID="bc9d2733-89fa-4079-9dc1-01983df773cb" Oct 31 04:03:44.447102 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:44.447054 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-6q4xv" event={"ID":"24e22999-2c81-4779-a901-5b9faa15c1cb","Type":"ContainerStarted","Data":"57cf3512e996df9d960e4bafaf7a1bfbd375b762f4972a30adf1afae2417f82a"} Oct 31 04:03:44.449914 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:44.449883 2577 generic.go:356] "Generic (PLEG): container finished" podID="79ae1195-4ff1-46da-b1e2-83042dba6caa" containerID="0ae494e05fd8874d01ee8cac631445033f44579b7d9d3037fcced92bce25054f" exitCode=0 Oct 31 04:03:44.450062 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:44.449995 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kmh48" event={"ID":"79ae1195-4ff1-46da-b1e2-83042dba6caa","Type":"ContainerDied","Data":"0ae494e05fd8874d01ee8cac631445033f44579b7d9d3037fcced92bce25054f"} Oct 31 04:03:44.533828 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:44.533734 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-6q4xv" podStartSLOduration=3.432300968 podStartE2EDuration="11.533715633s" totalImagesPullingTime="8.101414665s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 04:03:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:03:44.487540656 +0000 UTC m=+12.702592969" watchObservedRunningTime="2026-10-31 04:03:44.533715633 +0000 UTC m=+12.748767943" Oct 31 04:03:45.455052 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:45.455017 2577 generic.go:356] "Generic (PLEG): container finished" podID="79ae1195-4ff1-46da-b1e2-83042dba6caa" containerID="501a3129f65614bb455e84ae30001b4f37fcf3417744582f0b1c5a37919a1a67" exitCode=0 Oct 31 04:03:45.455719 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:45.455053 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kmh48" event={"ID":"79ae1195-4ff1-46da-b1e2-83042dba6caa","Type":"ContainerDied","Data":"501a3129f65614bb455e84ae30001b4f37fcf3417744582f0b1c5a37919a1a67"} Oct 31 04:03:45.456893 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:45.456865 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-b59p9" event={"ID":"d4eabffb-abf1-4409-94b5-c0489ab37853","Type":"ContainerStarted","Data":"c09e469a0a0cf4e8aec2f1691057c7eac564bd07bd6ee9a2f2827d0fc3fd7be0"} Oct 31 04:03:45.540480 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:45.540413 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-b59p9" podStartSLOduration=2.223351005 podStartE2EDuration="12.540393222s" totalImagesPullingTime="10.317042217s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 04:03:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:03:45.539795612 +0000 UTC m=+13.754847920" watchObservedRunningTime="2026-10-31 04:03:45.540393222 +0000 UTC m=+13.755445530" Oct 31 04:03:46.273143 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:46.273111 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gs42n" Oct 31 04:03:46.273322 ip-10-0-134-155 kubenswrapper[2577]: E1031 04:03:46.273248 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gs42n" podUID="e7f09238-2bbf-4587-9989-49419d90dabf" Oct 31 04:03:46.273634 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:46.273616 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w9sdm" Oct 31 04:03:46.273724 ip-10-0-134-155 kubenswrapper[2577]: E1031 04:03:46.273704 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w9sdm" podUID="f70f5eab-445f-430e-bfb1-92e4d620b90d" Oct 31 04:03:46.273809 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:46.273797 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tlcrp" Oct 31 04:03:46.273921 ip-10-0-134-155 kubenswrapper[2577]: E1031 04:03:46.273883 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tlcrp" podUID="bc9d2733-89fa-4079-9dc1-01983df773cb" Oct 31 04:03:47.463981 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:47.463744 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kmh48" event={"ID":"79ae1195-4ff1-46da-b1e2-83042dba6caa","Type":"ContainerStarted","Data":"907eb0f93f1109e8efd49fd6a207e71979c33b2bf22da0fa84e6ecee553bbe93"} Oct 31 04:03:48.273490 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:48.273446 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gs42n" Oct 31 04:03:48.273667 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:48.273510 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tlcrp" Oct 31 04:03:48.273667 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:48.273590 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w9sdm" Oct 31 04:03:48.273771 ip-10-0-134-155 kubenswrapper[2577]: E1031 04:03:48.273718 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gs42n" podUID="e7f09238-2bbf-4587-9989-49419d90dabf" Oct 31 04:03:48.273826 ip-10-0-134-155 kubenswrapper[2577]: E1031 04:03:48.273802 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w9sdm" podUID="f70f5eab-445f-430e-bfb1-92e4d620b90d" Oct 31 04:03:48.273915 ip-10-0-134-155 kubenswrapper[2577]: E1031 04:03:48.273896 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tlcrp" podUID="bc9d2733-89fa-4079-9dc1-01983df773cb" Oct 31 04:03:48.469552 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:48.469511 2577 generic.go:356] "Generic (PLEG): container finished" podID="79ae1195-4ff1-46da-b1e2-83042dba6caa" containerID="907eb0f93f1109e8efd49fd6a207e71979c33b2bf22da0fa84e6ecee553bbe93" exitCode=0 Oct 31 04:03:48.469552 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:48.469556 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kmh48" event={"ID":"79ae1195-4ff1-46da-b1e2-83042dba6caa","Type":"ContainerDied","Data":"907eb0f93f1109e8efd49fd6a207e71979c33b2bf22da0fa84e6ecee553bbe93"} Oct 31 04:03:49.474604 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:49.474571 2577 generic.go:356] "Generic (PLEG): container finished" podID="79ae1195-4ff1-46da-b1e2-83042dba6caa" containerID="ec7d07cb450cc1b873d8f9c7780382a3ea5bfd6c1f99b7e07bf439191856ae2c" exitCode=0 Oct 31 04:03:49.475024 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:49.474617 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kmh48" event={"ID":"79ae1195-4ff1-46da-b1e2-83042dba6caa","Type":"ContainerDied","Data":"ec7d07cb450cc1b873d8f9c7780382a3ea5bfd6c1f99b7e07bf439191856ae2c"} Oct 31 04:03:49.993749 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:49.993537 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bc9d2733-89fa-4079-9dc1-01983df773cb-original-pull-secret\") pod \"global-pull-secret-syncer-tlcrp\" (UID: \"bc9d2733-89fa-4079-9dc1-01983df773cb\") " pod="kube-system/global-pull-secret-syncer-tlcrp" Oct 31 04:03:49.993749 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:49.993718 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7f09238-2bbf-4587-9989-49419d90dabf-metrics-certs\") pod \"network-metrics-daemon-gs42n\" (UID: \"e7f09238-2bbf-4587-9989-49419d90dabf\") " pod="openshift-multus/network-metrics-daemon-gs42n" Oct 31 04:03:49.993937 ip-10-0-134-155 kubenswrapper[2577]: E1031 04:03:49.993677 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 04:03:49.993937 ip-10-0-134-155 kubenswrapper[2577]: E1031 04:03:49.993806 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 04:03:49.993937 ip-10-0-134-155 kubenswrapper[2577]: E1031 04:03:49.993827 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc9d2733-89fa-4079-9dc1-01983df773cb-original-pull-secret podName:bc9d2733-89fa-4079-9dc1-01983df773cb nodeName:}" failed. No retries permitted until 2026-10-31 04:04:05.993811505 +0000 UTC m=+34.208863790 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bc9d2733-89fa-4079-9dc1-01983df773cb-original-pull-secret") pod "global-pull-secret-syncer-tlcrp" (UID: "bc9d2733-89fa-4079-9dc1-01983df773cb") : object "kube-system"/"original-pull-secret" not registered Oct 31 04:03:49.993937 ip-10-0-134-155 kubenswrapper[2577]: E1031 04:03:49.993841 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7f09238-2bbf-4587-9989-49419d90dabf-metrics-certs podName:e7f09238-2bbf-4587-9989-49419d90dabf nodeName:}" failed. No retries permitted until 2026-10-31 04:04:05.993834741 +0000 UTC m=+34.208887033 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e7f09238-2bbf-4587-9989-49419d90dabf-metrics-certs") pod "network-metrics-daemon-gs42n" (UID: "e7f09238-2bbf-4587-9989-49419d90dabf") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 04:03:50.094920 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:50.094893 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jq5vs\" (UniqueName: \"kubernetes.io/projected/f70f5eab-445f-430e-bfb1-92e4d620b90d-kube-api-access-jq5vs\") pod \"network-check-target-w9sdm\" (UID: \"f70f5eab-445f-430e-bfb1-92e4d620b90d\") " pod="openshift-network-diagnostics/network-check-target-w9sdm" Oct 31 04:03:50.095050 ip-10-0-134-155 kubenswrapper[2577]: E1031 04:03:50.095035 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 04:03:50.095050 ip-10-0-134-155 kubenswrapper[2577]: E1031 04:03:50.095050 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 04:03:50.095127 ip-10-0-134-155 kubenswrapper[2577]: E1031 04:03:50.095059 2577 projected.go:196] Error preparing data for projected volume kube-api-access-jq5vs for pod openshift-network-diagnostics/network-check-target-w9sdm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 04:03:50.095127 ip-10-0-134-155 kubenswrapper[2577]: E1031 04:03:50.095100 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f70f5eab-445f-430e-bfb1-92e4d620b90d-kube-api-access-jq5vs podName:f70f5eab-445f-430e-bfb1-92e4d620b90d nodeName:}" failed. No retries permitted until 2026-10-31 04:04:06.095088251 +0000 UTC m=+34.310140537 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-jq5vs" (UniqueName: "kubernetes.io/projected/f70f5eab-445f-430e-bfb1-92e4d620b90d-kube-api-access-jq5vs") pod "network-check-target-w9sdm" (UID: "f70f5eab-445f-430e-bfb1-92e4d620b90d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 04:03:50.273283 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:50.273211 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tlcrp" Oct 31 04:03:50.273283 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:50.273246 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w9sdm" Oct 31 04:03:50.273458 ip-10-0-134-155 kubenswrapper[2577]: E1031 04:03:50.273299 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tlcrp" podUID="bc9d2733-89fa-4079-9dc1-01983df773cb" Oct 31 04:03:50.273458 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:50.273365 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gs42n" Oct 31 04:03:50.273458 ip-10-0-134-155 kubenswrapper[2577]: E1031 04:03:50.273448 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gs42n" podUID="e7f09238-2bbf-4587-9989-49419d90dabf" Oct 31 04:03:50.273538 ip-10-0-134-155 kubenswrapper[2577]: E1031 04:03:50.273499 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w9sdm" podUID="f70f5eab-445f-430e-bfb1-92e4d620b90d" Oct 31 04:03:50.480157 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:50.480127 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kmh48" event={"ID":"79ae1195-4ff1-46da-b1e2-83042dba6caa","Type":"ContainerStarted","Data":"f024a7d9f8b9aa270aab8f767f369b2d323054477368950a534ea74db44ae075"} Oct 31 04:03:50.482905 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:50.482882 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zbh9t" event={"ID":"f7a16817-8ea2-4649-9e42-3d340e1b9703","Type":"ContainerStarted","Data":"8c7521a3e985125feb9e1163dd39f9e756cd4c3c86ffa5873bd81b626ca1967f"} Oct 31 04:03:50.483005 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:50.482942 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zbh9t" event={"ID":"f7a16817-8ea2-4649-9e42-3d340e1b9703","Type":"ContainerStarted","Data":"f4996b75f625b7232de13bd42d411ad046b50dad0a5c3a0d6bd284cad20f6b41"} Oct 31 04:03:50.483005 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:50.482976 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zbh9t" event={"ID":"f7a16817-8ea2-4649-9e42-3d340e1b9703","Type":"ContainerStarted","Data":"e5b3b8503d9956c7ab1b0309acc48b91ae592bc8fbf091427eddbcd96fc321a5"} Oct 31 04:03:50.483076 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:50.483003 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zbh9t" event={"ID":"f7a16817-8ea2-4649-9e42-3d340e1b9703","Type":"ContainerStarted","Data":"af71718654c843d712e24b9bafa782247437e197bf6faa230bf8463f2c400463"} Oct 31 04:03:50.483076 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:50.483033 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zbh9t" event={"ID":"f7a16817-8ea2-4649-9e42-3d340e1b9703","Type":"ContainerStarted","Data":"b35a9e245fd5585edfd770409b67b89360e25bb4cb231d7b9c6cd35e7c10dfee"} Oct 31 04:03:50.483076 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:50.483060 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zbh9t" event={"ID":"f7a16817-8ea2-4649-9e42-3d340e1b9703","Type":"ContainerStarted","Data":"52a42af0ae61226699eb7a760c7315d365e03071036d8373487ef4462aa0f48e"} Oct 31 04:03:50.524640 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:50.524562 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-kmh48" podStartSLOduration=7.296206257 podStartE2EDuration="17.52455072s" totalImagesPullingTime="10.228344463s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 04:03:33 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:03:50.524037321 +0000 UTC m=+18.739089627" watchObservedRunningTime="2026-10-31 04:03:50.52455072 +0000 UTC m=+18.739603026" Oct 31 04:03:52.273838 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:52.273802 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gs42n" Oct 31 04:03:52.274501 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:52.273955 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tlcrp" Oct 31 04:03:52.274501 ip-10-0-134-155 kubenswrapper[2577]: E1031 04:03:52.273965 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gs42n" podUID="e7f09238-2bbf-4587-9989-49419d90dabf" Oct 31 04:03:52.274501 ip-10-0-134-155 kubenswrapper[2577]: E1031 04:03:52.274069 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tlcrp" podUID="bc9d2733-89fa-4079-9dc1-01983df773cb" Oct 31 04:03:52.274501 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:52.274135 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w9sdm" Oct 31 04:03:52.274501 ip-10-0-134-155 kubenswrapper[2577]: E1031 04:03:52.274210 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w9sdm" podUID="f70f5eab-445f-430e-bfb1-92e4d620b90d" Oct 31 04:03:52.490178 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:52.490143 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zbh9t" event={"ID":"f7a16817-8ea2-4649-9e42-3d340e1b9703","Type":"ContainerStarted","Data":"8eee6093c02f00113d00452fcb08124384563617877625141dd4dc90e14634d9"} Oct 31 04:03:54.273802 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:54.273712 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tlcrp" Oct 31 04:03:54.273802 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:54.273735 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gs42n" Oct 31 04:03:54.274521 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:54.273712 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w9sdm" Oct 31 04:03:54.274521 ip-10-0-134-155 kubenswrapper[2577]: E1031 04:03:54.273849 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tlcrp" podUID="bc9d2733-89fa-4079-9dc1-01983df773cb" Oct 31 04:03:54.274521 ip-10-0-134-155 kubenswrapper[2577]: E1031 04:03:54.273889 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w9sdm" podUID="f70f5eab-445f-430e-bfb1-92e4d620b90d" Oct 31 04:03:54.274521 ip-10-0-134-155 kubenswrapper[2577]: E1031 04:03:54.273966 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gs42n" podUID="e7f09238-2bbf-4587-9989-49419d90dabf" Oct 31 04:03:54.498165 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:54.498132 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zbh9t" event={"ID":"f7a16817-8ea2-4649-9e42-3d340e1b9703","Type":"ContainerStarted","Data":"6bd70fee70d58f1339c818d6aa5ce55071e47de2281459c56a7b9272a4c9dc6e"} Oct 31 04:03:54.498438 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:54.498423 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zbh9t" Oct 31 04:03:54.498487 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:54.498479 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zbh9t" Oct 31 04:03:54.513661 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:54.513638 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zbh9t" Oct 31 04:03:54.583769 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:54.583668 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-zbh9t" podStartSLOduration=6.559654558 podStartE2EDuration="21.583654401s" totalImagesPullingTime="15.023999843s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 04:03:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:03:54.583309414 +0000 UTC m=+22.798361721" watchObservedRunningTime="2026-10-31 04:03:54.583654401 +0000 UTC m=+22.798706713" Oct 31 04:03:55.500543 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:55.500501 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zbh9t" Oct 31 04:03:55.515340 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:55.515304 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zbh9t" Oct 31 04:03:56.042542 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:56.042287 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-tlcrp"] Oct 31 04:03:56.042794 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:56.042547 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tlcrp" Oct 31 04:03:56.042794 ip-10-0-134-155 kubenswrapper[2577]: E1031 04:03:56.042626 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tlcrp" podUID="bc9d2733-89fa-4079-9dc1-01983df773cb" Oct 31 04:03:56.045480 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:56.045452 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-w9sdm"] Oct 31 04:03:56.045610 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:56.045562 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w9sdm" Oct 31 04:03:56.045687 ip-10-0-134-155 kubenswrapper[2577]: E1031 04:03:56.045664 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w9sdm" podUID="f70f5eab-445f-430e-bfb1-92e4d620b90d" Oct 31 04:03:56.046000 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:56.045975 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-gs42n"] Oct 31 04:03:56.046109 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:56.046080 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gs42n" Oct 31 04:03:56.046174 ip-10-0-134-155 kubenswrapper[2577]: E1031 04:03:56.046159 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gs42n" podUID="e7f09238-2bbf-4587-9989-49419d90dabf" Oct 31 04:03:56.177566 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:56.177529 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-ntbm9" Oct 31 04:03:56.177748 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:56.177648 2577 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 31 04:03:56.178126 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:56.178104 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-ntbm9" Oct 31 04:03:57.273881 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:57.273855 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w9sdm" Oct 31 04:03:57.274278 ip-10-0-134-155 kubenswrapper[2577]: E1031 04:03:57.273948 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w9sdm" podUID="f70f5eab-445f-430e-bfb1-92e4d620b90d" Oct 31 04:03:58.273171 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:58.273143 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gs42n" Oct 31 04:03:58.273360 ip-10-0-134-155 kubenswrapper[2577]: E1031 04:03:58.273310 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gs42n" podUID="e7f09238-2bbf-4587-9989-49419d90dabf" Oct 31 04:03:58.273427 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:58.273346 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tlcrp" Oct 31 04:03:58.273473 ip-10-0-134-155 kubenswrapper[2577]: E1031 04:03:58.273448 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tlcrp" podUID="bc9d2733-89fa-4079-9dc1-01983df773cb" Oct 31 04:03:59.273376 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:03:59.273306 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w9sdm" Oct 31 04:03:59.273759 ip-10-0-134-155 kubenswrapper[2577]: E1031 04:03:59.273444 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w9sdm" podUID="f70f5eab-445f-430e-bfb1-92e4d620b90d" Oct 31 04:04:00.274063 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:04:00.274029 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tlcrp" Oct 31 04:04:00.274473 ip-10-0-134-155 kubenswrapper[2577]: E1031 04:04:00.274162 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tlcrp" podUID="bc9d2733-89fa-4079-9dc1-01983df773cb" Oct 31 04:04:00.274473 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:04:00.274206 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gs42n" Oct 31 04:04:00.274473 ip-10-0-134-155 kubenswrapper[2577]: E1031 04:04:00.274305 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gs42n" podUID="e7f09238-2bbf-4587-9989-49419d90dabf" Oct 31 04:04:01.273412 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:04:01.273309 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w9sdm" Oct 31 04:04:01.273564 ip-10-0-134-155 kubenswrapper[2577]: E1031 04:04:01.273426 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w9sdm" podUID="f70f5eab-445f-430e-bfb1-92e4d620b90d" Oct 31 04:04:02.121591 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:04:02.121563 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-155.ec2.internal" event="NodeReady" Oct 31 04:04:02.121949 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:04:02.121686 2577 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 31 04:04:02.274161 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:04:02.274126 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gs42n" Oct 31 04:04:02.274371 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:04:02.274303 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tlcrp" Oct 31 04:04:02.288860 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:04:02.288822 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 31 04:04:02.289080 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:04:02.289059 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-lwzcw\"" Oct 31 04:04:02.289165 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:04:02.289062 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 31 04:04:02.289165 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:04:02.289128 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-9876q\"" Oct 31 04:04:03.273115 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:04:03.273086 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w9sdm" Oct 31 04:04:03.288340 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:04:03.288310 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-brpbt\"" Oct 31 04:04:03.292318 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:04:03.292295 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 31 04:04:03.292413 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:04:03.292322 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 31 04:04:05.282466 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:04:05.282438 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-77z7p_1f2543cc-b528-41be-92e0-9d654405a723/dns-node-resolver/0.log" Oct 31 04:04:06.007984 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:04:06.007911 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bc9d2733-89fa-4079-9dc1-01983df773cb-original-pull-secret\") pod \"global-pull-secret-syncer-tlcrp\" (UID: \"bc9d2733-89fa-4079-9dc1-01983df773cb\") " pod="kube-system/global-pull-secret-syncer-tlcrp" Oct 31 04:04:06.007984 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:04:06.007948 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7f09238-2bbf-4587-9989-49419d90dabf-metrics-certs\") pod \"network-metrics-daemon-gs42n\" (UID: \"e7f09238-2bbf-4587-9989-49419d90dabf\") " pod="openshift-multus/network-metrics-daemon-gs42n" Oct 31 04:04:06.011203 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:04:06.011173 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7f09238-2bbf-4587-9989-49419d90dabf-metrics-certs\") pod \"network-metrics-daemon-gs42n\" (UID: \"e7f09238-2bbf-4587-9989-49419d90dabf\") " pod="openshift-multus/network-metrics-daemon-gs42n" Oct 31 04:04:06.011321 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:04:06.011207 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bc9d2733-89fa-4079-9dc1-01983df773cb-original-pull-secret\") pod \"global-pull-secret-syncer-tlcrp\" (UID: \"bc9d2733-89fa-4079-9dc1-01983df773cb\") " pod="kube-system/global-pull-secret-syncer-tlcrp" Oct 31 04:04:06.108789 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:04:06.108759 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jq5vs\" (UniqueName: \"kubernetes.io/projected/f70f5eab-445f-430e-bfb1-92e4d620b90d-kube-api-access-jq5vs\") pod \"network-check-target-w9sdm\" (UID: \"f70f5eab-445f-430e-bfb1-92e4d620b90d\") " pod="openshift-network-diagnostics/network-check-target-w9sdm" Oct 31 04:04:06.111402 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:04:06.111377 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jq5vs\" (UniqueName: \"kubernetes.io/projected/f70f5eab-445f-430e-bfb1-92e4d620b90d-kube-api-access-jq5vs\") pod \"network-check-target-w9sdm\" (UID: \"f70f5eab-445f-430e-bfb1-92e4d620b90d\") " pod="openshift-network-diagnostics/network-check-target-w9sdm" Oct 31 04:04:06.191452 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:04:06.191414 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gs42n" Oct 31 04:04:06.193573 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:04:06.193552 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tlcrp" Oct 31 04:04:06.334037 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:04:06.334003 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w9sdm" Oct 31 04:04:06.423195 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:04:06.423172 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-xmbxc_a90157bb-46c0-4afa-82b1-9f6819518744/node-ca/0.log" Oct 31 04:04:06.622462 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:04:06.622430 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-tlcrp"] Oct 31 04:04:06.626232 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:04:06.626203 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc9d2733_89fa_4079_9dc1_01983df773cb.slice/crio-f56712820eeb59c980c23b370ff8d528b4b9a17d25b621931a79cad3343585e5 WatchSource:0}: Error finding container f56712820eeb59c980c23b370ff8d528b4b9a17d25b621931a79cad3343585e5: Status 404 returned error can't find the container with id f56712820eeb59c980c23b370ff8d528b4b9a17d25b621931a79cad3343585e5 Oct 31 04:04:06.633006 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:04:06.632981 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-gs42n"] Oct 31 04:04:06.635775 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:04:06.635756 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7f09238_2bbf_4587_9989_49419d90dabf.slice/crio-601b281d91973258a35641bcf167e3607780239835331d37842df6ccb98916d9 WatchSource:0}: Error finding container 601b281d91973258a35641bcf167e3607780239835331d37842df6ccb98916d9: Status 404 returned error can't find the container with id 601b281d91973258a35641bcf167e3607780239835331d37842df6ccb98916d9 Oct 31 04:04:06.658108 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:04:06.658082 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-w9sdm"] Oct 31 04:04:06.660857 ip-10-0-134-155 kubenswrapper[2577]: W1031 04:04:06.660833 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf70f5eab_445f_430e_bfb1_92e4d620b90d.slice/crio-ee1850a84839fb35a22fdac4cdb23ad16a64f08240a18fadbc54cccb1e3aeb61 WatchSource:0}: Error finding container ee1850a84839fb35a22fdac4cdb23ad16a64f08240a18fadbc54cccb1e3aeb61: Status 404 returned error can't find the container with id ee1850a84839fb35a22fdac4cdb23ad16a64f08240a18fadbc54cccb1e3aeb61 Oct 31 04:04:07.529735 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:04:07.529699 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-w9sdm" event={"ID":"f70f5eab-445f-430e-bfb1-92e4d620b90d","Type":"ContainerStarted","Data":"ee1850a84839fb35a22fdac4cdb23ad16a64f08240a18fadbc54cccb1e3aeb61"} Oct 31 04:04:07.531375 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:04:07.531342 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-tlcrp" event={"ID":"bc9d2733-89fa-4079-9dc1-01983df773cb","Type":"ContainerStarted","Data":"f56712820eeb59c980c23b370ff8d528b4b9a17d25b621931a79cad3343585e5"} Oct 31 04:04:07.532524 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:04:07.532499 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-gs42n" event={"ID":"e7f09238-2bbf-4587-9989-49419d90dabf","Type":"ContainerStarted","Data":"601b281d91973258a35641bcf167e3607780239835331d37842df6ccb98916d9"} Oct 31 04:04:08.536937 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:04:08.536898 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-gs42n" event={"ID":"e7f09238-2bbf-4587-9989-49419d90dabf","Type":"ContainerStarted","Data":"154a8830e01fef19ee08c2084658bdfa7177e56af149005860fcddb896a04ab9"} Oct 31 04:04:08.537402 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:04:08.536949 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-gs42n" event={"ID":"e7f09238-2bbf-4587-9989-49419d90dabf","Type":"ContainerStarted","Data":"cac7d2a368ca838cd6acce3d09e83b0aaa7d245aec5ff5fdd6e9ff925016091b"} Oct 31 04:04:08.577406 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:04:08.577215 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-gs42n" podStartSLOduration=34.03931231 podStartE2EDuration="35.577196847s" totalImagesPullingTime="1.537884537s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:03:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:04:08.577038894 +0000 UTC m=+36.792091201" watchObservedRunningTime="2026-10-31 04:04:08.577196847 +0000 UTC m=+36.792249166" Oct 31 04:04:10.544790 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:04:10.544753 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-w9sdm" event={"ID":"f70f5eab-445f-430e-bfb1-92e4d620b90d","Type":"ContainerStarted","Data":"aef3b8c0361911cf9ef8a7a30ee94039780a0aff1db5bdf8e455096e09dc071c"} Oct 31 04:04:10.545236 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:04:10.544895 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-w9sdm" Oct 31 04:04:10.586352 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:04:10.583965 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-w9sdm" podStartSLOduration=33.902685686 podStartE2EDuration="37.583949042s" totalImagesPullingTime="3.681263356s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:03:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:04:10.583269994 +0000 UTC m=+38.798322301" watchObservedRunningTime="2026-10-31 04:04:10.583949042 +0000 UTC m=+38.799001349" Oct 31 04:04:12.221926 ip-10-0-134-155 kubenswrapper[2577]: I1031 04:04:12.221808 2577 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 04:04:12.221966 ip-10-0-134-155 systemd[1]: Stopping Kubernetes Kubelet... Oct 31 04:04:12.235863 ip-10-0-134-155 systemd[1]: kubelet.service: Deactivated successfully. Oct 31 04:04:12.236031 ip-10-0-134-155 systemd[1]: Stopped Kubernetes Kubelet. Oct 31 04:04:12.236081 ip-10-0-134-155 systemd[1]: kubelet.service: Consumed 2.084s CPU time. Oct 31 04:04:12.254055 ip-10-0-134-155 systemd[1]: Starting Kubernetes Kubelet... Oct 31 04:04:12.358391 ip-10-0-134-155 kubenswrapper[4376]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 04:04:12.358391 ip-10-0-134-155 kubenswrapper[4376]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 31 04:04:12.358391 ip-10-0-134-155 kubenswrapper[4376]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 04:04:12.358391 ip-10-0-134-155 kubenswrapper[4376]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 04:04:12.360556 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.360462 4376 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 04:04:12.360556 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.360555 4376 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 04:04:12.360556 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.360560 4376 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 04:04:12.360669 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.360564 4376 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 04:04:12.360669 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.360567 4376 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 04:04:12.360669 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.360570 4376 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 04:04:12.360669 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.360573 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 04:04:12.360669 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.360576 4376 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 04:04:12.360669 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.360578 4376 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 04:04:12.360669 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.360581 4376 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 04:04:12.360669 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.360584 4376 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 04:04:12.360669 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.360587 4376 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 04:04:12.360669 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.360589 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 04:04:12.360669 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.360592 4376 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 04:04:12.360669 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.360595 4376 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 04:04:12.360669 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.360597 4376 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 04:04:12.360669 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.360600 4376 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 04:04:12.360669 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.360602 4376 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 04:04:12.360669 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.360605 4376 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 04:04:12.360669 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.360608 4376 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 04:04:12.360669 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.360611 4376 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 04:04:12.360669 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.360613 4376 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 04:04:12.361144 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.360617 4376 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 04:04:12.361144 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.360619 4376 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 04:04:12.361144 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.360627 4376 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 04:04:12.361144 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.360631 4376 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 04:04:12.361144 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.360633 4376 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 04:04:12.361144 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.360636 4376 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 04:04:12.361144 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.360639 4376 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 04:04:12.361144 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.360641 4376 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 04:04:12.361144 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.360644 4376 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 04:04:12.361144 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.360646 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 04:04:12.361144 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.360649 4376 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 04:04:12.361144 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.360652 4376 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 04:04:12.361144 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.360655 4376 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 04:04:12.361144 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.360659 4376 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 04:04:12.361144 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.360662 4376 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 04:04:12.361144 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.360665 4376 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 04:04:12.361144 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.360668 4376 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 04:04:12.361144 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.360671 4376 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 04:04:12.361144 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.360674 4376 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 04:04:12.361631 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.360677 4376 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 04:04:12.361631 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.360680 4376 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 04:04:12.361631 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.360683 4376 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 04:04:12.361631 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.360686 4376 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 04:04:12.361631 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.360688 4376 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 04:04:12.361631 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.360691 4376 feature_gate.go:386] unrecognized feature gate: Example Oct 31 04:04:12.361631 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.360694 4376 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 04:04:12.361631 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.360696 4376 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 04:04:12.361631 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.360699 4376 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 04:04:12.361631 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.360702 4376 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 04:04:12.361631 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.360704 4376 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 04:04:12.361631 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.360707 4376 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 04:04:12.361631 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.360710 4376 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 04:04:12.361631 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.360713 4376 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 04:04:12.361631 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.360724 4376 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 04:04:12.361631 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.360727 4376 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 04:04:12.361631 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.360733 4376 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 04:04:12.361631 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.360737 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 04:04:12.361631 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.360741 4376 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 04:04:12.362118 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.360744 4376 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 04:04:12.362118 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.360748 4376 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 04:04:12.362118 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.360751 4376 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 04:04:12.362118 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.360754 4376 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 04:04:12.362118 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.360757 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 04:04:12.362118 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.360760 4376 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 04:04:12.362118 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.360763 4376 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 04:04:12.362118 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.360765 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 04:04:12.362118 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.360768 4376 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 04:04:12.362118 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.360771 4376 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 04:04:12.362118 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.360773 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 04:04:12.362118 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.360776 4376 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 04:04:12.362118 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.360778 4376 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 04:04:12.362118 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.360781 4376 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 04:04:12.362118 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.360783 4376 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 04:04:12.362118 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.360786 4376 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 04:04:12.362118 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.360789 4376 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 04:04:12.362118 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.360792 4376 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 04:04:12.362118 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.360794 4376 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 04:04:12.362607 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.360797 4376 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 04:04:12.362607 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.360799 4376 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 04:04:12.362607 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.360802 4376 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 04:04:12.362607 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.360804 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 04:04:12.362607 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.360807 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 04:04:12.362607 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.360810 4376 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 04:04:12.362607 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.360813 4376 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 04:04:12.362607 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.360817 4376 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 04:04:12.362607 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.360821 4376 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 04:04:12.362607 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.360824 4376 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 04:04:12.362607 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.360828 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 04:04:12.362607 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.360831 4376 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 04:04:12.362607 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.360834 4376 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 04:04:12.362607 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.360837 4376 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 04:04:12.362607 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.360840 4376 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 04:04:12.362607 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.360844 4376 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 04:04:12.362607 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.360846 4376 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 04:04:12.362607 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.360849 4376 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 04:04:12.362607 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.360852 4376 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 04:04:12.362607 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.360854 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 04:04:12.363099 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.360857 4376 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 04:04:12.363099 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.360859 4376 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 04:04:12.363099 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.360862 4376 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 04:04:12.363099 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.360865 4376 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 04:04:12.363099 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.360867 4376 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 04:04:12.363099 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.360870 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 04:04:12.363099 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.360874 4376 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 04:04:12.363099 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.361704 4376 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 04:04:12.363099 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.361711 4376 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 04:04:12.363099 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.361714 4376 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 04:04:12.363099 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.361717 4376 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 04:04:12.363099 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.361720 4376 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 04:04:12.363099 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.361722 4376 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 04:04:12.363099 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.361725 4376 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 04:04:12.363099 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.361727 4376 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 04:04:12.363099 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.361730 4376 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 04:04:12.363099 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.361733 4376 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 04:04:12.363099 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.361735 4376 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 04:04:12.363099 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.361738 4376 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 04:04:12.363099 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.361740 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 04:04:12.363646 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.361743 4376 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 04:04:12.363646 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.361746 4376 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 04:04:12.363646 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.361750 4376 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 04:04:12.363646 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.361753 4376 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 04:04:12.363646 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.361757 4376 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 04:04:12.363646 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.361759 4376 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 04:04:12.363646 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.361763 4376 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 04:04:12.363646 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.361766 4376 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 04:04:12.363646 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.361769 4376 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 04:04:12.363646 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.361771 4376 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 04:04:12.363646 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.361774 4376 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 04:04:12.363646 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.361777 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 04:04:12.363646 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.361779 4376 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 04:04:12.363646 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.361782 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 04:04:12.363646 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.361785 4376 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 04:04:12.363646 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.361788 4376 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 04:04:12.363646 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.361790 4376 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 04:04:12.363646 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.361793 4376 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 04:04:12.363646 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.361795 4376 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 04:04:12.364106 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.361798 4376 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 04:04:12.364106 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.361801 4376 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 04:04:12.364106 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.361804 4376 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 04:04:12.364106 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.361807 4376 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 04:04:12.364106 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.361810 4376 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 04:04:12.364106 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.361813 4376 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 04:04:12.364106 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.361816 4376 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 04:04:12.364106 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.361818 4376 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 04:04:12.364106 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.361820 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 04:04:12.364106 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.361823 4376 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 04:04:12.364106 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.361826 4376 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 04:04:12.364106 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.361828 4376 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 04:04:12.364106 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.361831 4376 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 04:04:12.364106 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.361833 4376 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 04:04:12.364106 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.361836 4376 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 04:04:12.364106 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.361838 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 04:04:12.364106 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.361841 4376 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 04:04:12.364106 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.361843 4376 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 04:04:12.364106 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.361846 4376 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 04:04:12.364106 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.361854 4376 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 04:04:12.364639 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.361857 4376 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 04:04:12.364639 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.361859 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 04:04:12.364639 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.361862 4376 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 04:04:12.364639 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.361864 4376 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 04:04:12.364639 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.361867 4376 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 04:04:12.364639 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.361870 4376 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 04:04:12.364639 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.361872 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 04:04:12.364639 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.361875 4376 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 04:04:12.364639 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.361877 4376 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 04:04:12.364639 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.361880 4376 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 04:04:12.364639 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.361883 4376 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 04:04:12.364639 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.361885 4376 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 04:04:12.364639 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.361888 4376 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 04:04:12.364639 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.361890 4376 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 04:04:12.364639 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.361894 4376 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 04:04:12.364639 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.361896 4376 feature_gate.go:386] unrecognized feature gate: Example Oct 31 04:04:12.364639 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.361899 4376 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 04:04:12.364639 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.361903 4376 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 04:04:12.364639 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.361905 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 04:04:12.364639 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.361909 4376 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 04:04:12.365150 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.361913 4376 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 04:04:12.365150 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.361917 4376 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 04:04:12.365150 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.361920 4376 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 04:04:12.365150 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.361922 4376 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 04:04:12.365150 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.361925 4376 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 04:04:12.365150 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.361927 4376 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 04:04:12.365150 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.361930 4376 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 04:04:12.365150 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.361933 4376 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 04:04:12.365150 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.361935 4376 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 04:04:12.365150 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.361938 4376 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 04:04:12.365150 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.361940 4376 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 04:04:12.365150 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.361945 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 04:04:12.365150 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.361948 4376 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 04:04:12.365150 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.361950 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 04:04:12.365150 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.361953 4376 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 04:04:12.365150 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.361955 4376 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 04:04:12.365150 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.361958 4376 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 04:04:12.365150 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.361961 4376 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 04:04:12.365150 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.361963 4376 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 04:04:12.365150 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.361966 4376 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 04:04:12.365656 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.361968 4376 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 04:04:12.365656 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.361971 4376 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 04:04:12.365656 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.361974 4376 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 04:04:12.365656 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.361977 4376 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 04:04:12.365656 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.361979 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 04:04:12.365656 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.361982 4376 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 04:04:12.365656 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.361985 4376 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 04:04:12.365656 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.361987 4376 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 04:04:12.365656 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.361991 4376 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 04:04:12.365656 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.361994 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 04:04:12.365656 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.361996 4376 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 04:04:12.365656 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.361999 4376 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 04:04:12.365656 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.362001 4376 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 04:04:12.365656 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.362004 4376 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 04:04:12.365656 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.362547 4376 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 04:04:12.365656 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.362553 4376 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 04:04:12.365656 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.362556 4376 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 04:04:12.365656 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.362559 4376 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 04:04:12.365656 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.362562 4376 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 04:04:12.366111 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.362564 4376 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 04:04:12.366111 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.362567 4376 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 04:04:12.366111 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.362570 4376 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 04:04:12.366111 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.362573 4376 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 04:04:12.366111 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.362577 4376 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 04:04:12.366111 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.362581 4376 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 04:04:12.366111 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.362583 4376 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 04:04:12.366111 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.362586 4376 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 04:04:12.366111 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.362589 4376 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 04:04:12.366111 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.362592 4376 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 04:04:12.366111 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.362594 4376 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 04:04:12.366111 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.362597 4376 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 04:04:12.366111 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.362599 4376 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 04:04:12.366111 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.362602 4376 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 04:04:12.366111 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.362604 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 04:04:12.366111 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.362607 4376 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 04:04:12.366111 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.362610 4376 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 04:04:12.366111 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.362612 4376 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 04:04:12.366111 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.362615 4376 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 04:04:12.366111 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.362619 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 04:04:12.366684 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.362622 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 04:04:12.366684 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.362624 4376 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 04:04:12.366684 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.362627 4376 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 04:04:12.366684 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.362630 4376 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 04:04:12.366684 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.362632 4376 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 04:04:12.366684 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.362635 4376 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 04:04:12.366684 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.362638 4376 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 04:04:12.366684 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.362640 4376 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 04:04:12.366684 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.362644 4376 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 04:04:12.366684 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.362648 4376 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 04:04:12.366684 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.362652 4376 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 04:04:12.366684 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.362654 4376 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 04:04:12.366684 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.362657 4376 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 04:04:12.366684 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.362659 4376 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 04:04:12.366684 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.362662 4376 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 04:04:12.366684 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.362665 4376 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 04:04:12.366684 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.362668 4376 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 04:04:12.366684 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.362671 4376 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 04:04:12.366684 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.362674 4376 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 04:04:12.367526 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.362676 4376 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 04:04:12.367526 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.362679 4376 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 04:04:12.367526 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.362681 4376 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 04:04:12.367526 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.362684 4376 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 04:04:12.367526 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.362686 4376 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 04:04:12.367526 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.362689 4376 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 04:04:12.367526 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.362691 4376 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 04:04:12.367526 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.362694 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 04:04:12.367526 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.362697 4376 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 04:04:12.367526 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.362699 4376 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 04:04:12.367526 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.362701 4376 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 04:04:12.367526 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.362704 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 04:04:12.367526 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.362707 4376 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 04:04:12.367526 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.362710 4376 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 04:04:12.367526 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.362712 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 04:04:12.367526 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.362715 4376 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 04:04:12.367526 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.362718 4376 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 04:04:12.367526 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.362720 4376 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 04:04:12.367526 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.362722 4376 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 04:04:12.367526 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.362725 4376 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 04:04:12.368427 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.362727 4376 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 04:04:12.368427 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.362730 4376 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 04:04:12.368427 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.362732 4376 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 04:04:12.368427 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.362735 4376 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 04:04:12.368427 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.362737 4376 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 04:04:12.368427 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.362740 4376 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 04:04:12.368427 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.362743 4376 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 04:04:12.368427 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.362745 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 04:04:12.368427 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.362748 4376 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 04:04:12.368427 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.362752 4376 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 04:04:12.368427 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.362755 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 04:04:12.368427 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.362759 4376 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 04:04:12.368427 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.362762 4376 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 04:04:12.368427 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.362765 4376 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 04:04:12.368427 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.362768 4376 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 04:04:12.368427 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.362770 4376 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 04:04:12.368427 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.362773 4376 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 04:04:12.368427 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.362776 4376 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 04:04:12.368427 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.362778 4376 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 04:04:12.369249 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.362780 4376 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 04:04:12.369249 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.362783 4376 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 04:04:12.369249 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.362785 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 04:04:12.369249 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.362788 4376 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 04:04:12.369249 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.362790 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 04:04:12.369249 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.362795 4376 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 04:04:12.369249 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.362797 4376 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 04:04:12.369249 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.362800 4376 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 04:04:12.369249 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.362802 4376 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 04:04:12.369249 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.362805 4376 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 04:04:12.369249 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.362808 4376 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 04:04:12.369249 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.362810 4376 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 04:04:12.369249 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.362813 4376 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 04:04:12.369249 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.362816 4376 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 04:04:12.369249 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.362819 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 04:04:12.369249 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.362821 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 04:04:12.369249 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.362824 4376 feature_gate.go:386] unrecognized feature gate: Example Oct 31 04:04:12.369249 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.362826 4376 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 04:04:12.369249 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.362829 4376 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 04:04:12.369249 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.362831 4376 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 04:04:12.370138 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.362835 4376 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 04:04:12.370138 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.362837 4376 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 04:04:12.370138 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.362840 4376 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 04:04:12.370138 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.362943 4376 flags.go:64] FLAG: --address="0.0.0.0" Oct 31 04:04:12.370138 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.362955 4376 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 31 04:04:12.370138 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.362966 4376 flags.go:64] FLAG: --anonymous-auth="true" Oct 31 04:04:12.370138 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.362972 4376 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 31 04:04:12.370138 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.362976 4376 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 31 04:04:12.370138 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.362979 4376 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 31 04:04:12.370138 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.362984 4376 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 31 04:04:12.370138 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.362988 4376 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 31 04:04:12.370138 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.362991 4376 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 31 04:04:12.370138 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.362995 4376 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 31 04:04:12.370138 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.362998 4376 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 31 04:04:12.370138 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.363001 4376 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 31 04:04:12.370138 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.363005 4376 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 31 04:04:12.370138 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.363008 4376 flags.go:64] FLAG: --cgroup-root="" Oct 31 04:04:12.370138 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.363016 4376 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 31 04:04:12.370138 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.363019 4376 flags.go:64] FLAG: --client-ca-file="" Oct 31 04:04:12.370138 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.363022 4376 flags.go:64] FLAG: --cloud-provider="external" Oct 31 04:04:12.370138 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.363025 4376 flags.go:64] FLAG: --cluster-dns="[]" Oct 31 04:04:12.370138 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.363029 4376 flags.go:64] FLAG: --cluster-domain="" Oct 31 04:04:12.370138 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.363033 4376 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 31 04:04:12.371163 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.363036 4376 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 31 04:04:12.371163 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.363039 4376 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 31 04:04:12.371163 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.363042 4376 flags.go:64] FLAG: --container-log-max-files="5" Oct 31 04:04:12.371163 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.363046 4376 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 31 04:04:12.371163 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.363049 4376 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 31 04:04:12.371163 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.363052 4376 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 31 04:04:12.371163 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.363056 4376 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 31 04:04:12.371163 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.363060 4376 flags.go:64] FLAG: --contention-profiling="false" Oct 31 04:04:12.371163 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.363063 4376 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 31 04:04:12.371163 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.363066 4376 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 31 04:04:12.371163 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.363069 4376 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 31 04:04:12.371163 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.363072 4376 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 31 04:04:12.371163 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.363076 4376 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 31 04:04:12.371163 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.363080 4376 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 31 04:04:12.371163 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.363083 4376 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 31 04:04:12.371163 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.363086 4376 flags.go:64] FLAG: --enable-load-reader="false" Oct 31 04:04:12.371163 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.363089 4376 flags.go:64] FLAG: --enable-server="true" Oct 31 04:04:12.371163 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.363092 4376 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 31 04:04:12.371163 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.363098 4376 flags.go:64] FLAG: --event-burst="100" Oct 31 04:04:12.371163 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.363101 4376 flags.go:64] FLAG: --event-qps="50" Oct 31 04:04:12.371163 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.363104 4376 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 31 04:04:12.371163 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.363108 4376 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 31 04:04:12.371163 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.363111 4376 flags.go:64] FLAG: --eviction-hard="" Oct 31 04:04:12.371163 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.363115 4376 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 31 04:04:12.372132 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.363118 4376 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 31 04:04:12.372132 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.363122 4376 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 31 04:04:12.372132 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.363126 4376 flags.go:64] FLAG: --eviction-soft="" Oct 31 04:04:12.372132 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.363129 4376 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 31 04:04:12.372132 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.363132 4376 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 31 04:04:12.372132 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.363135 4376 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 31 04:04:12.372132 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.363141 4376 flags.go:64] FLAG: --experimental-mounter-path="" Oct 31 04:04:12.372132 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.363144 4376 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 31 04:04:12.372132 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.363147 4376 flags.go:64] FLAG: --fail-swap-on="true" Oct 31 04:04:12.372132 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.363150 4376 flags.go:64] FLAG: --feature-gates="" Oct 31 04:04:12.372132 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.363154 4376 flags.go:64] FLAG: --file-check-frequency="20s" Oct 31 04:04:12.372132 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.363158 4376 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 31 04:04:12.372132 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.363161 4376 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 31 04:04:12.372132 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.363165 4376 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 31 04:04:12.372132 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.363168 4376 flags.go:64] FLAG: --healthz-port="10248" Oct 31 04:04:12.372132 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.363171 4376 flags.go:64] FLAG: --help="false" Oct 31 04:04:12.372132 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.363175 4376 flags.go:64] FLAG: --hostname-override="ip-10-0-134-155.ec2.internal" Oct 31 04:04:12.372132 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.363178 4376 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 31 04:04:12.372132 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.363181 4376 flags.go:64] FLAG: --http-check-frequency="20s" Oct 31 04:04:12.372132 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.363184 4376 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 31 04:04:12.372132 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.363187 4376 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 31 04:04:12.372132 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.363191 4376 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 31 04:04:12.372132 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.363194 4376 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 31 04:04:12.372132 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.363197 4376 flags.go:64] FLAG: --image-service-endpoint="" Oct 31 04:04:12.372906 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.363200 4376 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 31 04:04:12.372906 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.363204 4376 flags.go:64] FLAG: --kube-api-burst="100" Oct 31 04:04:12.372906 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.363207 4376 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 31 04:04:12.372906 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.363210 4376 flags.go:64] FLAG: --kube-api-qps="50" Oct 31 04:04:12.372906 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.363213 4376 flags.go:64] FLAG: --kube-reserved="" Oct 31 04:04:12.372906 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.363216 4376 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 31 04:04:12.372906 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.363219 4376 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 31 04:04:12.372906 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.363222 4376 flags.go:64] FLAG: --kubelet-cgroups="" Oct 31 04:04:12.372906 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.363225 4376 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 31 04:04:12.372906 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.363228 4376 flags.go:64] FLAG: --lock-file="" Oct 31 04:04:12.372906 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.363232 4376 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 31 04:04:12.372906 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.363235 4376 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 31 04:04:12.372906 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.363238 4376 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 31 04:04:12.372906 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.363244 4376 flags.go:64] FLAG: --log-json-split-stream="false" Oct 31 04:04:12.372906 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.363248 4376 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 31 04:04:12.372906 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.363251 4376 flags.go:64] FLAG: --log-text-split-stream="false" Oct 31 04:04:12.372906 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.363254 4376 flags.go:64] FLAG: --logging-format="text" Oct 31 04:04:12.372906 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.363258 4376 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 31 04:04:12.372906 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.363261 4376 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 31 04:04:12.372906 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.363264 4376 flags.go:64] FLAG: --manifest-url="" Oct 31 04:04:12.372906 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.363267 4376 flags.go:64] FLAG: --manifest-url-header="" Oct 31 04:04:12.372906 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.363272 4376 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 31 04:04:12.372906 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.363276 4376 flags.go:64] FLAG: --max-open-files="1000000" Oct 31 04:04:12.372906 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.363280 4376 flags.go:64] FLAG: --max-pods="110" Oct 31 04:04:12.372906 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.363283 4376 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 31 04:04:12.373595 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.363286 4376 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 31 04:04:12.373595 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.363289 4376 flags.go:64] FLAG: --memory-manager-policy="None" Oct 31 04:04:12.373595 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.363292 4376 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 31 04:04:12.373595 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.363295 4376 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 31 04:04:12.373595 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.363298 4376 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 31 04:04:12.373595 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.363301 4376 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 31 04:04:12.373595 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.363307 4376 flags.go:64] FLAG: --node-status-max-images="50" Oct 31 04:04:12.373595 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.363310 4376 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 31 04:04:12.373595 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.363314 4376 flags.go:64] FLAG: --oom-score-adj="-999" Oct 31 04:04:12.373595 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.363318 4376 flags.go:64] FLAG: --pod-cidr="" Oct 31 04:04:12.373595 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.363320 4376 flags.go:64] FLAG: --pod-manifest-path="" Oct 31 04:04:12.373595 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.363323 4376 flags.go:64] FLAG: --pod-max-pids="-1" Oct 31 04:04:12.373595 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.363339 4376 flags.go:64] FLAG: --pods-per-core="0" Oct 31 04:04:12.373595 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.363344 4376 flags.go:64] FLAG: --port="10250" Oct 31 04:04:12.373595 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.363348 4376 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 31 04:04:12.373595 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.363368 4376 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-06236bc10ca20d74b" Oct 31 04:04:12.373595 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.363372 4376 flags.go:64] FLAG: --qos-reserved="" Oct 31 04:04:12.373595 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.363375 4376 flags.go:64] FLAG: --read-only-port="10255" Oct 31 04:04:12.373595 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.363380 4376 flags.go:64] FLAG: --register-node="true" Oct 31 04:04:12.373595 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.363383 4376 flags.go:64] FLAG: --register-with-taints="" Oct 31 04:04:12.373595 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.363387 4376 flags.go:64] FLAG: --registry-burst="10" Oct 31 04:04:12.373595 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.363392 4376 flags.go:64] FLAG: --registry-qps="5" Oct 31 04:04:12.373595 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.363395 4376 flags.go:64] FLAG: --reserved-cpus="" Oct 31 04:04:12.373595 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.363398 4376 flags.go:64] FLAG: --reserved-memory="" Oct 31 04:04:12.373595 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.363402 4376 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 31 04:04:12.373595 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.363405 4376 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 31 04:04:12.374256 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.363409 4376 flags.go:64] FLAG: --rotate-certificates="false" Oct 31 04:04:12.374256 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.363412 4376 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 31 04:04:12.374256 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.363416 4376 flags.go:64] FLAG: --runonce="false" Oct 31 04:04:12.374256 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.363418 4376 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 31 04:04:12.374256 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.363422 4376 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 31 04:04:12.374256 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.363425 4376 flags.go:64] FLAG: --seccomp-default="false" Oct 31 04:04:12.374256 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.363428 4376 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 31 04:04:12.374256 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.363431 4376 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 31 04:04:12.374256 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.363435 4376 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 31 04:04:12.374256 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.363438 4376 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 31 04:04:12.374256 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.363441 4376 flags.go:64] FLAG: --storage-driver-password="root" Oct 31 04:04:12.374256 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.363444 4376 flags.go:64] FLAG: --storage-driver-secure="false" Oct 31 04:04:12.374256 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.363447 4376 flags.go:64] FLAG: --storage-driver-table="stats" Oct 31 04:04:12.374256 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.363451 4376 flags.go:64] FLAG: --storage-driver-user="root" Oct 31 04:04:12.374256 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.363453 4376 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 31 04:04:12.374256 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.363457 4376 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 31 04:04:12.374256 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.363460 4376 flags.go:64] FLAG: --system-cgroups="" Oct 31 04:04:12.374256 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.363463 4376 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 31 04:04:12.374256 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.363469 4376 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 31 04:04:12.374256 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.363472 4376 flags.go:64] FLAG: --tls-cert-file="" Oct 31 04:04:12.374256 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.363475 4376 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 31 04:04:12.374256 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.363479 4376 flags.go:64] FLAG: --tls-min-version="" Oct 31 04:04:12.374256 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.363482 4376 flags.go:64] FLAG: --tls-private-key-file="" Oct 31 04:04:12.374256 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.363485 4376 flags.go:64] FLAG: --topology-manager-policy="none" Oct 31 04:04:12.374256 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.363489 4376 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 31 04:04:12.374928 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.363493 4376 flags.go:64] FLAG: --topology-manager-scope="container" Oct 31 04:04:12.374928 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.363496 4376 flags.go:64] FLAG: --v="2" Oct 31 04:04:12.374928 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.363502 4376 flags.go:64] FLAG: --version="false" Oct 31 04:04:12.374928 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.363506 4376 flags.go:64] FLAG: --vmodule="" Oct 31 04:04:12.374928 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.363511 4376 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 31 04:04:12.374928 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.363514 4376 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 31 04:04:12.374928 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.363679 4376 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 04:04:12.374928 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.363684 4376 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 04:04:12.374928 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.363687 4376 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 04:04:12.374928 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.363690 4376 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 04:04:12.374928 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.363693 4376 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 04:04:12.374928 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.363696 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 04:04:12.374928 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.363699 4376 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 04:04:12.374928 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.363702 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 04:04:12.374928 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.363705 4376 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 04:04:12.374928 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.363707 4376 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 04:04:12.374928 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.363711 4376 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 04:04:12.374928 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.363715 4376 feature_gate.go:386] unrecognized feature gate: Example Oct 31 04:04:12.374928 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.363730 4376 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 04:04:12.374928 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.363735 4376 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 04:04:12.374928 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.363738 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 04:04:12.375456 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.363741 4376 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 04:04:12.375456 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.363744 4376 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 04:04:12.375456 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.363748 4376 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 04:04:12.375456 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.363751 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 04:04:12.375456 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.363754 4376 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 04:04:12.375456 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.363757 4376 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 04:04:12.375456 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.363760 4376 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 04:04:12.375456 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.363763 4376 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 04:04:12.375456 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.363765 4376 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 04:04:12.375456 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.363768 4376 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 04:04:12.375456 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.363772 4376 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 04:04:12.375456 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.363775 4376 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 04:04:12.375456 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.363778 4376 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 04:04:12.375456 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.363783 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 04:04:12.375456 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.363785 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 04:04:12.375456 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.363788 4376 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 04:04:12.375456 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.363790 4376 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 04:04:12.375456 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.363793 4376 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 04:04:12.375456 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.363796 4376 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 04:04:12.375949 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.363798 4376 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 04:04:12.375949 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.363801 4376 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 04:04:12.375949 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.363803 4376 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 04:04:12.375949 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.363806 4376 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 04:04:12.375949 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.363808 4376 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 04:04:12.375949 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.363811 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 04:04:12.375949 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.363814 4376 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 04:04:12.375949 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.363817 4376 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 04:04:12.375949 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.363819 4376 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 04:04:12.375949 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.363822 4376 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 04:04:12.375949 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.363824 4376 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 04:04:12.375949 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.363827 4376 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 04:04:12.375949 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.363829 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 04:04:12.375949 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.363832 4376 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 04:04:12.375949 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.363835 4376 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 04:04:12.375949 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.363837 4376 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 04:04:12.375949 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.363840 4376 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 04:04:12.375949 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.363843 4376 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 04:04:12.375949 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.363845 4376 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 04:04:12.375949 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.363848 4376 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 04:04:12.376617 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.363851 4376 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 04:04:12.376617 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.363854 4376 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 04:04:12.376617 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.363856 4376 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 04:04:12.376617 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.363860 4376 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 04:04:12.376617 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.363863 4376 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 04:04:12.376617 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.363865 4376 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 04:04:12.376617 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.363870 4376 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 04:04:12.376617 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.363872 4376 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 04:04:12.376617 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.363875 4376 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 04:04:12.376617 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.363878 4376 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 04:04:12.376617 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.363880 4376 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 04:04:12.376617 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.363883 4376 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 04:04:12.376617 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.363885 4376 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 04:04:12.376617 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.363888 4376 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 04:04:12.376617 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.363890 4376 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 04:04:12.376617 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.363893 4376 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 04:04:12.376617 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.363895 4376 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 04:04:12.376617 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.363898 4376 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 04:04:12.376617 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.363902 4376 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 04:04:12.376617 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.363906 4376 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 04:04:12.377279 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.363909 4376 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 04:04:12.377279 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.363912 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 04:04:12.377279 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.363915 4376 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 04:04:12.377279 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.363917 4376 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 04:04:12.377279 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.363920 4376 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 04:04:12.377279 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.363923 4376 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 04:04:12.377279 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.363926 4376 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 04:04:12.377279 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.363928 4376 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 04:04:12.377279 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.363931 4376 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 04:04:12.377279 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.363934 4376 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 04:04:12.377279 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.363937 4376 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 04:04:12.377279 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.363939 4376 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 04:04:12.377279 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.363942 4376 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 04:04:12.377279 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.363944 4376 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 04:04:12.377279 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.363947 4376 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 04:04:12.377279 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.363951 4376 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 04:04:12.377279 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.363953 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 04:04:12.377279 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.363956 4376 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 04:04:12.377279 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.363960 4376 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 04:04:12.377915 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.363963 4376 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 04:04:12.377915 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.363966 4376 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 04:04:12.377915 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.363968 4376 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 04:04:12.377915 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.363971 4376 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 04:04:12.377915 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.363974 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 04:04:12.377915 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.363976 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 04:04:12.377915 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.363979 4376 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 04:04:12.377915 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.363982 4376 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 04:04:12.377915 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.363984 4376 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 04:04:12.377915 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.363987 4376 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 04:04:12.377915 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.363989 4376 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 04:04:12.377915 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.363992 4376 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 04:04:12.377915 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.363994 4376 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 04:04:12.377915 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.364022 4376 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 04:04:12.377915 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.364063 4376 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 04:04:12.377915 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.364069 4376 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 04:04:12.377915 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.364072 4376 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 04:04:12.378323 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.364075 4376 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 04:04:12.378323 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.364077 4376 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 04:04:12.378323 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.364080 4376 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 04:04:12.378323 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.364082 4376 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 04:04:12.378323 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.364085 4376 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 04:04:12.378323 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.364088 4376 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 04:04:12.378323 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.364091 4376 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 04:04:12.378323 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.364094 4376 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 04:04:12.378323 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.364096 4376 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 04:04:12.378323 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.364100 4376 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 04:04:12.378323 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.364104 4376 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 04:04:12.378323 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.364109 4376 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 04:04:12.378323 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.364112 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 04:04:12.378323 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.364115 4376 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 04:04:12.378323 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.364118 4376 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 04:04:12.378323 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.364122 4376 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 04:04:12.378323 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.364125 4376 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 04:04:12.378323 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.364128 4376 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 04:04:12.378323 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.364130 4376 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 04:04:12.378826 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.364133 4376 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 04:04:12.378826 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.364135 4376 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 04:04:12.378826 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.364138 4376 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 04:04:12.378826 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.364140 4376 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 04:04:12.378826 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.364143 4376 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 04:04:12.378826 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.364146 4376 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 04:04:12.378826 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.364148 4376 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 04:04:12.378826 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.364151 4376 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 04:04:12.378826 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.364153 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 04:04:12.378826 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.364156 4376 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 04:04:12.378826 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.364159 4376 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 04:04:12.378826 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.364161 4376 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 04:04:12.378826 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.364164 4376 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 04:04:12.378826 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.364166 4376 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 04:04:12.378826 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.364169 4376 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 04:04:12.378826 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.364172 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 04:04:12.378826 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.364174 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 04:04:12.378826 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.364177 4376 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 04:04:12.378826 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.364179 4376 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 04:04:12.378826 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.364182 4376 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 04:04:12.379497 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.364185 4376 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 04:04:12.379497 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.364188 4376 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 04:04:12.379497 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.364190 4376 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 04:04:12.379497 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.364193 4376 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 04:04:12.379497 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.364196 4376 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 04:04:12.379497 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.364199 4376 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 04:04:12.379497 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.364201 4376 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 04:04:12.379497 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.364204 4376 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 04:04:12.379497 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.364208 4376 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 04:04:12.379497 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.364210 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 04:04:12.379497 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.364213 4376 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 04:04:12.379497 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.364216 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 04:04:12.379497 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.364218 4376 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 04:04:12.379497 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.364221 4376 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 04:04:12.379497 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.364225 4376 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 04:04:12.379497 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.364229 4376 feature_gate.go:386] unrecognized feature gate: Example Oct 31 04:04:12.379497 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.364232 4376 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 04:04:12.379497 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.364235 4376 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 04:04:12.379497 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.364237 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 04:04:12.379497 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.364240 4376 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 04:04:12.380016 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.364243 4376 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 04:04:12.380016 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.364245 4376 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 04:04:12.380016 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.364248 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 04:04:12.380016 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.364251 4376 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 04:04:12.380016 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.364254 4376 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 04:04:12.380016 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.364257 4376 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 04:04:12.380016 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.364260 4376 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 04:04:12.380016 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.364262 4376 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 04:04:12.380016 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.364265 4376 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 04:04:12.380016 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.364268 4376 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 04:04:12.380016 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.364270 4376 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 04:04:12.380016 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.364273 4376 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 04:04:12.380016 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.364276 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 04:04:12.380016 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.364279 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 04:04:12.380016 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.364281 4376 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 04:04:12.380016 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.364284 4376 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 04:04:12.380016 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.364287 4376 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 04:04:12.380016 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.364289 4376 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 04:04:12.380016 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.364292 4376 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 04:04:12.380545 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.364294 4376 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 04:04:12.380545 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.364299 4376 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 04:04:12.380545 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.364301 4376 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 04:04:12.380545 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.364304 4376 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 04:04:12.380545 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.364307 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 04:04:12.380545 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.364309 4376 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 04:04:12.380545 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.364312 4376 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 04:04:12.380545 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.364314 4376 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 04:04:12.380545 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.364317 4376 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 04:04:12.380545 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.364320 4376 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 04:04:12.380545 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.364322 4376 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 04:04:12.380545 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.364339 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 04:04:12.380545 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.364342 4376 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 04:04:12.380545 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.364344 4376 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 04:04:12.380545 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.364347 4376 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 04:04:12.380545 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.364350 4376 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 04:04:12.380545 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.364352 4376 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 04:04:12.380545 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.364355 4376 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 04:04:12.380545 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.364358 4376 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 04:04:12.380545 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.364360 4376 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 04:04:12.381033 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.364364 4376 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 04:04:12.381033 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.364366 4376 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 04:04:12.381033 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.364369 4376 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 04:04:12.381033 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.364371 4376 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 04:04:12.381033 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.364374 4376 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 04:04:12.381033 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.369708 4376 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 31 04:04:12.381033 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.369732 4376 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 31 04:04:12.381033 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.369757 4376 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 31 04:04:12.381033 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.369765 4376 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 31 04:04:12.381033 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.369869 4376 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 04:04:12.381033 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.369879 4376 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 04:04:12.381033 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.369885 4376 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 04:04:12.381033 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.369891 4376 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 04:04:12.381033 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.369895 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 04:04:12.381033 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.369899 4376 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 04:04:12.381033 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.369904 4376 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 04:04:12.381033 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.369916 4376 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 04:04:12.381033 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.369920 4376 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 04:04:12.381033 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.369925 4376 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 04:04:12.381033 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.369930 4376 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 04:04:12.381675 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.369942 4376 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 04:04:12.381675 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.369946 4376 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 04:04:12.381675 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.369950 4376 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 04:04:12.381675 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.369955 4376 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 04:04:12.381675 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.369959 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 04:04:12.381675 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.369963 4376 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 04:04:12.381675 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.369968 4376 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 04:04:12.381675 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.369972 4376 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 04:04:12.381675 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.369977 4376 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 04:04:12.381675 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.369982 4376 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 04:04:12.381675 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.369986 4376 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 04:04:12.381675 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.369990 4376 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 04:04:12.381675 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.370017 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 04:04:12.381675 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.370024 4376 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 04:04:12.381675 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.370031 4376 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 04:04:12.381675 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.370037 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 04:04:12.381675 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.370042 4376 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 04:04:12.381675 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.370047 4376 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 04:04:12.381675 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.370053 4376 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 04:04:12.382129 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.370059 4376 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 04:04:12.382129 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.370066 4376 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 04:04:12.382129 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.370071 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 04:04:12.382129 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.370075 4376 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 04:04:12.382129 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.370079 4376 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 04:04:12.382129 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.370083 4376 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 04:04:12.382129 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.370087 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 04:04:12.382129 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.370091 4376 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 04:04:12.382129 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.370095 4376 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 04:04:12.382129 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.370100 4376 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 04:04:12.382129 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.370105 4376 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 04:04:12.382129 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.370110 4376 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 04:04:12.382129 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.370114 4376 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 04:04:12.382129 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.370118 4376 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 04:04:12.382129 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.370130 4376 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 04:04:12.382129 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.370135 4376 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 04:04:12.382129 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.370139 4376 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 04:04:12.382129 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.370143 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 04:04:12.382129 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.370147 4376 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 04:04:12.382612 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.370151 4376 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 04:04:12.382612 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.370155 4376 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 04:04:12.382612 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.370159 4376 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 04:04:12.382612 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.370163 4376 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 04:04:12.382612 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.370167 4376 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 04:04:12.382612 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.370172 4376 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 04:04:12.382612 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.370177 4376 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 04:04:12.382612 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.370182 4376 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 04:04:12.382612 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.370186 4376 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 04:04:12.382612 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.370190 4376 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 04:04:12.382612 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.370195 4376 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 04:04:12.382612 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.370199 4376 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 04:04:12.382612 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.370203 4376 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 04:04:12.382612 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.370207 4376 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 04:04:12.382612 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.370211 4376 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 04:04:12.382612 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.370215 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 04:04:12.382612 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.370220 4376 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 04:04:12.382612 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.370224 4376 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 04:04:12.382612 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.370229 4376 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 04:04:12.383075 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.370233 4376 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 04:04:12.383075 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.370237 4376 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 04:04:12.383075 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.370241 4376 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 04:04:12.383075 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.370246 4376 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 04:04:12.383075 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.370250 4376 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 04:04:12.383075 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.370254 4376 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 04:04:12.383075 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.370258 4376 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 04:04:12.383075 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.370262 4376 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 04:04:12.383075 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.370267 4376 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 04:04:12.383075 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.370279 4376 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 04:04:12.383075 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.370283 4376 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 04:04:12.383075 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.370287 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 04:04:12.383075 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.370294 4376 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 04:04:12.383075 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.370300 4376 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 04:04:12.383075 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.370304 4376 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 04:04:12.383075 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.370309 4376 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 04:04:12.383075 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.370313 4376 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 04:04:12.383075 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.370317 4376 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 04:04:12.383075 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.370322 4376 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 04:04:12.383558 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.370342 4376 feature_gate.go:386] unrecognized feature gate: Example Oct 31 04:04:12.383558 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.370347 4376 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 04:04:12.383558 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.370351 4376 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 04:04:12.383558 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.370355 4376 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 04:04:12.383558 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.370359 4376 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 04:04:12.383558 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.370364 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 04:04:12.383558 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.370367 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 04:04:12.383558 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.370372 4376 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 04:04:12.383558 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.370376 4376 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 04:04:12.383558 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.370381 4376 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 04:04:12.383558 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.370385 4376 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 04:04:12.383558 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.370389 4376 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 04:04:12.383558 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.370393 4376 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 04:04:12.383558 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.370398 4376 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 04:04:12.383558 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.370402 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 04:04:12.383558 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.370406 4376 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 04:04:12.383558 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.370410 4376 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 04:04:12.383558 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.370415 4376 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 04:04:12.383558 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.370419 4376 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 04:04:12.384042 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.370463 4376 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 04:04:12.384042 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.371172 4376 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 04:04:12.384042 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.371187 4376 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 04:04:12.384042 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.371193 4376 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 04:04:12.384042 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.371226 4376 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 04:04:12.384042 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.371232 4376 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 04:04:12.384042 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.371237 4376 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 04:04:12.384042 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.371242 4376 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 04:04:12.384042 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.371253 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 04:04:12.384042 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.371258 4376 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 04:04:12.384042 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.371262 4376 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 04:04:12.384042 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.371267 4376 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 04:04:12.384042 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.371273 4376 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 04:04:12.384042 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.371278 4376 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 04:04:12.384042 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.371283 4376 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 04:04:12.384042 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.371288 4376 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 04:04:12.384552 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.371293 4376 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 04:04:12.384552 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.371298 4376 feature_gate.go:386] unrecognized feature gate: Example Oct 31 04:04:12.384552 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.371302 4376 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 04:04:12.384552 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.371307 4376 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 04:04:12.384552 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.371311 4376 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 04:04:12.384552 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.371321 4376 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 04:04:12.384552 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.371340 4376 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 04:04:12.384552 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.371346 4376 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 04:04:12.384552 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.371350 4376 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 04:04:12.384552 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.371355 4376 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 04:04:12.384552 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.371359 4376 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 04:04:12.384552 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.371364 4376 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 04:04:12.384552 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.371369 4376 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 04:04:12.384552 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.371373 4376 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 04:04:12.384552 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.371378 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 04:04:12.384552 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.371383 4376 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 04:04:12.384552 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.371387 4376 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 04:04:12.384552 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.371398 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 04:04:12.384552 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.371403 4376 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 04:04:12.384552 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.371407 4376 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 04:04:12.385141 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.371412 4376 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 04:04:12.385141 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.371446 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 04:04:12.385141 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.371452 4376 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 04:04:12.385141 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.371456 4376 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 04:04:12.385141 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.371461 4376 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 04:04:12.385141 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.371465 4376 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 04:04:12.385141 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.371469 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 04:04:12.385141 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.371474 4376 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 04:04:12.385141 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.371478 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 04:04:12.385141 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.371484 4376 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 04:04:12.385141 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.371494 4376 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 04:04:12.385141 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.371499 4376 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 04:04:12.385141 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.371504 4376 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 04:04:12.385141 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.371508 4376 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 04:04:12.385141 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.371513 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 04:04:12.385141 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.371517 4376 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 04:04:12.385141 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.371521 4376 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 04:04:12.385141 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.371525 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 04:04:12.385141 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.371530 4376 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 04:04:12.385141 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.371534 4376 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 04:04:12.385790 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.371538 4376 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 04:04:12.385790 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.371543 4376 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 04:04:12.385790 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.371556 4376 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 04:04:12.385790 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.371564 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 04:04:12.385790 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.371570 4376 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 04:04:12.385790 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.371575 4376 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 04:04:12.385790 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.371580 4376 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 04:04:12.385790 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.371585 4376 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 04:04:12.385790 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.371589 4376 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 04:04:12.385790 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.371594 4376 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 04:04:12.385790 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.371598 4376 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 04:04:12.385790 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.371603 4376 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 04:04:12.385790 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.371607 4376 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 04:04:12.385790 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.371612 4376 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 04:04:12.385790 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.371644 4376 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 04:04:12.385790 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.371655 4376 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 04:04:12.385790 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.371660 4376 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 04:04:12.385790 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.371665 4376 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 04:04:12.385790 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.371670 4376 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 04:04:12.386509 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.371675 4376 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 04:04:12.386509 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.371679 4376 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 04:04:12.386509 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.371685 4376 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 04:04:12.386509 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.371689 4376 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 04:04:12.386509 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.371694 4376 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 04:04:12.386509 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.371701 4376 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 04:04:12.386509 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.371707 4376 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 04:04:12.386509 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.371712 4376 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 04:04:12.386509 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.371722 4376 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 04:04:12.386509 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.371727 4376 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 04:04:12.386509 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.371732 4376 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 04:04:12.386509 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.371736 4376 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 04:04:12.386509 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.371741 4376 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 04:04:12.386509 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.371745 4376 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 04:04:12.386509 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.371749 4376 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 04:04:12.386509 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.371754 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 04:04:12.386509 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.371759 4376 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 04:04:12.386509 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.371763 4376 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 04:04:12.386509 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.371768 4376 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 04:04:12.386509 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.371772 4376 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 04:04:12.387249 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.371777 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 04:04:12.387249 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.371786 4376 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 04:04:12.387249 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.371791 4376 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 04:04:12.387249 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.371795 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 04:04:12.387249 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.371800 4376 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 04:04:12.387249 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.371804 4376 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 04:04:12.387249 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.371809 4376 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 04:04:12.387249 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.371813 4376 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 04:04:12.387249 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.371845 4376 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 04:04:12.387249 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.371851 4376 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 04:04:12.387249 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.371856 4376 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 04:04:12.387249 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.371860 4376 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 04:04:12.387249 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.372503 4376 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 04:04:12.387249 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.372584 4376 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 04:04:12.387249 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.372591 4376 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 04:04:12.387249 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.372594 4376 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 04:04:12.387837 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.372598 4376 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 04:04:12.387837 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.372601 4376 feature_gate.go:386] unrecognized feature gate: Example Oct 31 04:04:12.387837 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.372604 4376 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 04:04:12.387837 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.372607 4376 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 04:04:12.387837 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.372609 4376 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 04:04:12.387837 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.372612 4376 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 04:04:12.387837 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.372615 4376 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 04:04:12.387837 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.372618 4376 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 04:04:12.387837 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.372620 4376 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 04:04:12.387837 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.372623 4376 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 04:04:12.387837 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.372625 4376 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 04:04:12.387837 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.372628 4376 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 04:04:12.387837 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.372631 4376 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 04:04:12.387837 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.372634 4376 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 04:04:12.387837 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.372637 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 04:04:12.387837 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.372639 4376 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 04:04:12.387837 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.372644 4376 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 04:04:12.387837 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.372647 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 04:04:12.387837 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.372650 4376 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 04:04:12.387837 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.372653 4376 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 04:04:12.388339 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.372655 4376 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 04:04:12.388339 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.372658 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 04:04:12.388339 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.372661 4376 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 04:04:12.388339 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.372664 4376 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 04:04:12.388339 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.372666 4376 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 04:04:12.388339 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.372670 4376 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 04:04:12.388339 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.372672 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 04:04:12.388339 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.372676 4376 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 04:04:12.388339 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.372678 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 04:04:12.388339 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.372682 4376 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 04:04:12.388339 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.372684 4376 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 04:04:12.388339 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.372687 4376 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 04:04:12.388339 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.372690 4376 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 04:04:12.388339 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.372693 4376 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 04:04:12.388339 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.372695 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 04:04:12.388339 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.372698 4376 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 04:04:12.388339 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.372700 4376 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 04:04:12.388339 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.372703 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 04:04:12.388339 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.372706 4376 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 04:04:12.388339 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.372708 4376 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 04:04:12.388891 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.372711 4376 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 04:04:12.388891 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.372721 4376 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 04:04:12.388891 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.372727 4376 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 04:04:12.388891 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.372732 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 04:04:12.388891 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.372735 4376 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 04:04:12.388891 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.372738 4376 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 04:04:12.388891 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.372740 4376 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 04:04:12.388891 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.372743 4376 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 04:04:12.388891 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.372746 4376 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 04:04:12.388891 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.372748 4376 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 04:04:12.388891 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.372751 4376 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 04:04:12.388891 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.372754 4376 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 04:04:12.388891 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.372757 4376 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 04:04:12.388891 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.372759 4376 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 04:04:12.388891 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.372762 4376 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 04:04:12.388891 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.372765 4376 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 04:04:12.388891 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.372768 4376 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 04:04:12.388891 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.372771 4376 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 04:04:12.388891 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.372774 4376 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 04:04:12.389474 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.372776 4376 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 04:04:12.389474 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.372779 4376 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 04:04:12.389474 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.372782 4376 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 04:04:12.389474 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.372785 4376 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 04:04:12.389474 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.372787 4376 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 04:04:12.389474 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.372791 4376 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 04:04:12.389474 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.372795 4376 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 04:04:12.389474 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.372799 4376 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 04:04:12.389474 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.372802 4376 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 04:04:12.389474 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.372805 4376 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 04:04:12.389474 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.372807 4376 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 04:04:12.389474 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.372810 4376 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 04:04:12.389474 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.372812 4376 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 04:04:12.389474 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.372816 4376 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 04:04:12.389474 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.372819 4376 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 04:04:12.389474 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.372822 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 04:04:12.389474 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.372824 4376 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 04:04:12.389474 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.372827 4376 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 04:04:12.389474 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.372830 4376 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 04:04:12.389474 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.372832 4376 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 04:04:12.390242 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.372835 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 04:04:12.390242 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.372837 4376 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 04:04:12.390242 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.372840 4376 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 04:04:12.390242 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.372843 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 04:04:12.390242 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.372845 4376 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 04:04:12.390242 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.372848 4376 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 04:04:12.390242 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.372850 4376 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 04:04:12.390242 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.372853 4376 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 04:04:12.390242 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.372856 4376 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 04:04:12.390242 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.372858 4376 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 04:04:12.390242 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.372861 4376 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 04:04:12.390242 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.372864 4376 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 04:04:12.390242 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.372866 4376 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 04:04:12.390242 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.372869 4376 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 04:04:12.390242 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.372872 4376 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 04:04:12.390242 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.372875 4376 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 04:04:12.390242 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.372877 4376 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 04:04:12.390242 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.372880 4376 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 04:04:12.390242 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.372883 4376 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 04:04:12.390808 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.372886 4376 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 04:04:12.390808 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.372889 4376 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 04:04:12.390808 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.372891 4376 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 04:04:12.390808 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.372894 4376 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 04:04:12.390808 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:12.372898 4376 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 04:04:12.390808 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.372964 4376 server.go:957] "Client rotation is on, will bootstrap in background" Oct 31 04:04:12.390808 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.374027 4376 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 31 04:04:12.390808 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.374111 4376 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 31 04:04:12.390808 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.374292 4376 server.go:1014] "Starting client certificate rotation" Oct 31 04:04:12.390808 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.374443 4376 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 31 04:04:12.390808 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.374495 4376 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-30 03:58:33 +0000 UTC" deadline="2028-07-09 14:56:16.014382846 +0000 UTC" Oct 31 04:04:12.390808 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.374524 4376 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14818h52m3.639861344s" Oct 31 04:04:12.390808 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.374796 4376 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 04:04:12.390808 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.375905 4376 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 04:04:12.390808 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.377266 4376 log.go:25] "Validated CRI v1 runtime API" Oct 31 04:04:12.390808 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.378739 4376 log.go:25] "Validated CRI v1 image API" Oct 31 04:04:12.391207 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.379499 4376 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 31 04:04:12.391207 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.381022 4376 fs.go:135] Filesystem UUIDs: map[55f77d54-5511-4a13-b35b-75d0d3b4bf71:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 850455b8-6530-41bf-bdbc-00680ae5e0be:/dev/nvme0n1p3] Oct 31 04:04:12.391285 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.381045 4376 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1ddb915eccc0737c404aca07c5758b96a9f0fdd59c1cf88f059dae0e94b737d9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1ddb915eccc0737c404aca07c5758b96a9f0fdd59c1cf88f059dae0e94b737d9/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5f9774c0d0ed14f1f2a9fe916e5c2762491290e9c194e57e3d8d9dd40b8ed118/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5f9774c0d0ed14f1f2a9fe916e5c2762491290e9c194e57e3d8d9dd40b8ed118/userdata/shm major:0 minor:74 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/601b281d91973258a35641bcf167e3607780239835331d37842df6ccb98916d9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/601b281d91973258a35641bcf167e3607780239835331d37842df6ccb98916d9/userdata/shm major:0 minor:202 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8022cc9c6ccbd005a0ecdab2ba41a52a5e4c05e4b51c3c7f012c5cd630f6341b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8022cc9c6ccbd005a0ecdab2ba41a52a5e4c05e4b51c3c7f012c5cd630f6341b/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a2f924ac02df888acb1b969d58be1f782b5eae0d396b10c804368d942460fcd5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a2f924ac02df888acb1b969d58be1f782b5eae0d396b10c804368d942460fcd5/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a69bc663b12a32d16dcb8255d451bb0ace44a40d73d2144fdd60c44368664668/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a69bc663b12a32d16dcb8255d451bb0ace44a40d73d2144fdd60c44368664668/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cdec06bf47f6a54fe3f2dd0b5ffb405a4dad6a7395313767f517d814dd709cc5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cdec06bf47f6a54fe3f2dd0b5ffb405a4dad6a7395313767f517d814dd709cc5/userdata/shm major:0 minor:69 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/dfb45165dd6714f2e2e521be1f2c11d492273d10525ed8974ea1c0601f249a0e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/dfb45165dd6714f2e2e521be1f2c11d492273d10525ed8974ea1c0601f249a0e/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ea0ff4d5abf545a0400b791d9d5625dc627feb17b9582ee4ecc29c8827acf93a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ea0ff4d5abf545a0400b791d9d5625dc627feb17b9582ee4ecc29c8827acf93a/userdata/shm major:0 minor:87 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ee1850a84839fb35a22fdac4cdb23ad16a64f08240a18fadbc54cccb1e3aeb61/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ee1850a84839fb35a22fdac4cdb23ad16a64f08240a18fadbc54cccb1e3aeb61/userdata/shm major:0 minor:206 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f16830d905f69c2c30a7e8fef47baa94b6a2db1b4880d62ab6f0b7895f17e68e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f16830d905f69c2c30a7e8fef47baa94b6a2db1b4880d62ab6f0b7895f17e68e/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f56712820eeb59c980c23b370ff8d528b4b9a17d25b621931a79cad3343585e5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f56712820eeb59c980c23b370ff8d528b4b9a17d25b621931a79cad3343585e5/userdata/shm major:0 minor:204 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f66a0ba4c2917e14250e724b408ee86370622c0619ec3bd5ab230906b66c353a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f66a0ba4c2917e14250e724b408ee86370622c0619ec3bd5ab230906b66c353a/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/153e979c-3c5d-4c4f-b838-fc26db0b5cac/volumes/kubernetes.io~projected/kube-api-access-vp84q:{mountpoint:/var/lib/kubelet/pods/153e979c-3c5d-4c4f-b838-fc26db0b5cac/volumes/kubernetes.io~projected/kube-api-access-vp84q major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1f2543cc-b528-41be-92e0-9d654405a723/volumes/kubernetes.io~projected/kube-api-access-fxgq7:{mountpoint:/var/lib/kubelet/pods/1f2543cc-b528-41be-92e0-9d654405a723/volumes/kubernetes.io~projected/kube-api-access-fxgq7 major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/24e22999-2c81-4779-a901-5b9faa15c1cb/volumes/kubernetes.io~projected/kube-api-access-v2zgx:{mountpoint:/var/lib/kubelet/pods/24e22999-2c81-4779-a901-5b9faa15c1cb/volumes/kubernetes.io~projected/kube-api-access-v2zgx major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/79ae1195-4ff1-46da-b1e2-83042dba6caa/volumes/kubernetes.io~projected/kube-api-access-wgdpm:{mountpoint:/var/lib/kubelet/pods/79ae1195-4ff1-46da-b1e2-83042dba6caa/volumes/kubernetes.io~projected/kube-api-access-wgdpm major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a90157bb-46c0-4afa-82b1-9f6819518744/volumes/kubernetes.io~projected/kube-api-access-786gp:{mountpoint:/var/lib/kubelet/pods/a90157bb-46c0-4afa-82b1-9f6819518744/volumes/kubernetes.io~projected/kube-api-access-786gp major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bc9d2733-89fa-4079-9dc1-01983df773cb/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/bc9d2733-89fa-4079-9dc1-01983df773cb/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c581e590-87e5-4cd7-8d84-eabb4ee21fd1/volumes/kubernetes.io~projected/kube-api-access-wwltl:{mountpoint:/var/lib/kubelet/pods/c581e590-87e5-4cd7-8d84-eabb4ee21fd1/volumes/kubernetes.io~projected/kube-api-access-wwltl major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d4eabffb-abf1-4409-94b5-c0489ab37853/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/d4eabffb-abf1-4409-94b5-c0489ab37853/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d4eabffb-abf1-4409-94b5-c0489ab37853/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/d4eabffb-abf1-4409-94b5-c0489ab37853/volumes/kubernetes.io~empty-dir/tmp major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d4eabffb-abf1-4409-94b5-c0489ab37853/volumes/kubernetes.io~projected/kube-api-access-n54k7:{mountpoint:/var/lib/kubelet/pods/d4eabffb-abf1-4409-94b5-c0489ab37853/volumes/kubernetes.io~projected/kube-api-access-n54k7 major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e7f09238-2bbf-4587-9989-49419d90dabf/volumes/kubernetes.io~projected/kube-api-access-9r7fx:{mountpoint:/var/lib/kubelet/pods/e7f09238-2bbf-4587-9989-49419d90dabf/volumes/kubernetes.io~projected/kube-api-access-9r7fx major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e7f09238-2bbf-4587-9989-49419d90dabf/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/e7f09238-2bbf-4587-9989-49419d90dabf/volumes/kubernetes.io~secret/metrics-certs major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f12d4ae1-e673-4a9e-bd43-92d6e9d5a023/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/f12d4ae1-e673-4a9e-bd43-92d6e9d5a023/volumes/kubernetes.io~secret/agent-certs major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f70f5eab-445f-430e-bfb1-92e4d620b90d/volumes/kubernetes.io~projected/kube-api-access-jq5vs:{mountpoint:/var/lib/kubelet/pods/f70f5eab-445f-430e-bfb1-92e4d620b90d/volumes/kubernetes.io~projected/kube-api-access-jq5vs major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f7a16817-8ea2-4649-9e42-3d340e1b9703/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/f7a16817-8ea2-4649-9e42-3d340e1b9703/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f7a16817-8ea2-4649-9e42-3d340e1b9703/volumes/kubernetes.io~projected/kube-api-access-dkkp9:{mountpoint:/var/lib/kubelet/pods/f7a16817-8ea2-4649-9e42-3d340e1b9703/volumes/kubernetes.io~projected/kube-api-access-dkkp9 major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f7a16817-8ea2-4649-9e42-3d340e1b9703/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/f7a16817-8ea2-4649-9e42-3d340e1b9703/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:58 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/9e74b1a2153f65ee7b2ec9a70b8a4cf6a47df856b96533102fb77a5f92e017d7/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/611a7385fcf59e07eed54c2814a11d1b5396f415d9af72a3626740db19d92ab0/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/ed6048bec1f5d6bf3d161ee8fc7a49cca987b9fa3fb0557e3790befed8b7c55a/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/74934ae05a97732513f7ca6328cb404d66a28ad1e820c7c7bc1ae12634acd2c1/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/589c752e7c28e7478356cf4497657612ca0227a1f84163c4da00d086690df2b0/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/0c926fd6a7a6b9c08928b88d8e50fef07736b6a9b8c54890f186aa6957a75b72/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/b2c6f1bcb44d367175ce93295326799d71d28f16151c842dfb378dfd5da77702/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/3654561a59a601a4df62d226bada87214cea5b2b5c46de789178b1c2f75d2a88/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/2f2e0ba224c3288c23d02b5c565298690d499d393054957df1e25a25da05121f/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/7e4a078959765ce838b547a3f8bb86e3d7ef56c03f03ae95a10d618e3fc71c80/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/5a59288505792949290bfc96996b011a5289fdf1c5694d3825316055d4e45075/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/20b1abe7d96aecf62a25da74435d4f2cf0cabecd314ea53dcfc8abf03a37a83d/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/a7c463c696a62cbf5cef364499dce53d94a5be288cb90b7621b2fc7f573d258f/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-158:{mountpoint:/var/lib/containers/storage/overlay/6f3c91b487ef7fee7db357a4a05a88c2f3950aefaab4f2ae3b608e5a069c87ba/merged major:0 minor:158 fsType:overlay blockSize:0} overlay_0-163:{mountpoint:/var/lib/containers/storage/overlay/d46f42751e61a3737ed2b45522964800e1f6a3d76ac990b003d6abede3d7fe91/merged major:0 minor:163 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/dcb149f40cc32348c662df42b0175ea8ce3561baf222a3633ba9132dab59f8a1/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/adaf70fde1932140bc79ce80b19789c5979ea67d73ef8e481d33d5e55b0fd535/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/e52fa91d67e00cfe57d5f9bd928da4c7ce5967f7071d0d91b3b0966221e010a3/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/50d2d61b183c1e93367e7913cf54f84309986ba7283c576e0204bb481c93e4a7/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/e39d46c5474d599bced7bf382eb0cb5024987f2e841f34a08b1933ae4a190648/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/9599cdc453693ce0b159aec889fc7395dd75ea85b76d678fca096689fab0c360/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-208:{mountpoint:/var/lib/containers/storage/overlay/75ccb34c1dc233e2f7dea36c164071dbbf017dd1f662a3935033fae059aaf880/merged major:0 minor:208 fsType:overlay blockSize:0} overlay_0-210:{mountpoint:/var/lib/containers/storage/overlay/ddd5e2b91266d373b16e86b8ce273346931794172d94da2d7dd6b7fdf89d4bfb/merged major:0 minor:210 fsType:overlay blockSize:0} overlay_0-212:{mountpoint:/var/lib/containers/storage/overlay/809ac88735bd34f6173041b8c33e41634ea991cb7565ce171831b6c72833cea6/merged major:0 minor:212 fsType:overlay blockSize:0} overlay_0-214:{mountpoint:/var/lib/containers/storage/overlay/0e2de66b1189d9fa5f8565e71c5c64709a1393e29c6e7944dd5a470bb59021c2/merged major:0 minor:214 fsType:overlay blockSize:0} overlay_0-216:{mountpoint:/var/lib/containers/storage/overlay/fda73ead5633ea167238bafab712731f525e71f8df088dd58363fca76ab10b9f/merged major:0 minor:216 fsType:overlay blockSize:0} overlay_0-222:{mountpoint:/var/lib/containers/storage/overlay/744ab857da8a64ea0fd823b9f8baf4575696e26c34b9bf75fb8900c46a3e3e2b/merged major:0 minor:222 fsType:overlay blockSize:0} overlay_0-227:{mountpoint:/var/lib/containers/storage/overlay/830c5cee2b654bf538e40d49be3fbda41c29718e98504843dd548d82ddbaf174/merged major:0 minor:227 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/c23286770ac39bf27a51799ead38716d743e6ab79dfeea277574b6f2fb624843/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/7c536804b5f5e403aa871ec8322bf853e80a96f267f1563649d3da99f81d66d2/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-71:{mountpoint:/var/lib/containers/storage/overlay/debc56ca80a67859d381f606ec35bc9140f95ce6e64fa77023d82dab4fcf039e/merged major:0 minor:71 fsType:overlay blockSize:0} overlay_0-75:{mountpoint:/var/lib/containers/storage/overlay/b400ecd51c8013353a859176ffc375c521f8b3bae5a0651ba92de786803887fa/merged major:0 minor:75 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/36b9d3d2a3f2bd004fdb32efde520254327a21d4ffe08e55e901cd1585570f76/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/78fe3a3f64b3b67319d8d922259bdff371192ce7e9b0a72c9970815ad87a5167/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/ac5f2382de818da09482aee95e51c3d61bd6022b185e08ecd4a93731ca6b5c16/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/90b033abcf35e8a6df51d53bf3d918a7432841f98499b0ef5b93193b0fd2a207/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/d869ecb8d560826bbc0d52f82415eb9c32155d8ef3b10027877044e1725da908/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/5cc4a8a5ad01fd9d7adbdb8e9681e0560991aa7ab652f3a2916e3d6ac7736a30/merged major:0 minor:96 fsType:overlay blockSize:0}] Oct 31 04:04:12.391446 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.388471 4376 manager.go:217] Machine: {Timestamp:2026-10-31 04:04:12.38779349 +0000 UTC m=+0.102128098 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099010 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec22d5b32bd50a250ea3be956df55a11 SystemUUID:ec22d5b3-2bd5-0a25-0ea3-be956df55a11 BootID:665524b7-0d7c-415e-90cc-cb508656d751 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/79ae1195-4ff1-46da-b1e2-83042dba6caa/volumes/kubernetes.io~projected/kube-api-access-wgdpm DeviceMajor:0 DeviceMinor:64 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d4eabffb-abf1-4409-94b5-c0489ab37853/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:55 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f7a16817-8ea2-4649-9e42-3d340e1b9703/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:58 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-158 DeviceMajor:0 DeviceMinor:158 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f12d4ae1-e673-4a9e-bd43-92d6e9d5a023/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:56 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-222 DeviceMajor:0 DeviceMinor:222 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/24e22999-2c81-4779-a901-5b9faa15c1cb/volumes/kubernetes.io~projected/kube-api-access-v2zgx DeviceMajor:0 DeviceMinor:63 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d4eabffb-abf1-4409-94b5-c0489ab37853/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:57 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/1f2543cc-b528-41be-92e0-9d654405a723/volumes/kubernetes.io~projected/kube-api-access-fxgq7 DeviceMajor:0 DeviceMinor:67 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f56712820eeb59c980c23b370ff8d528b4b9a17d25b621931a79cad3343585e5/userdata/shm DeviceMajor:0 DeviceMinor:204 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-214 DeviceMajor:0 DeviceMinor:214 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a2f924ac02df888acb1b969d58be1f782b5eae0d396b10c804368d942460fcd5/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1ddb915eccc0737c404aca07c5758b96a9f0fdd59c1cf88f059dae0e94b737d9/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-75 DeviceMajor:0 DeviceMinor:75 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-163 DeviceMajor:0 DeviceMinor:163 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-210 DeviceMajor:0 DeviceMinor:210 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a90157bb-46c0-4afa-82b1-9f6819518744/volumes/kubernetes.io~projected/kube-api-access-786gp DeviceMajor:0 DeviceMinor:61 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e7f09238-2bbf-4587-9989-49419d90dabf/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:197 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-212 DeviceMajor:0 DeviceMinor:212 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8022cc9c6ccbd005a0ecdab2ba41a52a5e4c05e4b51c3c7f012c5cd630f6341b/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f7a16817-8ea2-4649-9e42-3d340e1b9703/volumes/kubernetes.io~projected/kube-api-access-dkkp9 DeviceMajor:0 DeviceMinor:62 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d4eabffb-abf1-4409-94b5-c0489ab37853/volumes/kubernetes.io~projected/kube-api-access-n54k7 DeviceMajor:0 DeviceMinor:65 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ea0ff4d5abf545a0400b791d9d5625dc627feb17b9582ee4ecc29c8827acf93a/userdata/shm DeviceMajor:0 DeviceMinor:87 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f7a16817-8ea2-4649-9e42-3d340e1b9703/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5f9774c0d0ed14f1f2a9fe916e5c2762491290e9c194e57e3d8d9dd40b8ed118/userdata/shm DeviceMajor:0 DeviceMinor:74 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f70f5eab-445f-430e-bfb1-92e4d620b90d/volumes/kubernetes.io~projected/kube-api-access-jq5vs DeviceMajor:0 DeviceMinor:201 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/601b281d91973258a35641bcf167e3607780239835331d37842df6ccb98916d9/userdata/shm DeviceMajor:0 DeviceMinor:202 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-227 DeviceMajor:0 DeviceMinor:227 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cdec06bf47f6a54fe3f2dd0b5ffb405a4dad6a7395313767f517d814dd709cc5/userdata/shm DeviceMajor:0 DeviceMinor:69 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/153e979c-3c5d-4c4f-b838-fc26db0b5cac/volumes/kubernetes.io~projected/kube-api-access-vp84q DeviceMajor:0 DeviceMinor:66 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-71 DeviceMajor:0 DeviceMinor:71 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bc9d2733-89fa-4079-9dc1-01983df773cb/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:45 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c581e590-87e5-4cd7-8d84-eabb4ee21fd1/volumes/kubernetes.io~projected/kube-api-access-wwltl DeviceMajor:0 DeviceMinor:59 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f16830d905f69c2c30a7e8fef47baa94b6a2db1b4880d62ab6f0b7895f17e68e/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f66a0ba4c2917e14250e724b408ee86370622c0619ec3bd5ab230906b66c353a/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-208 DeviceMajor:0 DeviceMinor:208 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-216 DeviceMajor:0 DeviceMinor:216 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ee1850a84839fb35a22fdac4cdb23ad16a64f08240a18fadbc54cccb1e3aeb61/userdata/shm DeviceMajor:0 DeviceMinor:206 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/dfb45165dd6714f2e2e521be1f2c11d492273d10525ed8974ea1c0601f249a0e/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e7f09238-2bbf-4587-9989-49419d90dabf/volumes/kubernetes.io~projected/kube-api-access-9r7fx DeviceMajor:0 DeviceMinor:60 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a69bc663b12a32d16dcb8255d451bb0ace44a40d73d2144fdd60c44368664668/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:601b281d9197325 MacAddress:e2:ff:e2:a8:69:72 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:c9:f3:37:cd Speed:0 Mtu:9001} {Name:br-int MacAddress:be:ab:fe:11:d9:56 Speed:0 Mtu:8901} {Name:ee1850a84839fb3 MacAddress:b6:01:5c:5c:42:3c Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:c9:f3:37:cd Speed:0 Mtu:9001} {Name:f56712820eeb59c MacAddress:f6:42:55:6c:33:ec Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:36:66:9c:de:4b:49 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:36:74:41:0b:f7:0e Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 31 04:04:12.391446 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.388823 4376 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 31 04:04:12.391446 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.388885 4376 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 31 04:04:12.391446 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.390728 4376 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 31 04:04:12.391446 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.390765 4376 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-155.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 31 04:04:12.391446 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.390905 4376 topology_manager.go:143] "Creating topology manager with none policy" Oct 31 04:04:12.391446 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.390914 4376 container_manager_linux.go:311] "Creating device plugin manager" Oct 31 04:04:12.391446 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.390928 4376 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 04:04:12.391446 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.390942 4376 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 04:04:12.391446 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.391051 4376 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 31 04:04:12.391446 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.391222 4376 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 31 04:04:12.391446 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.391356 4376 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 31 04:04:12.391446 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.391392 4376 kubelet.go:493] "Attempting to sync node with API server" Oct 31 04:04:12.391446 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.391405 4376 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 31 04:04:12.391446 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.391415 4376 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 31 04:04:12.391446 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.391423 4376 kubelet.go:405] "Adding apiserver pod source" Oct 31 04:04:12.391446 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.391431 4376 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 31 04:04:12.392201 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.392185 4376 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 31 04:04:12.392201 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.392202 4376 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 04:04:12.392938 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.392914 4376 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 31 04:04:12.394871 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.394856 4376 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 31 04:04:12.394871 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.394870 4376 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 04:04:12.394998 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.394965 4376 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 31 04:04:12.395239 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.395227 4376 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 31 04:04:12.395268 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.395257 4376 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 31 04:04:12.395368 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.395359 4376 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 31 04:04:12.395406 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.395373 4376 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 31 04:04:12.395406 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.395379 4376 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 31 04:04:12.395406 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.395385 4376 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 31 04:04:12.395406 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.395391 4376 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 31 04:04:12.395406 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.395396 4376 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 31 04:04:12.395406 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.395402 4376 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 31 04:04:12.395406 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.395408 4376 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 31 04:04:12.395610 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.395415 4376 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 31 04:04:12.395610 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.395421 4376 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 31 04:04:12.395610 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.395429 4376 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 31 04:04:12.395610 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.395438 4376 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 31 04:04:12.395610 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.395464 4376 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 31 04:04:12.395610 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.395470 4376 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 31 04:04:12.398733 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.398718 4376 server.go:1263] "Started kubelet" Oct 31 04:04:12.398822 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.398799 4376 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 31 04:04:12.398874 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.398804 4376 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 31 04:04:12.398923 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.398877 4376 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 31 04:04:12.399273 ip-10-0-134-155 systemd[1]: Started Kubernetes Kubelet. Oct 31 04:04:12.399447 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.399297 4376 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 31 04:04:12.401366 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.399899 4376 server.go:317] "Adding debug handlers to kubelet server" Oct 31 04:04:12.403305 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.403286 4376 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 31 04:04:12.404841 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.403304 4376 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 31 04:04:12.404841 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.403402 4376 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 03:58:34 +0000 UTC" deadline="2028-07-20 11:18:29.860373872 +0000 UTC" Oct 31 04:04:12.404841 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.403461 4376 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15079h14m17.456916677s" Oct 31 04:04:12.404841 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:04:12.403524 4376 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-155.ec2.internal\" not found" Oct 31 04:04:12.404841 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.403558 4376 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 31 04:04:12.404841 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.403568 4376 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 31 04:04:12.404841 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.403684 4376 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 31 04:04:12.409473 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.406059 4376 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f7a16817-8ea2-4649-9e42-3d340e1b9703" volumeName="kubernetes.io/configmap/f7a16817-8ea2-4649-9e42-3d340e1b9703-env-overrides" seLinuxMountContext="" Oct 31 04:04:12.409473 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.406103 4376 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f7a16817-8ea2-4649-9e42-3d340e1b9703" volumeName="kubernetes.io/configmap/f7a16817-8ea2-4649-9e42-3d340e1b9703-ovnkube-script-lib" seLinuxMountContext="" Oct 31 04:04:12.409473 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.406109 4376 factory.go:55] Registering systemd factory Oct 31 04:04:12.409473 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.406124 4376 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f7a16817-8ea2-4649-9e42-3d340e1b9703" volumeName="kubernetes.io/projected/f7a16817-8ea2-4649-9e42-3d340e1b9703-kube-api-access-dkkp9" seLinuxMountContext="" Oct 31 04:04:12.409473 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.406140 4376 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="153e979c-3c5d-4c4f-b838-fc26db0b5cac" volumeName="kubernetes.io/configmap/153e979c-3c5d-4c4f-b838-fc26db0b5cac-cni-binary-copy" seLinuxMountContext="" Oct 31 04:04:12.409473 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.406153 4376 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1f2543cc-b528-41be-92e0-9d654405a723" volumeName="kubernetes.io/projected/1f2543cc-b528-41be-92e0-9d654405a723-kube-api-access-fxgq7" seLinuxMountContext="" Oct 31 04:04:12.409473 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.406200 4376 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="24e22999-2c81-4779-a901-5b9faa15c1cb" volumeName="kubernetes.io/configmap/24e22999-2c81-4779-a901-5b9faa15c1cb-iptables-alerter-script" seLinuxMountContext="" Oct 31 04:04:12.409473 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.406215 4376 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="79ae1195-4ff1-46da-b1e2-83042dba6caa" volumeName="kubernetes.io/configmap/79ae1195-4ff1-46da-b1e2-83042dba6caa-cni-binary-copy" seLinuxMountContext="" Oct 31 04:04:12.409473 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.406231 4376 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="79ae1195-4ff1-46da-b1e2-83042dba6caa" volumeName="kubernetes.io/projected/79ae1195-4ff1-46da-b1e2-83042dba6caa-kube-api-access-wgdpm" seLinuxMountContext="" Oct 31 04:04:12.409473 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.406247 4376 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f7a16817-8ea2-4649-9e42-3d340e1b9703" volumeName="kubernetes.io/configmap/f7a16817-8ea2-4649-9e42-3d340e1b9703-ovnkube-config" seLinuxMountContext="" Oct 31 04:04:12.409473 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.406259 4376 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f7a16817-8ea2-4649-9e42-3d340e1b9703" volumeName="kubernetes.io/secret/f7a16817-8ea2-4649-9e42-3d340e1b9703-ovn-node-metrics-cert" seLinuxMountContext="" Oct 31 04:04:12.409473 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.406406 4376 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="153e979c-3c5d-4c4f-b838-fc26db0b5cac" volumeName="kubernetes.io/projected/153e979c-3c5d-4c4f-b838-fc26db0b5cac-kube-api-access-vp84q" seLinuxMountContext="" Oct 31 04:04:12.409473 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.406417 4376 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="24e22999-2c81-4779-a901-5b9faa15c1cb" volumeName="kubernetes.io/projected/24e22999-2c81-4779-a901-5b9faa15c1cb-kube-api-access-v2zgx" seLinuxMountContext="" Oct 31 04:04:12.409473 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.406431 4376 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="79ae1195-4ff1-46da-b1e2-83042dba6caa" volumeName="kubernetes.io/configmap/79ae1195-4ff1-46da-b1e2-83042dba6caa-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 31 04:04:12.409473 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.406447 4376 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bc9d2733-89fa-4079-9dc1-01983df773cb" volumeName="kubernetes.io/secret/bc9d2733-89fa-4079-9dc1-01983df773cb-original-pull-secret" seLinuxMountContext="" Oct 31 04:04:12.409473 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.406528 4376 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e7f09238-2bbf-4587-9989-49419d90dabf" volumeName="kubernetes.io/projected/e7f09238-2bbf-4587-9989-49419d90dabf-kube-api-access-9r7fx" seLinuxMountContext="" Oct 31 04:04:12.409473 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.406539 4376 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e7f09238-2bbf-4587-9989-49419d90dabf" volumeName="kubernetes.io/secret/e7f09238-2bbf-4587-9989-49419d90dabf-metrics-certs" seLinuxMountContext="" Oct 31 04:04:12.409473 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.406550 4376 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a90157bb-46c0-4afa-82b1-9f6819518744" volumeName="kubernetes.io/configmap/a90157bb-46c0-4afa-82b1-9f6819518744-serviceca" seLinuxMountContext="" Oct 31 04:04:12.409473 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.406561 4376 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a90157bb-46c0-4afa-82b1-9f6819518744" volumeName="kubernetes.io/projected/a90157bb-46c0-4afa-82b1-9f6819518744-kube-api-access-786gp" seLinuxMountContext="" Oct 31 04:04:12.409473 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.406574 4376 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="153e979c-3c5d-4c4f-b838-fc26db0b5cac" volumeName="kubernetes.io/configmap/153e979c-3c5d-4c4f-b838-fc26db0b5cac-multus-daemon-config" seLinuxMountContext="" Oct 31 04:04:12.409473 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.406128 4376 factory.go:223] Registration of the systemd container factory successfully Oct 31 04:04:12.409473 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.406583 4376 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1f2543cc-b528-41be-92e0-9d654405a723" volumeName="kubernetes.io/empty-dir/1f2543cc-b528-41be-92e0-9d654405a723-tmp-dir" seLinuxMountContext="" Oct 31 04:04:12.409473 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.406765 4376 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="79ae1195-4ff1-46da-b1e2-83042dba6caa" volumeName="kubernetes.io/configmap/79ae1195-4ff1-46da-b1e2-83042dba6caa-cni-sysctl-allowlist" seLinuxMountContext="" Oct 31 04:04:12.409473 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.406783 4376 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d4eabffb-abf1-4409-94b5-c0489ab37853" volumeName="kubernetes.io/empty-dir/d4eabffb-abf1-4409-94b5-c0489ab37853-etc-tuned" seLinuxMountContext="" Oct 31 04:04:12.409473 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.406804 4376 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d4eabffb-abf1-4409-94b5-c0489ab37853" volumeName="kubernetes.io/empty-dir/d4eabffb-abf1-4409-94b5-c0489ab37853-tmp" seLinuxMountContext="" Oct 31 04:04:12.409473 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.406822 4376 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f12d4ae1-e673-4a9e-bd43-92d6e9d5a023" volumeName="kubernetes.io/configmap/f12d4ae1-e673-4a9e-bd43-92d6e9d5a023-konnectivity-ca" seLinuxMountContext="" Oct 31 04:04:12.409473 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.406835 4376 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c581e590-87e5-4cd7-8d84-eabb4ee21fd1" volumeName="kubernetes.io/projected/c581e590-87e5-4cd7-8d84-eabb4ee21fd1-kube-api-access-wwltl" seLinuxMountContext="" Oct 31 04:04:12.409473 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.406859 4376 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d4eabffb-abf1-4409-94b5-c0489ab37853" volumeName="kubernetes.io/projected/d4eabffb-abf1-4409-94b5-c0489ab37853-kube-api-access-n54k7" seLinuxMountContext="" Oct 31 04:04:12.409473 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.406873 4376 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f12d4ae1-e673-4a9e-bd43-92d6e9d5a023" volumeName="kubernetes.io/secret/f12d4ae1-e673-4a9e-bd43-92d6e9d5a023-agent-certs" seLinuxMountContext="" Oct 31 04:04:12.409473 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.406893 4376 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f70f5eab-445f-430e-bfb1-92e4d620b90d" volumeName="kubernetes.io/projected/f70f5eab-445f-430e-bfb1-92e4d620b90d-kube-api-access-jq5vs" seLinuxMountContext="" Oct 31 04:04:12.409473 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.406919 4376 reconstruct.go:97] "Volume reconstruction finished" Oct 31 04:04:12.409473 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.406927 4376 reconciler.go:29] "Reconciler: start to sync state" Oct 31 04:04:12.409473 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.407154 4376 factory.go:153] Registering CRI-O factory Oct 31 04:04:12.409473 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.407181 4376 factory.go:223] Registration of the crio container factory successfully Oct 31 04:04:12.409473 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.407246 4376 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 31 04:04:12.409473 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.407266 4376 factory.go:103] Registering Raw factory Oct 31 04:04:12.409473 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.407287 4376 manager.go:1199] Started watching for new ooms in manager Oct 31 04:04:12.409473 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.407735 4376 manager.go:319] Starting recovery of all containers Oct 31 04:04:12.433736 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.433680 4376 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 31 04:04:12.435214 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.435181 4376 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 31 04:04:12.435214 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.435214 4376 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 31 04:04:12.435404 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.435239 4376 kubelet.go:2521] "Starting kubelet main sync loop" Oct 31 04:04:12.435404 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:04:12.435279 4376 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 31 04:04:12.443946 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.443912 4376 generic.go:356] "Generic (PLEG): container finished" podID="79ae1195-4ff1-46da-b1e2-83042dba6caa" containerID="ec7d07cb450cc1b873d8f9c7780382a3ea5bfd6c1f99b7e07bf439191856ae2c" exitCode=0 Oct 31 04:04:12.443946 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.443942 4376 generic.go:356] "Generic (PLEG): container finished" podID="79ae1195-4ff1-46da-b1e2-83042dba6caa" containerID="907eb0f93f1109e8efd49fd6a207e71979c33b2bf22da0fa84e6ecee553bbe93" exitCode=0 Oct 31 04:04:12.444130 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.443953 4376 generic.go:356] "Generic (PLEG): container finished" podID="79ae1195-4ff1-46da-b1e2-83042dba6caa" containerID="501a3129f65614bb455e84ae30001b4f37fcf3417744582f0b1c5a37919a1a67" exitCode=0 Oct 31 04:04:12.444130 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.443974 4376 generic.go:356] "Generic (PLEG): container finished" podID="79ae1195-4ff1-46da-b1e2-83042dba6caa" containerID="0ae494e05fd8874d01ee8cac631445033f44579b7d9d3037fcced92bce25054f" exitCode=0 Oct 31 04:04:12.444130 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.443984 4376 generic.go:356] "Generic (PLEG): container finished" podID="79ae1195-4ff1-46da-b1e2-83042dba6caa" containerID="b7c07b812fa6f40ca769dff85f75ecdbb93b0cfc33ac64de3c1127b2d7178162" exitCode=0 Oct 31 04:04:12.444130 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.443991 4376 generic.go:356] "Generic (PLEG): container finished" podID="79ae1195-4ff1-46da-b1e2-83042dba6caa" containerID="39af99a8302ba7f2a6db408099a24e9fae48b6152d199627fcb141f3cefff6c1" exitCode=0 Oct 31 04:04:12.453357 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.453305 4376 generic.go:356] "Generic (PLEG): container finished" podID="b3351600635c8d30ebae275373d4b16d" containerID="08120e1a540caf7fcb57f78e1a171909962020370b9a57ef9e4cb786e45ceee3" exitCode=0 Oct 31 04:04:12.458023 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.457996 4376 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 31 04:04:12.459456 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.459427 4376 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 04:04:12.471997 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.471914 4376 manager.go:324] Recovery completed Oct 31 04:04:12.473045 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.473017 4376 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 04:04:12.473170 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.473049 4376 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 04:04:12.473612 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.473591 4376 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 04:04:12.487512 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.487484 4376 cpu_manager.go:226] "Starting" policy="none" Oct 31 04:04:12.487512 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.487501 4376 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 31 04:04:12.487691 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.487522 4376 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 31 04:04:12.487691 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.487669 4376 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 31 04:04:12.487744 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.487682 4376 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 31 04:04:12.487744 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.487699 4376 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 31 04:04:12.487744 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.487704 4376 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 31 04:04:12.487744 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.487709 4376 policy_none.go:50] "Start" Oct 31 04:04:12.487744 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.487718 4376 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 31 04:04:12.487744 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.487742 4376 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 31 04:04:12.487902 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.487869 4376 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 31 04:04:12.487902 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.487877 4376 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 31 04:04:12.487902 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.487882 4376 policy_none.go:44] "Start" Oct 31 04:04:12.492028 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.491983 4376 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 31 04:04:12.494661 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.494647 4376 manager.go:341] "Starting Device Plugin manager" Oct 31 04:04:12.495184 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:04:12.494685 4376 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 31 04:04:12.495229 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.495190 4376 server.go:87] "Starting device plugin registration server" Oct 31 04:04:12.495483 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.495472 4376 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 31 04:04:12.495518 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.495485 4376 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 31 04:04:12.495613 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.495588 4376 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 31 04:04:12.495749 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.495698 4376 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 31 04:04:12.495749 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.495739 4376 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 31 04:04:12.495749 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.495746 4376 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 31 04:04:12.495899 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.495815 4376 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-31T04:04:12.495726632Z","UUID":"647315fc-5bbb-45d1-a305-2e5fe6b9931b","Handler":null,"Name":"","Endpoint":""} Oct 31 04:04:12.496836 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:04:12.496815 4376 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 31 04:04:12.497850 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.497478 4376 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 31 04:04:12.497850 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.497505 4376 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 31 04:04:12.535929 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.535872 4376 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-134-155.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-155.ec2.internal"] Oct 31 04:04:12.536353 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.536307 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-155.ec2.internal" event={"ID":"b3351600635c8d30ebae275373d4b16d","Type":"ContainerStarted","Data":"50fef1270a6438617bb2c72fb215da4f2c237e4ac94be7b51b9d641177bacede"} Oct 31 04:04:12.536397 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.536364 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-155.ec2.internal" event={"ID":"b3351600635c8d30ebae275373d4b16d","Type":"ContainerDied","Data":"08120e1a540caf7fcb57f78e1a171909962020370b9a57ef9e4cb786e45ceee3"} Oct 31 04:04:12.536397 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.536386 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-155.ec2.internal" event={"ID":"b3351600635c8d30ebae275373d4b16d","Type":"ContainerStarted","Data":"1ddb915eccc0737c404aca07c5758b96a9f0fdd59c1cf88f059dae0e94b737d9"} Oct 31 04:04:12.536451 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.536423 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-155.ec2.internal" event={"ID":"db7fb1e6e2bd3665becb16d5fafc3d7e","Type":"ContainerStarted","Data":"5a30c74cd15688f629eff52dc6b13ac010fb54e444f607c992e06f0fa1f8fc39"} Oct 31 04:04:12.536451 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.536434 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-155.ec2.internal" event={"ID":"db7fb1e6e2bd3665becb16d5fafc3d7e","Type":"ContainerStarted","Data":"8022cc9c6ccbd005a0ecdab2ba41a52a5e4c05e4b51c3c7f012c5cd630f6341b"} Oct 31 04:04:12.536554 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.536504 4376 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-155.ec2.internal" Oct 31 04:04:12.536554 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.536523 4376 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-155.ec2.internal" Oct 31 04:04:12.559775 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.559745 4376 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 04:04:12.559892 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:04:12.559804 4376 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-134-155.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-155.ec2.internal" Oct 31 04:04:12.560339 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.560312 4376 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 04:04:12.560402 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:04:12.560364 4376 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-134-155.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-155.ec2.internal" Oct 31 04:04:12.596488 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.596459 4376 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 04:04:12.597816 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.597799 4376 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-155.ec2.internal" event="NodeHasSufficientMemory" Oct 31 04:04:12.597895 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.597828 4376 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-155.ec2.internal" event="NodeHasNoDiskPressure" Oct 31 04:04:12.597895 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.597839 4376 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-155.ec2.internal" event="NodeHasSufficientPID" Oct 31 04:04:12.597895 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.597868 4376 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-134-155.ec2.internal" Oct 31 04:04:12.633073 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.633041 4376 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-134-155.ec2.internal" Oct 31 04:04:12.633193 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.633131 4376 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-134-155.ec2.internal" Oct 31 04:04:12.659066 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.659031 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/db7fb1e6e2bd3665becb16d5fafc3d7e-config\") pod \"kube-apiserver-proxy-ip-10-0-134-155.ec2.internal\" (UID: \"db7fb1e6e2bd3665becb16d5fafc3d7e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-155.ec2.internal" Oct 31 04:04:12.659066 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.659065 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b3351600635c8d30ebae275373d4b16d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-155.ec2.internal\" (UID: \"b3351600635c8d30ebae275373d4b16d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-155.ec2.internal" Oct 31 04:04:12.659220 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.659083 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b3351600635c8d30ebae275373d4b16d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-155.ec2.internal\" (UID: \"b3351600635c8d30ebae275373d4b16d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-155.ec2.internal" Oct 31 04:04:12.760267 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.760175 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/db7fb1e6e2bd3665becb16d5fafc3d7e-config\") pod \"kube-apiserver-proxy-ip-10-0-134-155.ec2.internal\" (UID: \"db7fb1e6e2bd3665becb16d5fafc3d7e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-155.ec2.internal" Oct 31 04:04:12.760267 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.760204 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/db7fb1e6e2bd3665becb16d5fafc3d7e-config\") pod \"kube-apiserver-proxy-ip-10-0-134-155.ec2.internal\" (UID: \"db7fb1e6e2bd3665becb16d5fafc3d7e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-155.ec2.internal" Oct 31 04:04:12.760267 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.760240 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b3351600635c8d30ebae275373d4b16d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-155.ec2.internal\" (UID: \"b3351600635c8d30ebae275373d4b16d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-155.ec2.internal" Oct 31 04:04:12.760267 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.760268 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b3351600635c8d30ebae275373d4b16d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-155.ec2.internal\" (UID: \"b3351600635c8d30ebae275373d4b16d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-155.ec2.internal" Oct 31 04:04:12.760539 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.760316 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b3351600635c8d30ebae275373d4b16d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-155.ec2.internal\" (UID: \"b3351600635c8d30ebae275373d4b16d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-155.ec2.internal" Oct 31 04:04:12.760539 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:12.760364 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b3351600635c8d30ebae275373d4b16d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-155.ec2.internal\" (UID: \"b3351600635c8d30ebae275373d4b16d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-155.ec2.internal" Oct 31 04:04:13.391768 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.391727 4376 apiserver.go:52] "Watching apiserver" Oct 31 04:04:13.434065 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.434030 4376 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 31 04:04:13.434507 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.434477 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-134-155.ec2.internal","openshift-cluster-node-tuning-operator/tuned-b59p9","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zw4rh","openshift-dns/node-resolver-77z7p","openshift-multus/network-metrics-daemon-gs42n","openshift-network-diagnostics/network-check-target-w9sdm","openshift-ovn-kubernetes/ovnkube-node-zbh9t","kube-system/global-pull-secret-syncer-tlcrp","kube-system/konnectivity-agent-ntbm9","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-155.ec2.internal","openshift-multus/multus-zmrmk","openshift-image-registry/node-ca-xmbxc","openshift-multus/multus-additional-cni-plugins-kmh48","openshift-network-operator/iptables-alerter-6q4xv"] Oct 31 04:04:13.451159 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.451115 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 31 04:04:13.451589 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.451570 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-kfpck\"" Oct 31 04:04:13.451676 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.451577 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-ctbtg\"" Oct 31 04:04:13.451846 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.451821 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-pjn2p\"" Oct 31 04:04:13.451977 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.451866 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 31 04:04:13.452059 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.451977 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 31 04:04:13.452059 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.452000 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 31 04:04:13.452059 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.452005 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 31 04:04:13.452059 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.452021 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 31 04:04:13.452206 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.452057 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 31 04:04:13.452461 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.452438 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 31 04:04:13.452567 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.452448 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 31 04:04:13.452567 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.452520 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 31 04:04:13.452567 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.452535 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 31 04:04:13.452882 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.452866 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 31 04:04:13.461933 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.461903 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 31 04:04:13.462174 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.462147 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 31 04:04:13.462174 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.462152 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 31 04:04:13.463052 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.463030 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 31 04:04:13.463097 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.463081 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 31 04:04:13.463133 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.463106 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 31 04:04:13.463243 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.463230 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 31 04:04:13.463303 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.463292 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 31 04:04:13.463368 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.463300 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 31 04:04:13.463368 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.463317 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 31 04:04:13.463979 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.463965 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 31 04:04:13.464067 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.464053 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 31 04:04:13.464125 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.464062 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 31 04:04:13.464125 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.464058 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 31 04:04:13.464125 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.464122 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 31 04:04:13.464204 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.464137 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 31 04:04:13.470310 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.470291 4376 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-155.ec2.internal" Oct 31 04:04:13.470407 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.470379 4376 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-155.ec2.internal" Oct 31 04:04:13.493608 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.493579 4376 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 04:04:13.493698 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:04:13.493639 4376 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-134-155.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-155.ec2.internal" Oct 31 04:04:13.496448 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.496430 4376 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 04:04:13.496518 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:04:13.496478 4376 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-134-155.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-155.ec2.internal" Oct 31 04:04:13.505678 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.505653 4376 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 31 04:04:13.552612 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.552572 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-45qpm\"" Oct 31 04:04:13.552612 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.552593 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-brbhc\"" Oct 31 04:04:13.552858 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.552659 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-9gl9k\"" Oct 31 04:04:13.552858 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.552692 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-brpbt\"" Oct 31 04:04:13.552858 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.552709 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 31 04:04:13.552858 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.552753 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-ldktz\"" Oct 31 04:04:13.552858 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.552755 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-xzg5s\"" Oct 31 04:04:13.552858 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.552783 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-9876q\"" Oct 31 04:04:13.552858 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.552788 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-lwzcw\"" Oct 31 04:04:13.552858 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.552785 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-vc7dp\"" Oct 31 04:04:13.552858 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.552830 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 31 04:04:13.553090 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.552904 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 31 04:04:13.565038 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.564994 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d4eabffb-abf1-4409-94b5-c0489ab37853-etc-sysctl-conf\") pod \"tuned-b59p9\" (UID: \"d4eabffb-abf1-4409-94b5-c0489ab37853\") " pod="openshift-cluster-node-tuning-operator/tuned-b59p9" Oct 31 04:04:13.565038 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.565027 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d4eabffb-abf1-4409-94b5-c0489ab37853-sys\") pod \"tuned-b59p9\" (UID: \"d4eabffb-abf1-4409-94b5-c0489ab37853\") " pod="openshift-cluster-node-tuning-operator/tuned-b59p9" Oct 31 04:04:13.565136 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.565046 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d4eabffb-abf1-4409-94b5-c0489ab37853-var-lib-kubelet\") pod \"tuned-b59p9\" (UID: \"d4eabffb-abf1-4409-94b5-c0489ab37853\") " pod="openshift-cluster-node-tuning-operator/tuned-b59p9" Oct 31 04:04:13.565136 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.565062 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/153e979c-3c5d-4c4f-b838-fc26db0b5cac-hostroot\") pod \"multus-zmrmk\" (UID: \"153e979c-3c5d-4c4f-b838-fc26db0b5cac\") " pod="openshift-multus/multus-zmrmk" Oct 31 04:04:13.565136 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.565121 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a90157bb-46c0-4afa-82b1-9f6819518744-host\") pod \"node-ca-xmbxc\" (UID: \"a90157bb-46c0-4afa-82b1-9f6819518744\") " pod="openshift-image-registry/node-ca-xmbxc" Oct 31 04:04:13.565248 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.565157 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f7a16817-8ea2-4649-9e42-3d340e1b9703-ovnkube-script-lib\") pod \"ovnkube-node-zbh9t\" (UID: \"f7a16817-8ea2-4649-9e42-3d340e1b9703\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbh9t" Oct 31 04:04:13.565248 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.565177 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c581e590-87e5-4cd7-8d84-eabb4ee21fd1-device-dir\") pod \"aws-ebs-csi-driver-node-zw4rh\" (UID: \"c581e590-87e5-4cd7-8d84-eabb4ee21fd1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zw4rh" Oct 31 04:04:13.565248 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.565236 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/79ae1195-4ff1-46da-b1e2-83042dba6caa-system-cni-dir\") pod \"multus-additional-cni-plugins-kmh48\" (UID: \"79ae1195-4ff1-46da-b1e2-83042dba6caa\") " pod="openshift-multus/multus-additional-cni-plugins-kmh48" Oct 31 04:04:13.565363 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.565269 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d4eabffb-abf1-4409-94b5-c0489ab37853-etc-modprobe-d\") pod \"tuned-b59p9\" (UID: \"d4eabffb-abf1-4409-94b5-c0489ab37853\") " pod="openshift-cluster-node-tuning-operator/tuned-b59p9" Oct 31 04:04:13.565363 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.565289 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d4eabffb-abf1-4409-94b5-c0489ab37853-host\") pod \"tuned-b59p9\" (UID: \"d4eabffb-abf1-4409-94b5-c0489ab37853\") " pod="openshift-cluster-node-tuning-operator/tuned-b59p9" Oct 31 04:04:13.565363 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.565305 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bc9d2733-89fa-4079-9dc1-01983df773cb-kubelet-config\") pod \"global-pull-secret-syncer-tlcrp\" (UID: \"bc9d2733-89fa-4079-9dc1-01983df773cb\") " pod="kube-system/global-pull-secret-syncer-tlcrp" Oct 31 04:04:13.565448 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.565413 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f7a16817-8ea2-4649-9e42-3d340e1b9703-systemd-units\") pod \"ovnkube-node-zbh9t\" (UID: \"f7a16817-8ea2-4649-9e42-3d340e1b9703\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbh9t" Oct 31 04:04:13.565448 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.565439 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c581e590-87e5-4cd7-8d84-eabb4ee21fd1-etc-selinux\") pod \"aws-ebs-csi-driver-node-zw4rh\" (UID: \"c581e590-87e5-4cd7-8d84-eabb4ee21fd1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zw4rh" Oct 31 04:04:13.565501 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.565462 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2zgx\" (UniqueName: \"kubernetes.io/projected/24e22999-2c81-4779-a901-5b9faa15c1cb-kube-api-access-v2zgx\") pod \"iptables-alerter-6q4xv\" (UID: \"24e22999-2c81-4779-a901-5b9faa15c1cb\") " pod="openshift-network-operator/iptables-alerter-6q4xv" Oct 31 04:04:13.565501 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.565473 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f7a16817-8ea2-4649-9e42-3d340e1b9703-ovnkube-script-lib\") pod \"ovnkube-node-zbh9t\" (UID: \"f7a16817-8ea2-4649-9e42-3d340e1b9703\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbh9t" Oct 31 04:04:13.565501 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.565479 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f7a16817-8ea2-4649-9e42-3d340e1b9703-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zbh9t\" (UID: \"f7a16817-8ea2-4649-9e42-3d340e1b9703\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbh9t" Oct 31 04:04:13.565682 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.565666 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1f2543cc-b528-41be-92e0-9d654405a723-hosts-file\") pod \"node-resolver-77z7p\" (UID: \"1f2543cc-b528-41be-92e0-9d654405a723\") " pod="openshift-dns/node-resolver-77z7p" Oct 31 04:04:13.565722 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.565697 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/153e979c-3c5d-4c4f-b838-fc26db0b5cac-host-var-lib-cni-bin\") pod \"multus-zmrmk\" (UID: \"153e979c-3c5d-4c4f-b838-fc26db0b5cac\") " pod="openshift-multus/multus-zmrmk" Oct 31 04:04:13.565755 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.565723 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f7a16817-8ea2-4649-9e42-3d340e1b9703-host-run-ovn-kubernetes\") pod \"ovnkube-node-zbh9t\" (UID: \"f7a16817-8ea2-4649-9e42-3d340e1b9703\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbh9t" Oct 31 04:04:13.565784 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.565757 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c581e590-87e5-4cd7-8d84-eabb4ee21fd1-sys-fs\") pod \"aws-ebs-csi-driver-node-zw4rh\" (UID: \"c581e590-87e5-4cd7-8d84-eabb4ee21fd1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zw4rh" Oct 31 04:04:13.565812 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.565786 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/153e979c-3c5d-4c4f-b838-fc26db0b5cac-multus-conf-dir\") pod \"multus-zmrmk\" (UID: \"153e979c-3c5d-4c4f-b838-fc26db0b5cac\") " pod="openshift-multus/multus-zmrmk" Oct 31 04:04:13.565843 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.565828 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f7a16817-8ea2-4649-9e42-3d340e1b9703-host-cni-bin\") pod \"ovnkube-node-zbh9t\" (UID: \"f7a16817-8ea2-4649-9e42-3d340e1b9703\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbh9t" Oct 31 04:04:13.565875 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.565855 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/79ae1195-4ff1-46da-b1e2-83042dba6caa-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kmh48\" (UID: \"79ae1195-4ff1-46da-b1e2-83042dba6caa\") " pod="openshift-multus/multus-additional-cni-plugins-kmh48" Oct 31 04:04:13.565903 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.565875 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bc9d2733-89fa-4079-9dc1-01983df773cb-dbus\") pod \"global-pull-secret-syncer-tlcrp\" (UID: \"bc9d2733-89fa-4079-9dc1-01983df773cb\") " pod="kube-system/global-pull-secret-syncer-tlcrp" Oct 31 04:04:13.565934 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.565901 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f7a16817-8ea2-4649-9e42-3d340e1b9703-var-lib-openvswitch\") pod \"ovnkube-node-zbh9t\" (UID: \"f7a16817-8ea2-4649-9e42-3d340e1b9703\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbh9t" Oct 31 04:04:13.565934 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.565929 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/153e979c-3c5d-4c4f-b838-fc26db0b5cac-host-var-lib-cni-multus\") pod \"multus-zmrmk\" (UID: \"153e979c-3c5d-4c4f-b838-fc26db0b5cac\") " pod="openshift-multus/multus-zmrmk" Oct 31 04:04:13.566002 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.565954 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7f09238-2bbf-4587-9989-49419d90dabf-metrics-certs\") pod \"network-metrics-daemon-gs42n\" (UID: \"e7f09238-2bbf-4587-9989-49419d90dabf\") " pod="openshift-multus/network-metrics-daemon-gs42n" Oct 31 04:04:13.566002 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.565977 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/153e979c-3c5d-4c4f-b838-fc26db0b5cac-multus-cni-dir\") pod \"multus-zmrmk\" (UID: \"153e979c-3c5d-4c4f-b838-fc26db0b5cac\") " pod="openshift-multus/multus-zmrmk" Oct 31 04:04:13.566002 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.565995 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/153e979c-3c5d-4c4f-b838-fc26db0b5cac-host-run-netns\") pod \"multus-zmrmk\" (UID: \"153e979c-3c5d-4c4f-b838-fc26db0b5cac\") " pod="openshift-multus/multus-zmrmk" Oct 31 04:04:13.566101 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.566013 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f7a16817-8ea2-4649-9e42-3d340e1b9703-run-openvswitch\") pod \"ovnkube-node-zbh9t\" (UID: \"f7a16817-8ea2-4649-9e42-3d340e1b9703\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbh9t" Oct 31 04:04:13.566101 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.566036 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxgq7\" (UniqueName: \"kubernetes.io/projected/1f2543cc-b528-41be-92e0-9d654405a723-kube-api-access-fxgq7\") pod \"node-resolver-77z7p\" (UID: \"1f2543cc-b528-41be-92e0-9d654405a723\") " pod="openshift-dns/node-resolver-77z7p" Oct 31 04:04:13.566101 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.566051 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d4eabffb-abf1-4409-94b5-c0489ab37853-etc-sysconfig\") pod \"tuned-b59p9\" (UID: \"d4eabffb-abf1-4409-94b5-c0489ab37853\") " pod="openshift-cluster-node-tuning-operator/tuned-b59p9" Oct 31 04:04:13.566101 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.566090 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d4eabffb-abf1-4409-94b5-c0489ab37853-tmp\") pod \"tuned-b59p9\" (UID: \"d4eabffb-abf1-4409-94b5-c0489ab37853\") " pod="openshift-cluster-node-tuning-operator/tuned-b59p9" Oct 31 04:04:13.566231 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.566147 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bc9d2733-89fa-4079-9dc1-01983df773cb-original-pull-secret\") pod \"global-pull-secret-syncer-tlcrp\" (UID: \"bc9d2733-89fa-4079-9dc1-01983df773cb\") " pod="kube-system/global-pull-secret-syncer-tlcrp" Oct 31 04:04:13.566231 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.566189 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f2543cc-b528-41be-92e0-9d654405a723-tmp-dir\") pod \"node-resolver-77z7p\" (UID: \"1f2543cc-b528-41be-92e0-9d654405a723\") " pod="openshift-dns/node-resolver-77z7p" Oct 31 04:04:13.566231 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.566198 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d4eabffb-abf1-4409-94b5-c0489ab37853-tmp\") pod \"tuned-b59p9\" (UID: \"d4eabffb-abf1-4409-94b5-c0489ab37853\") " pod="openshift-cluster-node-tuning-operator/tuned-b59p9" Oct 31 04:04:13.566231 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.566211 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7f09238-2bbf-4587-9989-49419d90dabf-metrics-certs\") pod \"network-metrics-daemon-gs42n\" (UID: \"e7f09238-2bbf-4587-9989-49419d90dabf\") " pod="openshift-multus/network-metrics-daemon-gs42n" Oct 31 04:04:13.566231 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.566218 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9r7fx\" (UniqueName: \"kubernetes.io/projected/e7f09238-2bbf-4587-9989-49419d90dabf-kube-api-access-9r7fx\") pod \"network-metrics-daemon-gs42n\" (UID: \"e7f09238-2bbf-4587-9989-49419d90dabf\") " pod="openshift-multus/network-metrics-daemon-gs42n" Oct 31 04:04:13.566432 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.566215 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/79ae1195-4ff1-46da-b1e2-83042dba6caa-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kmh48\" (UID: \"79ae1195-4ff1-46da-b1e2-83042dba6caa\") " pod="openshift-multus/multus-additional-cni-plugins-kmh48" Oct 31 04:04:13.566432 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.566281 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f2543cc-b528-41be-92e0-9d654405a723-tmp-dir\") pod \"node-resolver-77z7p\" (UID: \"1f2543cc-b528-41be-92e0-9d654405a723\") " pod="openshift-dns/node-resolver-77z7p" Oct 31 04:04:13.566432 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.566322 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/79ae1195-4ff1-46da-b1e2-83042dba6caa-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kmh48\" (UID: \"79ae1195-4ff1-46da-b1e2-83042dba6caa\") " pod="openshift-multus/multus-additional-cni-plugins-kmh48" Oct 31 04:04:13.566432 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.566399 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f12d4ae1-e673-4a9e-bd43-92d6e9d5a023-agent-certs\") pod \"konnectivity-agent-ntbm9\" (UID: \"f12d4ae1-e673-4a9e-bd43-92d6e9d5a023\") " pod="kube-system/konnectivity-agent-ntbm9" Oct 31 04:04:13.566546 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.566434 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d4eabffb-abf1-4409-94b5-c0489ab37853-etc-sysctl-d\") pod \"tuned-b59p9\" (UID: \"d4eabffb-abf1-4409-94b5-c0489ab37853\") " pod="openshift-cluster-node-tuning-operator/tuned-b59p9" Oct 31 04:04:13.566546 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.566460 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bc9d2733-89fa-4079-9dc1-01983df773cb-original-pull-secret\") pod \"global-pull-secret-syncer-tlcrp\" (UID: \"bc9d2733-89fa-4079-9dc1-01983df773cb\") " pod="kube-system/global-pull-secret-syncer-tlcrp" Oct 31 04:04:13.566546 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.566521 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/153e979c-3c5d-4c4f-b838-fc26db0b5cac-etc-kubernetes\") pod \"multus-zmrmk\" (UID: \"153e979c-3c5d-4c4f-b838-fc26db0b5cac\") " pod="openshift-multus/multus-zmrmk" Oct 31 04:04:13.566637 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.566565 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a90157bb-46c0-4afa-82b1-9f6819518744-serviceca\") pod \"node-ca-xmbxc\" (UID: \"a90157bb-46c0-4afa-82b1-9f6819518744\") " pod="openshift-image-registry/node-ca-xmbxc" Oct 31 04:04:13.566637 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.566595 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f7a16817-8ea2-4649-9e42-3d340e1b9703-run-ovn\") pod \"ovnkube-node-zbh9t\" (UID: \"f7a16817-8ea2-4649-9e42-3d340e1b9703\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbh9t" Oct 31 04:04:13.566637 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.566616 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d4eabffb-abf1-4409-94b5-c0489ab37853-etc-kubernetes\") pod \"tuned-b59p9\" (UID: \"d4eabffb-abf1-4409-94b5-c0489ab37853\") " pod="openshift-cluster-node-tuning-operator/tuned-b59p9" Oct 31 04:04:13.566719 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.566709 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d4eabffb-abf1-4409-94b5-c0489ab37853-etc-tuned\") pod \"tuned-b59p9\" (UID: \"d4eabffb-abf1-4409-94b5-c0489ab37853\") " pod="openshift-cluster-node-tuning-operator/tuned-b59p9" Oct 31 04:04:13.566748 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.566729 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f12d4ae1-e673-4a9e-bd43-92d6e9d5a023-agent-certs\") pod \"konnectivity-agent-ntbm9\" (UID: \"f12d4ae1-e673-4a9e-bd43-92d6e9d5a023\") " pod="kube-system/konnectivity-agent-ntbm9" Oct 31 04:04:13.566786 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.566762 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/153e979c-3c5d-4c4f-b838-fc26db0b5cac-host-run-multus-certs\") pod \"multus-zmrmk\" (UID: \"153e979c-3c5d-4c4f-b838-fc26db0b5cac\") " pod="openshift-multus/multus-zmrmk" Oct 31 04:04:13.566817 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.566784 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d4eabffb-abf1-4409-94b5-c0489ab37853-etc-tuned\") pod \"tuned-b59p9\" (UID: \"d4eabffb-abf1-4409-94b5-c0489ab37853\") " pod="openshift-cluster-node-tuning-operator/tuned-b59p9" Oct 31 04:04:13.566817 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.566787 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f7a16817-8ea2-4649-9e42-3d340e1b9703-host-slash\") pod \"ovnkube-node-zbh9t\" (UID: \"f7a16817-8ea2-4649-9e42-3d340e1b9703\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbh9t" Oct 31 04:04:13.566869 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.566823 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f7a16817-8ea2-4649-9e42-3d340e1b9703-env-overrides\") pod \"ovnkube-node-zbh9t\" (UID: \"f7a16817-8ea2-4649-9e42-3d340e1b9703\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbh9t" Oct 31 04:04:13.566869 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.566850 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/24e22999-2c81-4779-a901-5b9faa15c1cb-host-slash\") pod \"iptables-alerter-6q4xv\" (UID: \"24e22999-2c81-4779-a901-5b9faa15c1cb\") " pod="openshift-network-operator/iptables-alerter-6q4xv" Oct 31 04:04:13.566930 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.566882 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d4eabffb-abf1-4409-94b5-c0489ab37853-etc-systemd\") pod \"tuned-b59p9\" (UID: \"d4eabffb-abf1-4409-94b5-c0489ab37853\") " pod="openshift-cluster-node-tuning-operator/tuned-b59p9" Oct 31 04:04:13.566930 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.566922 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/153e979c-3c5d-4c4f-b838-fc26db0b5cac-multus-daemon-config\") pod \"multus-zmrmk\" (UID: \"153e979c-3c5d-4c4f-b838-fc26db0b5cac\") " pod="openshift-multus/multus-zmrmk" Oct 31 04:04:13.566996 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.566982 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f7a16817-8ea2-4649-9e42-3d340e1b9703-env-overrides\") pod \"ovnkube-node-zbh9t\" (UID: \"f7a16817-8ea2-4649-9e42-3d340e1b9703\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbh9t" Oct 31 04:04:13.567091 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.567079 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-786gp\" (UniqueName: \"kubernetes.io/projected/a90157bb-46c0-4afa-82b1-9f6819518744-kube-api-access-786gp\") pod \"node-ca-xmbxc\" (UID: \"a90157bb-46c0-4afa-82b1-9f6819518744\") " pod="openshift-image-registry/node-ca-xmbxc" Oct 31 04:04:13.567123 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.567109 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f7a16817-8ea2-4649-9e42-3d340e1b9703-host-run-netns\") pod \"ovnkube-node-zbh9t\" (UID: \"f7a16817-8ea2-4649-9e42-3d340e1b9703\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbh9t" Oct 31 04:04:13.567172 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.567125 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwltl\" (UniqueName: \"kubernetes.io/projected/c581e590-87e5-4cd7-8d84-eabb4ee21fd1-kube-api-access-wwltl\") pod \"aws-ebs-csi-driver-node-zw4rh\" (UID: \"c581e590-87e5-4cd7-8d84-eabb4ee21fd1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zw4rh" Oct 31 04:04:13.567220 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.567168 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/153e979c-3c5d-4c4f-b838-fc26db0b5cac-multus-daemon-config\") pod \"multus-zmrmk\" (UID: \"153e979c-3c5d-4c4f-b838-fc26db0b5cac\") " pod="openshift-multus/multus-zmrmk" Oct 31 04:04:13.567220 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.567171 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/79ae1195-4ff1-46da-b1e2-83042dba6caa-cni-binary-copy\") pod \"multus-additional-cni-plugins-kmh48\" (UID: \"79ae1195-4ff1-46da-b1e2-83042dba6caa\") " pod="openshift-multus/multus-additional-cni-plugins-kmh48" Oct 31 04:04:13.567320 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.567219 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a90157bb-46c0-4afa-82b1-9f6819518744-serviceca\") pod \"node-ca-xmbxc\" (UID: \"a90157bb-46c0-4afa-82b1-9f6819518744\") " pod="openshift-image-registry/node-ca-xmbxc" Oct 31 04:04:13.567320 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.567224 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jq5vs\" (UniqueName: \"kubernetes.io/projected/f70f5eab-445f-430e-bfb1-92e4d620b90d-kube-api-access-jq5vs\") pod \"network-check-target-w9sdm\" (UID: \"f70f5eab-445f-430e-bfb1-92e4d620b90d\") " pod="openshift-network-diagnostics/network-check-target-w9sdm" Oct 31 04:04:13.567320 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.567251 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d4eabffb-abf1-4409-94b5-c0489ab37853-run\") pod \"tuned-b59p9\" (UID: \"d4eabffb-abf1-4409-94b5-c0489ab37853\") " pod="openshift-cluster-node-tuning-operator/tuned-b59p9" Oct 31 04:04:13.567320 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.567286 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/153e979c-3c5d-4c4f-b838-fc26db0b5cac-host-run-k8s-cni-cncf-io\") pod \"multus-zmrmk\" (UID: \"153e979c-3c5d-4c4f-b838-fc26db0b5cac\") " pod="openshift-multus/multus-zmrmk" Oct 31 04:04:13.567484 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.567319 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vp84q\" (UniqueName: \"kubernetes.io/projected/153e979c-3c5d-4c4f-b838-fc26db0b5cac-kube-api-access-vp84q\") pod \"multus-zmrmk\" (UID: \"153e979c-3c5d-4c4f-b838-fc26db0b5cac\") " pod="openshift-multus/multus-zmrmk" Oct 31 04:04:13.567484 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.567377 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f7a16817-8ea2-4649-9e42-3d340e1b9703-etc-openvswitch\") pod \"ovnkube-node-zbh9t\" (UID: \"f7a16817-8ea2-4649-9e42-3d340e1b9703\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbh9t" Oct 31 04:04:13.567484 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.567377 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/79ae1195-4ff1-46da-b1e2-83042dba6caa-cni-binary-copy\") pod \"multus-additional-cni-plugins-kmh48\" (UID: \"79ae1195-4ff1-46da-b1e2-83042dba6caa\") " pod="openshift-multus/multus-additional-cni-plugins-kmh48" Oct 31 04:04:13.567484 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.567416 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f7a16817-8ea2-4649-9e42-3d340e1b9703-log-socket\") pod \"ovnkube-node-zbh9t\" (UID: \"f7a16817-8ea2-4649-9e42-3d340e1b9703\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbh9t" Oct 31 04:04:13.567484 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.567439 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c581e590-87e5-4cd7-8d84-eabb4ee21fd1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zw4rh\" (UID: \"c581e590-87e5-4cd7-8d84-eabb4ee21fd1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zw4rh" Oct 31 04:04:13.567484 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.567471 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n54k7\" (UniqueName: \"kubernetes.io/projected/d4eabffb-abf1-4409-94b5-c0489ab37853-kube-api-access-n54k7\") pod \"tuned-b59p9\" (UID: \"d4eabffb-abf1-4409-94b5-c0489ab37853\") " pod="openshift-cluster-node-tuning-operator/tuned-b59p9" Oct 31 04:04:13.567651 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.567497 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f7a16817-8ea2-4649-9e42-3d340e1b9703-host-kubelet\") pod \"ovnkube-node-zbh9t\" (UID: \"f7a16817-8ea2-4649-9e42-3d340e1b9703\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbh9t" Oct 31 04:04:13.567651 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.567511 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d4eabffb-abf1-4409-94b5-c0489ab37853-lib-modules\") pod \"tuned-b59p9\" (UID: \"d4eabffb-abf1-4409-94b5-c0489ab37853\") " pod="openshift-cluster-node-tuning-operator/tuned-b59p9" Oct 31 04:04:13.567651 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.567526 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f7a16817-8ea2-4649-9e42-3d340e1b9703-run-systemd\") pod \"ovnkube-node-zbh9t\" (UID: \"f7a16817-8ea2-4649-9e42-3d340e1b9703\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbh9t" Oct 31 04:04:13.567651 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.567541 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/153e979c-3c5d-4c4f-b838-fc26db0b5cac-host-var-lib-kubelet\") pod \"multus-zmrmk\" (UID: \"153e979c-3c5d-4c4f-b838-fc26db0b5cac\") " pod="openshift-multus/multus-zmrmk" Oct 31 04:04:13.567651 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.567562 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/153e979c-3c5d-4c4f-b838-fc26db0b5cac-cnibin\") pod \"multus-zmrmk\" (UID: \"153e979c-3c5d-4c4f-b838-fc26db0b5cac\") " pod="openshift-multus/multus-zmrmk" Oct 31 04:04:13.567651 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.567589 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/153e979c-3c5d-4c4f-b838-fc26db0b5cac-cni-binary-copy\") pod \"multus-zmrmk\" (UID: \"153e979c-3c5d-4c4f-b838-fc26db0b5cac\") " pod="openshift-multus/multus-zmrmk" Oct 31 04:04:13.567651 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.567645 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/153e979c-3c5d-4c4f-b838-fc26db0b5cac-multus-socket-dir-parent\") pod \"multus-zmrmk\" (UID: \"153e979c-3c5d-4c4f-b838-fc26db0b5cac\") " pod="openshift-multus/multus-zmrmk" Oct 31 04:04:13.567832 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.567734 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkkp9\" (UniqueName: \"kubernetes.io/projected/f7a16817-8ea2-4649-9e42-3d340e1b9703-kube-api-access-dkkp9\") pod \"ovnkube-node-zbh9t\" (UID: \"f7a16817-8ea2-4649-9e42-3d340e1b9703\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbh9t" Oct 31 04:04:13.567832 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.567768 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c581e590-87e5-4cd7-8d84-eabb4ee21fd1-socket-dir\") pod \"aws-ebs-csi-driver-node-zw4rh\" (UID: \"c581e590-87e5-4cd7-8d84-eabb4ee21fd1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zw4rh" Oct 31 04:04:13.567832 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.567789 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/153e979c-3c5d-4c4f-b838-fc26db0b5cac-cni-binary-copy\") pod \"multus-zmrmk\" (UID: \"153e979c-3c5d-4c4f-b838-fc26db0b5cac\") " pod="openshift-multus/multus-zmrmk" Oct 31 04:04:13.567913 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.567862 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/153e979c-3c5d-4c4f-b838-fc26db0b5cac-system-cni-dir\") pod \"multus-zmrmk\" (UID: \"153e979c-3c5d-4c4f-b838-fc26db0b5cac\") " pod="openshift-multus/multus-zmrmk" Oct 31 04:04:13.567913 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.567878 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f7a16817-8ea2-4649-9e42-3d340e1b9703-node-log\") pod \"ovnkube-node-zbh9t\" (UID: \"f7a16817-8ea2-4649-9e42-3d340e1b9703\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbh9t" Oct 31 04:04:13.567969 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.567915 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f7a16817-8ea2-4649-9e42-3d340e1b9703-host-cni-netd\") pod \"ovnkube-node-zbh9t\" (UID: \"f7a16817-8ea2-4649-9e42-3d340e1b9703\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbh9t" Oct 31 04:04:13.567969 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.567944 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f7a16817-8ea2-4649-9e42-3d340e1b9703-ovnkube-config\") pod \"ovnkube-node-zbh9t\" (UID: \"f7a16817-8ea2-4649-9e42-3d340e1b9703\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbh9t" Oct 31 04:04:13.568024 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.567968 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c581e590-87e5-4cd7-8d84-eabb4ee21fd1-registration-dir\") pod \"aws-ebs-csi-driver-node-zw4rh\" (UID: \"c581e590-87e5-4cd7-8d84-eabb4ee21fd1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zw4rh" Oct 31 04:04:13.568024 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.567995 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/79ae1195-4ff1-46da-b1e2-83042dba6caa-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kmh48\" (UID: \"79ae1195-4ff1-46da-b1e2-83042dba6caa\") " pod="openshift-multus/multus-additional-cni-plugins-kmh48" Oct 31 04:04:13.568099 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.568086 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f12d4ae1-e673-4a9e-bd43-92d6e9d5a023-konnectivity-ca\") pod \"konnectivity-agent-ntbm9\" (UID: \"f12d4ae1-e673-4a9e-bd43-92d6e9d5a023\") " pod="kube-system/konnectivity-agent-ntbm9" Oct 31 04:04:13.568153 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.568137 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f7a16817-8ea2-4649-9e42-3d340e1b9703-ovnkube-config\") pod \"ovnkube-node-zbh9t\" (UID: \"f7a16817-8ea2-4649-9e42-3d340e1b9703\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbh9t" Oct 31 04:04:13.568183 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.568154 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/24e22999-2c81-4779-a901-5b9faa15c1cb-iptables-alerter-script\") pod \"iptables-alerter-6q4xv\" (UID: \"24e22999-2c81-4779-a901-5b9faa15c1cb\") " pod="openshift-network-operator/iptables-alerter-6q4xv" Oct 31 04:04:13.568183 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.568175 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f7a16817-8ea2-4649-9e42-3d340e1b9703-ovn-node-metrics-cert\") pod \"ovnkube-node-zbh9t\" (UID: \"f7a16817-8ea2-4649-9e42-3d340e1b9703\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbh9t" Oct 31 04:04:13.568234 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.568213 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/79ae1195-4ff1-46da-b1e2-83042dba6caa-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kmh48\" (UID: \"79ae1195-4ff1-46da-b1e2-83042dba6caa\") " pod="openshift-multus/multus-additional-cni-plugins-kmh48" Oct 31 04:04:13.568323 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.568309 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/79ae1195-4ff1-46da-b1e2-83042dba6caa-cnibin\") pod \"multus-additional-cni-plugins-kmh48\" (UID: \"79ae1195-4ff1-46da-b1e2-83042dba6caa\") " pod="openshift-multus/multus-additional-cni-plugins-kmh48" Oct 31 04:04:13.568386 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.568351 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f12d4ae1-e673-4a9e-bd43-92d6e9d5a023-konnectivity-ca\") pod \"konnectivity-agent-ntbm9\" (UID: \"f12d4ae1-e673-4a9e-bd43-92d6e9d5a023\") " pod="kube-system/konnectivity-agent-ntbm9" Oct 31 04:04:13.568424 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.568386 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/24e22999-2c81-4779-a901-5b9faa15c1cb-iptables-alerter-script\") pod \"iptables-alerter-6q4xv\" (UID: \"24e22999-2c81-4779-a901-5b9faa15c1cb\") " pod="openshift-network-operator/iptables-alerter-6q4xv" Oct 31 04:04:13.568424 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.568393 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgdpm\" (UniqueName: \"kubernetes.io/projected/79ae1195-4ff1-46da-b1e2-83042dba6caa-kube-api-access-wgdpm\") pod \"multus-additional-cni-plugins-kmh48\" (UID: \"79ae1195-4ff1-46da-b1e2-83042dba6caa\") " pod="openshift-multus/multus-additional-cni-plugins-kmh48" Oct 31 04:04:13.568576 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.568562 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f7a16817-8ea2-4649-9e42-3d340e1b9703-ovn-node-metrics-cert\") pod \"ovnkube-node-zbh9t\" (UID: \"f7a16817-8ea2-4649-9e42-3d340e1b9703\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbh9t" Oct 31 04:04:13.591589 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.591554 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2zgx\" (UniqueName: \"kubernetes.io/projected/24e22999-2c81-4779-a901-5b9faa15c1cb-kube-api-access-v2zgx\") pod \"iptables-alerter-6q4xv\" (UID: \"24e22999-2c81-4779-a901-5b9faa15c1cb\") " pod="openshift-network-operator/iptables-alerter-6q4xv" Oct 31 04:04:13.593802 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.593771 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vp84q\" (UniqueName: \"kubernetes.io/projected/153e979c-3c5d-4c4f-b838-fc26db0b5cac-kube-api-access-vp84q\") pod \"multus-zmrmk\" (UID: \"153e979c-3c5d-4c4f-b838-fc26db0b5cac\") " pod="openshift-multus/multus-zmrmk" Oct 31 04:04:13.593943 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.593821 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgdpm\" (UniqueName: \"kubernetes.io/projected/79ae1195-4ff1-46da-b1e2-83042dba6caa-kube-api-access-wgdpm\") pod \"multus-additional-cni-plugins-kmh48\" (UID: \"79ae1195-4ff1-46da-b1e2-83042dba6caa\") " pod="openshift-multus/multus-additional-cni-plugins-kmh48" Oct 31 04:04:13.593943 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.593821 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9r7fx\" (UniqueName: \"kubernetes.io/projected/e7f09238-2bbf-4587-9989-49419d90dabf-kube-api-access-9r7fx\") pod \"network-metrics-daemon-gs42n\" (UID: \"e7f09238-2bbf-4587-9989-49419d90dabf\") " pod="openshift-multus/network-metrics-daemon-gs42n" Oct 31 04:04:13.593943 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.593895 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jq5vs\" (UniqueName: \"kubernetes.io/projected/f70f5eab-445f-430e-bfb1-92e4d620b90d-kube-api-access-jq5vs\") pod \"network-check-target-w9sdm\" (UID: \"f70f5eab-445f-430e-bfb1-92e4d620b90d\") " pod="openshift-network-diagnostics/network-check-target-w9sdm" Oct 31 04:04:13.594078 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.593937 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxgq7\" (UniqueName: \"kubernetes.io/projected/1f2543cc-b528-41be-92e0-9d654405a723-kube-api-access-fxgq7\") pod \"node-resolver-77z7p\" (UID: \"1f2543cc-b528-41be-92e0-9d654405a723\") " pod="openshift-dns/node-resolver-77z7p" Oct 31 04:04:13.594078 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.593974 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-786gp\" (UniqueName: \"kubernetes.io/projected/a90157bb-46c0-4afa-82b1-9f6819518744-kube-api-access-786gp\") pod \"node-ca-xmbxc\" (UID: \"a90157bb-46c0-4afa-82b1-9f6819518744\") " pod="openshift-image-registry/node-ca-xmbxc" Oct 31 04:04:13.594078 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.593989 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkkp9\" (UniqueName: \"kubernetes.io/projected/f7a16817-8ea2-4649-9e42-3d340e1b9703-kube-api-access-dkkp9\") pod \"ovnkube-node-zbh9t\" (UID: \"f7a16817-8ea2-4649-9e42-3d340e1b9703\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbh9t" Oct 31 04:04:13.594293 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.594273 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n54k7\" (UniqueName: \"kubernetes.io/projected/d4eabffb-abf1-4409-94b5-c0489ab37853-kube-api-access-n54k7\") pod \"tuned-b59p9\" (UID: \"d4eabffb-abf1-4409-94b5-c0489ab37853\") " pod="openshift-cluster-node-tuning-operator/tuned-b59p9" Oct 31 04:04:13.594293 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.594287 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwltl\" (UniqueName: \"kubernetes.io/projected/c581e590-87e5-4cd7-8d84-eabb4ee21fd1-kube-api-access-wwltl\") pod \"aws-ebs-csi-driver-node-zw4rh\" (UID: \"c581e590-87e5-4cd7-8d84-eabb4ee21fd1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zw4rh" Oct 31 04:04:13.669216 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.669122 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/153e979c-3c5d-4c4f-b838-fc26db0b5cac-host-run-multus-certs\") pod \"multus-zmrmk\" (UID: \"153e979c-3c5d-4c4f-b838-fc26db0b5cac\") " pod="openshift-multus/multus-zmrmk" Oct 31 04:04:13.669216 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.669161 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f7a16817-8ea2-4649-9e42-3d340e1b9703-host-slash\") pod \"ovnkube-node-zbh9t\" (UID: \"f7a16817-8ea2-4649-9e42-3d340e1b9703\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbh9t" Oct 31 04:04:13.669216 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.669178 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/24e22999-2c81-4779-a901-5b9faa15c1cb-host-slash\") pod \"iptables-alerter-6q4xv\" (UID: \"24e22999-2c81-4779-a901-5b9faa15c1cb\") " pod="openshift-network-operator/iptables-alerter-6q4xv" Oct 31 04:04:13.669479 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.669223 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f7a16817-8ea2-4649-9e42-3d340e1b9703-host-slash\") pod \"ovnkube-node-zbh9t\" (UID: \"f7a16817-8ea2-4649-9e42-3d340e1b9703\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbh9t" Oct 31 04:04:13.669479 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.669227 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/153e979c-3c5d-4c4f-b838-fc26db0b5cac-host-run-multus-certs\") pod \"multus-zmrmk\" (UID: \"153e979c-3c5d-4c4f-b838-fc26db0b5cac\") " pod="openshift-multus/multus-zmrmk" Oct 31 04:04:13.669479 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.669252 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d4eabffb-abf1-4409-94b5-c0489ab37853-etc-systemd\") pod \"tuned-b59p9\" (UID: \"d4eabffb-abf1-4409-94b5-c0489ab37853\") " pod="openshift-cluster-node-tuning-operator/tuned-b59p9" Oct 31 04:04:13.669479 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.669269 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f7a16817-8ea2-4649-9e42-3d340e1b9703-host-run-netns\") pod \"ovnkube-node-zbh9t\" (UID: \"f7a16817-8ea2-4649-9e42-3d340e1b9703\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbh9t" Oct 31 04:04:13.669479 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.669271 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/24e22999-2c81-4779-a901-5b9faa15c1cb-host-slash\") pod \"iptables-alerter-6q4xv\" (UID: \"24e22999-2c81-4779-a901-5b9faa15c1cb\") " pod="openshift-network-operator/iptables-alerter-6q4xv" Oct 31 04:04:13.669479 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.669288 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d4eabffb-abf1-4409-94b5-c0489ab37853-run\") pod \"tuned-b59p9\" (UID: \"d4eabffb-abf1-4409-94b5-c0489ab37853\") " pod="openshift-cluster-node-tuning-operator/tuned-b59p9" Oct 31 04:04:13.669479 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.669312 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/153e979c-3c5d-4c4f-b838-fc26db0b5cac-host-run-k8s-cni-cncf-io\") pod \"multus-zmrmk\" (UID: \"153e979c-3c5d-4c4f-b838-fc26db0b5cac\") " pod="openshift-multus/multus-zmrmk" Oct 31 04:04:13.669479 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.669349 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f7a16817-8ea2-4649-9e42-3d340e1b9703-etc-openvswitch\") pod \"ovnkube-node-zbh9t\" (UID: \"f7a16817-8ea2-4649-9e42-3d340e1b9703\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbh9t" Oct 31 04:04:13.669479 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.669312 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d4eabffb-abf1-4409-94b5-c0489ab37853-etc-systemd\") pod \"tuned-b59p9\" (UID: \"d4eabffb-abf1-4409-94b5-c0489ab37853\") " pod="openshift-cluster-node-tuning-operator/tuned-b59p9" Oct 31 04:04:13.669479 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.669394 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d4eabffb-abf1-4409-94b5-c0489ab37853-run\") pod \"tuned-b59p9\" (UID: \"d4eabffb-abf1-4409-94b5-c0489ab37853\") " pod="openshift-cluster-node-tuning-operator/tuned-b59p9" Oct 31 04:04:13.669479 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.669397 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/153e979c-3c5d-4c4f-b838-fc26db0b5cac-host-run-k8s-cni-cncf-io\") pod \"multus-zmrmk\" (UID: \"153e979c-3c5d-4c4f-b838-fc26db0b5cac\") " pod="openshift-multus/multus-zmrmk" Oct 31 04:04:13.669479 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.669391 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f7a16817-8ea2-4649-9e42-3d340e1b9703-host-run-netns\") pod \"ovnkube-node-zbh9t\" (UID: \"f7a16817-8ea2-4649-9e42-3d340e1b9703\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbh9t" Oct 31 04:04:13.669479 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.669436 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f7a16817-8ea2-4649-9e42-3d340e1b9703-etc-openvswitch\") pod \"ovnkube-node-zbh9t\" (UID: \"f7a16817-8ea2-4649-9e42-3d340e1b9703\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbh9t" Oct 31 04:04:13.669479 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.669461 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f7a16817-8ea2-4649-9e42-3d340e1b9703-log-socket\") pod \"ovnkube-node-zbh9t\" (UID: \"f7a16817-8ea2-4649-9e42-3d340e1b9703\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbh9t" Oct 31 04:04:13.669479 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.669479 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c581e590-87e5-4cd7-8d84-eabb4ee21fd1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zw4rh\" (UID: \"c581e590-87e5-4cd7-8d84-eabb4ee21fd1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zw4rh" Oct 31 04:04:13.669875 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.669497 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f7a16817-8ea2-4649-9e42-3d340e1b9703-host-kubelet\") pod \"ovnkube-node-zbh9t\" (UID: \"f7a16817-8ea2-4649-9e42-3d340e1b9703\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbh9t" Oct 31 04:04:13.669875 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.669514 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f7a16817-8ea2-4649-9e42-3d340e1b9703-log-socket\") pod \"ovnkube-node-zbh9t\" (UID: \"f7a16817-8ea2-4649-9e42-3d340e1b9703\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbh9t" Oct 31 04:04:13.669875 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.669519 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d4eabffb-abf1-4409-94b5-c0489ab37853-lib-modules\") pod \"tuned-b59p9\" (UID: \"d4eabffb-abf1-4409-94b5-c0489ab37853\") " pod="openshift-cluster-node-tuning-operator/tuned-b59p9" Oct 31 04:04:13.669875 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.669554 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f7a16817-8ea2-4649-9e42-3d340e1b9703-host-kubelet\") pod \"ovnkube-node-zbh9t\" (UID: \"f7a16817-8ea2-4649-9e42-3d340e1b9703\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbh9t" Oct 31 04:04:13.669875 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.669555 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c581e590-87e5-4cd7-8d84-eabb4ee21fd1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zw4rh\" (UID: \"c581e590-87e5-4cd7-8d84-eabb4ee21fd1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zw4rh" Oct 31 04:04:13.669875 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.669572 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f7a16817-8ea2-4649-9e42-3d340e1b9703-run-systemd\") pod \"ovnkube-node-zbh9t\" (UID: \"f7a16817-8ea2-4649-9e42-3d340e1b9703\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbh9t" Oct 31 04:04:13.669875 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.669593 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/153e979c-3c5d-4c4f-b838-fc26db0b5cac-host-var-lib-kubelet\") pod \"multus-zmrmk\" (UID: \"153e979c-3c5d-4c4f-b838-fc26db0b5cac\") " pod="openshift-multus/multus-zmrmk" Oct 31 04:04:13.669875 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.669610 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f7a16817-8ea2-4649-9e42-3d340e1b9703-run-systemd\") pod \"ovnkube-node-zbh9t\" (UID: \"f7a16817-8ea2-4649-9e42-3d340e1b9703\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbh9t" Oct 31 04:04:13.669875 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.669616 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/153e979c-3c5d-4c4f-b838-fc26db0b5cac-cnibin\") pod \"multus-zmrmk\" (UID: \"153e979c-3c5d-4c4f-b838-fc26db0b5cac\") " pod="openshift-multus/multus-zmrmk" Oct 31 04:04:13.669875 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.669637 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d4eabffb-abf1-4409-94b5-c0489ab37853-lib-modules\") pod \"tuned-b59p9\" (UID: \"d4eabffb-abf1-4409-94b5-c0489ab37853\") " pod="openshift-cluster-node-tuning-operator/tuned-b59p9" Oct 31 04:04:13.669875 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.669641 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/153e979c-3c5d-4c4f-b838-fc26db0b5cac-host-var-lib-kubelet\") pod \"multus-zmrmk\" (UID: \"153e979c-3c5d-4c4f-b838-fc26db0b5cac\") " pod="openshift-multus/multus-zmrmk" Oct 31 04:04:13.669875 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.669641 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/153e979c-3c5d-4c4f-b838-fc26db0b5cac-multus-socket-dir-parent\") pod \"multus-zmrmk\" (UID: \"153e979c-3c5d-4c4f-b838-fc26db0b5cac\") " pod="openshift-multus/multus-zmrmk" Oct 31 04:04:13.669875 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.669670 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c581e590-87e5-4cd7-8d84-eabb4ee21fd1-socket-dir\") pod \"aws-ebs-csi-driver-node-zw4rh\" (UID: \"c581e590-87e5-4cd7-8d84-eabb4ee21fd1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zw4rh" Oct 31 04:04:13.669875 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.669680 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/153e979c-3c5d-4c4f-b838-fc26db0b5cac-multus-socket-dir-parent\") pod \"multus-zmrmk\" (UID: \"153e979c-3c5d-4c4f-b838-fc26db0b5cac\") " pod="openshift-multus/multus-zmrmk" Oct 31 04:04:13.669875 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.669680 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/153e979c-3c5d-4c4f-b838-fc26db0b5cac-cnibin\") pod \"multus-zmrmk\" (UID: \"153e979c-3c5d-4c4f-b838-fc26db0b5cac\") " pod="openshift-multus/multus-zmrmk" Oct 31 04:04:13.669875 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.669691 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/153e979c-3c5d-4c4f-b838-fc26db0b5cac-system-cni-dir\") pod \"multus-zmrmk\" (UID: \"153e979c-3c5d-4c4f-b838-fc26db0b5cac\") " pod="openshift-multus/multus-zmrmk" Oct 31 04:04:13.669875 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.669712 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f7a16817-8ea2-4649-9e42-3d340e1b9703-node-log\") pod \"ovnkube-node-zbh9t\" (UID: \"f7a16817-8ea2-4649-9e42-3d340e1b9703\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbh9t" Oct 31 04:04:13.669875 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.669741 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f7a16817-8ea2-4649-9e42-3d340e1b9703-host-cni-netd\") pod \"ovnkube-node-zbh9t\" (UID: \"f7a16817-8ea2-4649-9e42-3d340e1b9703\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbh9t" Oct 31 04:04:13.669875 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.669758 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c581e590-87e5-4cd7-8d84-eabb4ee21fd1-registration-dir\") pod \"aws-ebs-csi-driver-node-zw4rh\" (UID: \"c581e590-87e5-4cd7-8d84-eabb4ee21fd1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zw4rh" Oct 31 04:04:13.669875 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.669771 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/153e979c-3c5d-4c4f-b838-fc26db0b5cac-system-cni-dir\") pod \"multus-zmrmk\" (UID: \"153e979c-3c5d-4c4f-b838-fc26db0b5cac\") " pod="openshift-multus/multus-zmrmk" Oct 31 04:04:13.669875 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.669775 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/79ae1195-4ff1-46da-b1e2-83042dba6caa-cnibin\") pod \"multus-additional-cni-plugins-kmh48\" (UID: \"79ae1195-4ff1-46da-b1e2-83042dba6caa\") " pod="openshift-multus/multus-additional-cni-plugins-kmh48" Oct 31 04:04:13.669875 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.669781 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f7a16817-8ea2-4649-9e42-3d340e1b9703-node-log\") pod \"ovnkube-node-zbh9t\" (UID: \"f7a16817-8ea2-4649-9e42-3d340e1b9703\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbh9t" Oct 31 04:04:13.669875 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.669792 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f7a16817-8ea2-4649-9e42-3d340e1b9703-host-cni-netd\") pod \"ovnkube-node-zbh9t\" (UID: \"f7a16817-8ea2-4649-9e42-3d340e1b9703\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbh9t" Oct 31 04:04:13.669875 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.669807 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d4eabffb-abf1-4409-94b5-c0489ab37853-etc-sysctl-conf\") pod \"tuned-b59p9\" (UID: \"d4eabffb-abf1-4409-94b5-c0489ab37853\") " pod="openshift-cluster-node-tuning-operator/tuned-b59p9" Oct 31 04:04:13.669875 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.669820 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/79ae1195-4ff1-46da-b1e2-83042dba6caa-cnibin\") pod \"multus-additional-cni-plugins-kmh48\" (UID: \"79ae1195-4ff1-46da-b1e2-83042dba6caa\") " pod="openshift-multus/multus-additional-cni-plugins-kmh48" Oct 31 04:04:13.669875 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.669825 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c581e590-87e5-4cd7-8d84-eabb4ee21fd1-registration-dir\") pod \"aws-ebs-csi-driver-node-zw4rh\" (UID: \"c581e590-87e5-4cd7-8d84-eabb4ee21fd1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zw4rh" Oct 31 04:04:13.669875 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.669835 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d4eabffb-abf1-4409-94b5-c0489ab37853-sys\") pod \"tuned-b59p9\" (UID: \"d4eabffb-abf1-4409-94b5-c0489ab37853\") " pod="openshift-cluster-node-tuning-operator/tuned-b59p9" Oct 31 04:04:13.669875 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.669861 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d4eabffb-abf1-4409-94b5-c0489ab37853-var-lib-kubelet\") pod \"tuned-b59p9\" (UID: \"d4eabffb-abf1-4409-94b5-c0489ab37853\") " pod="openshift-cluster-node-tuning-operator/tuned-b59p9" Oct 31 04:04:13.669875 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.669870 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c581e590-87e5-4cd7-8d84-eabb4ee21fd1-socket-dir\") pod \"aws-ebs-csi-driver-node-zw4rh\" (UID: \"c581e590-87e5-4cd7-8d84-eabb4ee21fd1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zw4rh" Oct 31 04:04:13.669875 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.669877 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d4eabffb-abf1-4409-94b5-c0489ab37853-sys\") pod \"tuned-b59p9\" (UID: \"d4eabffb-abf1-4409-94b5-c0489ab37853\") " pod="openshift-cluster-node-tuning-operator/tuned-b59p9" Oct 31 04:04:13.669875 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.669882 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/153e979c-3c5d-4c4f-b838-fc26db0b5cac-hostroot\") pod \"multus-zmrmk\" (UID: \"153e979c-3c5d-4c4f-b838-fc26db0b5cac\") " pod="openshift-multus/multus-zmrmk" Oct 31 04:04:13.671290 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.669903 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a90157bb-46c0-4afa-82b1-9f6819518744-host\") pod \"node-ca-xmbxc\" (UID: \"a90157bb-46c0-4afa-82b1-9f6819518744\") " pod="openshift-image-registry/node-ca-xmbxc" Oct 31 04:04:13.671290 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.669911 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d4eabffb-abf1-4409-94b5-c0489ab37853-var-lib-kubelet\") pod \"tuned-b59p9\" (UID: \"d4eabffb-abf1-4409-94b5-c0489ab37853\") " pod="openshift-cluster-node-tuning-operator/tuned-b59p9" Oct 31 04:04:13.671290 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.669913 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d4eabffb-abf1-4409-94b5-c0489ab37853-etc-sysctl-conf\") pod \"tuned-b59p9\" (UID: \"d4eabffb-abf1-4409-94b5-c0489ab37853\") " pod="openshift-cluster-node-tuning-operator/tuned-b59p9" Oct 31 04:04:13.671290 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.669926 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c581e590-87e5-4cd7-8d84-eabb4ee21fd1-device-dir\") pod \"aws-ebs-csi-driver-node-zw4rh\" (UID: \"c581e590-87e5-4cd7-8d84-eabb4ee21fd1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zw4rh" Oct 31 04:04:13.671290 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.669937 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a90157bb-46c0-4afa-82b1-9f6819518744-host\") pod \"node-ca-xmbxc\" (UID: \"a90157bb-46c0-4afa-82b1-9f6819518744\") " pod="openshift-image-registry/node-ca-xmbxc" Oct 31 04:04:13.671290 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.669938 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/153e979c-3c5d-4c4f-b838-fc26db0b5cac-hostroot\") pod \"multus-zmrmk\" (UID: \"153e979c-3c5d-4c4f-b838-fc26db0b5cac\") " pod="openshift-multus/multus-zmrmk" Oct 31 04:04:13.671290 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.669945 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/79ae1195-4ff1-46da-b1e2-83042dba6caa-system-cni-dir\") pod \"multus-additional-cni-plugins-kmh48\" (UID: \"79ae1195-4ff1-46da-b1e2-83042dba6caa\") " pod="openshift-multus/multus-additional-cni-plugins-kmh48" Oct 31 04:04:13.671290 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.669960 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d4eabffb-abf1-4409-94b5-c0489ab37853-etc-modprobe-d\") pod \"tuned-b59p9\" (UID: \"d4eabffb-abf1-4409-94b5-c0489ab37853\") " pod="openshift-cluster-node-tuning-operator/tuned-b59p9" Oct 31 04:04:13.671290 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.669974 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d4eabffb-abf1-4409-94b5-c0489ab37853-host\") pod \"tuned-b59p9\" (UID: \"d4eabffb-abf1-4409-94b5-c0489ab37853\") " pod="openshift-cluster-node-tuning-operator/tuned-b59p9" Oct 31 04:04:13.671290 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.669981 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c581e590-87e5-4cd7-8d84-eabb4ee21fd1-device-dir\") pod \"aws-ebs-csi-driver-node-zw4rh\" (UID: \"c581e590-87e5-4cd7-8d84-eabb4ee21fd1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zw4rh" Oct 31 04:04:13.671290 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.669979 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/79ae1195-4ff1-46da-b1e2-83042dba6caa-system-cni-dir\") pod \"multus-additional-cni-plugins-kmh48\" (UID: \"79ae1195-4ff1-46da-b1e2-83042dba6caa\") " pod="openshift-multus/multus-additional-cni-plugins-kmh48" Oct 31 04:04:13.671290 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.670010 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d4eabffb-abf1-4409-94b5-c0489ab37853-host\") pod \"tuned-b59p9\" (UID: \"d4eabffb-abf1-4409-94b5-c0489ab37853\") " pod="openshift-cluster-node-tuning-operator/tuned-b59p9" Oct 31 04:04:13.671290 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.670021 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bc9d2733-89fa-4079-9dc1-01983df773cb-kubelet-config\") pod \"global-pull-secret-syncer-tlcrp\" (UID: \"bc9d2733-89fa-4079-9dc1-01983df773cb\") " pod="kube-system/global-pull-secret-syncer-tlcrp" Oct 31 04:04:13.671290 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.670056 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d4eabffb-abf1-4409-94b5-c0489ab37853-etc-modprobe-d\") pod \"tuned-b59p9\" (UID: \"d4eabffb-abf1-4409-94b5-c0489ab37853\") " pod="openshift-cluster-node-tuning-operator/tuned-b59p9" Oct 31 04:04:13.671290 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.670058 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f7a16817-8ea2-4649-9e42-3d340e1b9703-systemd-units\") pod \"ovnkube-node-zbh9t\" (UID: \"f7a16817-8ea2-4649-9e42-3d340e1b9703\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbh9t" Oct 31 04:04:13.671290 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.670076 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bc9d2733-89fa-4079-9dc1-01983df773cb-kubelet-config\") pod \"global-pull-secret-syncer-tlcrp\" (UID: \"bc9d2733-89fa-4079-9dc1-01983df773cb\") " pod="kube-system/global-pull-secret-syncer-tlcrp" Oct 31 04:04:13.671290 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.670081 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f7a16817-8ea2-4649-9e42-3d340e1b9703-systemd-units\") pod \"ovnkube-node-zbh9t\" (UID: \"f7a16817-8ea2-4649-9e42-3d340e1b9703\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbh9t" Oct 31 04:04:13.671290 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.670085 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c581e590-87e5-4cd7-8d84-eabb4ee21fd1-etc-selinux\") pod \"aws-ebs-csi-driver-node-zw4rh\" (UID: \"c581e590-87e5-4cd7-8d84-eabb4ee21fd1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zw4rh" Oct 31 04:04:13.671290 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.670171 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c581e590-87e5-4cd7-8d84-eabb4ee21fd1-etc-selinux\") pod \"aws-ebs-csi-driver-node-zw4rh\" (UID: \"c581e590-87e5-4cd7-8d84-eabb4ee21fd1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zw4rh" Oct 31 04:04:13.671290 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.670226 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f7a16817-8ea2-4649-9e42-3d340e1b9703-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zbh9t\" (UID: \"f7a16817-8ea2-4649-9e42-3d340e1b9703\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbh9t" Oct 31 04:04:13.671290 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.670114 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f7a16817-8ea2-4649-9e42-3d340e1b9703-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zbh9t\" (UID: \"f7a16817-8ea2-4649-9e42-3d340e1b9703\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbh9t" Oct 31 04:04:13.671290 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.670638 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1f2543cc-b528-41be-92e0-9d654405a723-hosts-file\") pod \"node-resolver-77z7p\" (UID: \"1f2543cc-b528-41be-92e0-9d654405a723\") " pod="openshift-dns/node-resolver-77z7p" Oct 31 04:04:13.671290 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.670663 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/153e979c-3c5d-4c4f-b838-fc26db0b5cac-host-var-lib-cni-bin\") pod \"multus-zmrmk\" (UID: \"153e979c-3c5d-4c4f-b838-fc26db0b5cac\") " pod="openshift-multus/multus-zmrmk" Oct 31 04:04:13.671290 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.670686 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f7a16817-8ea2-4649-9e42-3d340e1b9703-host-run-ovn-kubernetes\") pod \"ovnkube-node-zbh9t\" (UID: \"f7a16817-8ea2-4649-9e42-3d340e1b9703\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbh9t" Oct 31 04:04:13.671290 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.670709 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c581e590-87e5-4cd7-8d84-eabb4ee21fd1-sys-fs\") pod \"aws-ebs-csi-driver-node-zw4rh\" (UID: \"c581e590-87e5-4cd7-8d84-eabb4ee21fd1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zw4rh" Oct 31 04:04:13.671290 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.670728 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1f2543cc-b528-41be-92e0-9d654405a723-hosts-file\") pod \"node-resolver-77z7p\" (UID: \"1f2543cc-b528-41be-92e0-9d654405a723\") " pod="openshift-dns/node-resolver-77z7p" Oct 31 04:04:13.671290 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.670731 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/153e979c-3c5d-4c4f-b838-fc26db0b5cac-multus-conf-dir\") pod \"multus-zmrmk\" (UID: \"153e979c-3c5d-4c4f-b838-fc26db0b5cac\") " pod="openshift-multus/multus-zmrmk" Oct 31 04:04:13.671290 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.670730 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/153e979c-3c5d-4c4f-b838-fc26db0b5cac-host-var-lib-cni-bin\") pod \"multus-zmrmk\" (UID: \"153e979c-3c5d-4c4f-b838-fc26db0b5cac\") " pod="openshift-multus/multus-zmrmk" Oct 31 04:04:13.671290 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.670752 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f7a16817-8ea2-4649-9e42-3d340e1b9703-host-run-ovn-kubernetes\") pod \"ovnkube-node-zbh9t\" (UID: \"f7a16817-8ea2-4649-9e42-3d340e1b9703\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbh9t" Oct 31 04:04:13.671290 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.670757 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f7a16817-8ea2-4649-9e42-3d340e1b9703-host-cni-bin\") pod \"ovnkube-node-zbh9t\" (UID: \"f7a16817-8ea2-4649-9e42-3d340e1b9703\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbh9t" Oct 31 04:04:13.671290 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.670776 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/153e979c-3c5d-4c4f-b838-fc26db0b5cac-multus-conf-dir\") pod \"multus-zmrmk\" (UID: \"153e979c-3c5d-4c4f-b838-fc26db0b5cac\") " pod="openshift-multus/multus-zmrmk" Oct 31 04:04:13.671290 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.670780 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bc9d2733-89fa-4079-9dc1-01983df773cb-dbus\") pod \"global-pull-secret-syncer-tlcrp\" (UID: \"bc9d2733-89fa-4079-9dc1-01983df773cb\") " pod="kube-system/global-pull-secret-syncer-tlcrp" Oct 31 04:04:13.671290 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.670779 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c581e590-87e5-4cd7-8d84-eabb4ee21fd1-sys-fs\") pod \"aws-ebs-csi-driver-node-zw4rh\" (UID: \"c581e590-87e5-4cd7-8d84-eabb4ee21fd1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zw4rh" Oct 31 04:04:13.671290 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.670803 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f7a16817-8ea2-4649-9e42-3d340e1b9703-var-lib-openvswitch\") pod \"ovnkube-node-zbh9t\" (UID: \"f7a16817-8ea2-4649-9e42-3d340e1b9703\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbh9t" Oct 31 04:04:13.671290 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.670823 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f7a16817-8ea2-4649-9e42-3d340e1b9703-host-cni-bin\") pod \"ovnkube-node-zbh9t\" (UID: \"f7a16817-8ea2-4649-9e42-3d340e1b9703\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbh9t" Oct 31 04:04:13.671290 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.670826 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/153e979c-3c5d-4c4f-b838-fc26db0b5cac-host-var-lib-cni-multus\") pod \"multus-zmrmk\" (UID: \"153e979c-3c5d-4c4f-b838-fc26db0b5cac\") " pod="openshift-multus/multus-zmrmk" Oct 31 04:04:13.671290 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.670863 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f7a16817-8ea2-4649-9e42-3d340e1b9703-var-lib-openvswitch\") pod \"ovnkube-node-zbh9t\" (UID: \"f7a16817-8ea2-4649-9e42-3d340e1b9703\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbh9t" Oct 31 04:04:13.671290 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.670870 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/153e979c-3c5d-4c4f-b838-fc26db0b5cac-multus-cni-dir\") pod \"multus-zmrmk\" (UID: \"153e979c-3c5d-4c4f-b838-fc26db0b5cac\") " pod="openshift-multus/multus-zmrmk" Oct 31 04:04:13.671290 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.670894 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/153e979c-3c5d-4c4f-b838-fc26db0b5cac-host-run-netns\") pod \"multus-zmrmk\" (UID: \"153e979c-3c5d-4c4f-b838-fc26db0b5cac\") " pod="openshift-multus/multus-zmrmk" Oct 31 04:04:13.671290 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.670911 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f7a16817-8ea2-4649-9e42-3d340e1b9703-run-openvswitch\") pod \"ovnkube-node-zbh9t\" (UID: \"f7a16817-8ea2-4649-9e42-3d340e1b9703\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbh9t" Oct 31 04:04:13.671290 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.670924 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/153e979c-3c5d-4c4f-b838-fc26db0b5cac-multus-cni-dir\") pod \"multus-zmrmk\" (UID: \"153e979c-3c5d-4c4f-b838-fc26db0b5cac\") " pod="openshift-multus/multus-zmrmk" Oct 31 04:04:13.671290 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.670871 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bc9d2733-89fa-4079-9dc1-01983df773cb-dbus\") pod \"global-pull-secret-syncer-tlcrp\" (UID: \"bc9d2733-89fa-4079-9dc1-01983df773cb\") " pod="kube-system/global-pull-secret-syncer-tlcrp" Oct 31 04:04:13.671290 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.670871 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/153e979c-3c5d-4c4f-b838-fc26db0b5cac-host-var-lib-cni-multus\") pod \"multus-zmrmk\" (UID: \"153e979c-3c5d-4c4f-b838-fc26db0b5cac\") " pod="openshift-multus/multus-zmrmk" Oct 31 04:04:13.671290 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.670926 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d4eabffb-abf1-4409-94b5-c0489ab37853-etc-sysconfig\") pod \"tuned-b59p9\" (UID: \"d4eabffb-abf1-4409-94b5-c0489ab37853\") " pod="openshift-cluster-node-tuning-operator/tuned-b59p9" Oct 31 04:04:13.671290 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.670927 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/153e979c-3c5d-4c4f-b838-fc26db0b5cac-host-run-netns\") pod \"multus-zmrmk\" (UID: \"153e979c-3c5d-4c4f-b838-fc26db0b5cac\") " pod="openshift-multus/multus-zmrmk" Oct 31 04:04:13.671290 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.670958 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d4eabffb-abf1-4409-94b5-c0489ab37853-etc-sysconfig\") pod \"tuned-b59p9\" (UID: \"d4eabffb-abf1-4409-94b5-c0489ab37853\") " pod="openshift-cluster-node-tuning-operator/tuned-b59p9" Oct 31 04:04:13.671290 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.670974 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/79ae1195-4ff1-46da-b1e2-83042dba6caa-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kmh48\" (UID: \"79ae1195-4ff1-46da-b1e2-83042dba6caa\") " pod="openshift-multus/multus-additional-cni-plugins-kmh48" Oct 31 04:04:13.671290 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.670982 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f7a16817-8ea2-4649-9e42-3d340e1b9703-run-openvswitch\") pod \"ovnkube-node-zbh9t\" (UID: \"f7a16817-8ea2-4649-9e42-3d340e1b9703\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbh9t" Oct 31 04:04:13.671290 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.670999 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d4eabffb-abf1-4409-94b5-c0489ab37853-etc-sysctl-d\") pod \"tuned-b59p9\" (UID: \"d4eabffb-abf1-4409-94b5-c0489ab37853\") " pod="openshift-cluster-node-tuning-operator/tuned-b59p9" Oct 31 04:04:13.671290 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.671021 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/153e979c-3c5d-4c4f-b838-fc26db0b5cac-etc-kubernetes\") pod \"multus-zmrmk\" (UID: \"153e979c-3c5d-4c4f-b838-fc26db0b5cac\") " pod="openshift-multus/multus-zmrmk" Oct 31 04:04:13.671290 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.671037 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/79ae1195-4ff1-46da-b1e2-83042dba6caa-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kmh48\" (UID: \"79ae1195-4ff1-46da-b1e2-83042dba6caa\") " pod="openshift-multus/multus-additional-cni-plugins-kmh48" Oct 31 04:04:13.671290 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.671040 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f7a16817-8ea2-4649-9e42-3d340e1b9703-run-ovn\") pod \"ovnkube-node-zbh9t\" (UID: \"f7a16817-8ea2-4649-9e42-3d340e1b9703\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbh9t" Oct 31 04:04:13.671290 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.671062 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f7a16817-8ea2-4649-9e42-3d340e1b9703-run-ovn\") pod \"ovnkube-node-zbh9t\" (UID: \"f7a16817-8ea2-4649-9e42-3d340e1b9703\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbh9t" Oct 31 04:04:13.671290 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.671063 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d4eabffb-abf1-4409-94b5-c0489ab37853-etc-kubernetes\") pod \"tuned-b59p9\" (UID: \"d4eabffb-abf1-4409-94b5-c0489ab37853\") " pod="openshift-cluster-node-tuning-operator/tuned-b59p9" Oct 31 04:04:13.671290 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.671099 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/153e979c-3c5d-4c4f-b838-fc26db0b5cac-etc-kubernetes\") pod \"multus-zmrmk\" (UID: \"153e979c-3c5d-4c4f-b838-fc26db0b5cac\") " pod="openshift-multus/multus-zmrmk" Oct 31 04:04:13.671290 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.671121 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d4eabffb-abf1-4409-94b5-c0489ab37853-etc-kubernetes\") pod \"tuned-b59p9\" (UID: \"d4eabffb-abf1-4409-94b5-c0489ab37853\") " pod="openshift-cluster-node-tuning-operator/tuned-b59p9" Oct 31 04:04:13.671290 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:13.671129 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d4eabffb-abf1-4409-94b5-c0489ab37853-etc-sysctl-d\") pod \"tuned-b59p9\" (UID: \"d4eabffb-abf1-4409-94b5-c0489ab37853\") " pod="openshift-cluster-node-tuning-operator/tuned-b59p9" Oct 31 04:04:20.366462 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:20.366433 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zbh9t" Oct 31 04:04:21.974315 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:21.974288 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-tj4g2"] Oct 31 04:04:21.977642 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:21.977617 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-74d6m"] Oct 31 04:04:21.977780 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:21.977727 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-tj4g2" Oct 31 04:04:21.979088 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:21.979067 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-74d6m" Oct 31 04:04:22.001377 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:22.001355 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Oct 31 04:04:22.001516 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:22.001447 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Oct 31 04:04:22.001605 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:22.001573 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Oct 31 04:04:22.001667 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:22.001620 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-vmbk6\"" Oct 31 04:04:22.001721 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:22.001674 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-vvvxw\"" Oct 31 04:04:22.014842 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:22.014819 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-tj4g2"] Oct 31 04:04:22.015088 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:22.015071 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-74d6m"] Oct 31 04:04:22.027064 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:22.027035 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-hkmvc"] Oct 31 04:04:22.028677 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:22.028663 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-hkmvc" Oct 31 04:04:22.047267 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:22.047246 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 31 04:04:22.047466 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:22.047449 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 31 04:04:22.051204 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:22.051182 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-hkmvc"] Oct 31 04:04:22.068724 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:22.068691 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-fdtl5"] Oct 31 04:04:22.070417 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:22.070401 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-rv7hd"] Oct 31 04:04:22.070533 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:22.070516 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-fdtl5" Oct 31 04:04:22.071930 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:22.071914 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-799cd67bd8-fwz76"] Oct 31 04:04:22.072038 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:22.072024 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-rv7hd" Oct 31 04:04:22.073335 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:22.073313 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-799cd67bd8-fwz76" Oct 31 04:04:22.092651 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:22.092628 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-rv7hd"] Oct 31 04:04:22.095942 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:22.095922 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 31 04:04:22.096102 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:22.095974 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 31 04:04:22.096195 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:22.096144 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 31 04:04:22.097114 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:22.097090 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-fdtl5"] Oct 31 04:04:22.109900 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:22.109878 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 31 04:04:22.110022 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:22.109925 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Oct 31 04:04:22.110697 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:22.110682 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-799cd67bd8-fwz76"] Oct 31 04:04:22.111700 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:22.111688 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 31 04:04:22.112878 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:22.112863 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/a6c57686-4e0f-43ed-8ff0-843f3d9b3310-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-tj4g2\" (UID: \"a6c57686-4e0f-43ed-8ff0-843f3d9b3310\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-tj4g2" Oct 31 04:04:22.112932 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:22.112887 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b4d3b61b-4743-496c-9215-cd6766e296f1-cert\") pod \"ingress-canary-hkmvc\" (UID: \"b4d3b61b-4743-496c-9215-cd6766e296f1\") " pod="openshift-ingress-canary/ingress-canary-hkmvc" Oct 31 04:04:22.112932 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:22.112923 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/43d6c532-a6b9-4f34-8d70-8bab1a8c3fd9-tmp\") pod \"downloads-6dd789c8b7-74d6m\" (UID: \"43d6c532-a6b9-4f34-8d70-8bab1a8c3fd9\") " pod="openshift-console/downloads-6dd789c8b7-74d6m" Oct 31 04:04:22.112992 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:22.112941 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwzl6\" (UniqueName: \"kubernetes.io/projected/43d6c532-a6b9-4f34-8d70-8bab1a8c3fd9-kube-api-access-bwzl6\") pod \"downloads-6dd789c8b7-74d6m\" (UID: \"43d6c532-a6b9-4f34-8d70-8bab1a8c3fd9\") " pod="openshift-console/downloads-6dd789c8b7-74d6m" Oct 31 04:04:22.113039 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:22.113022 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwg7k\" (UniqueName: \"kubernetes.io/projected/b4d3b61b-4743-496c-9215-cd6766e296f1-kube-api-access-pwg7k\") pod \"ingress-canary-hkmvc\" (UID: \"b4d3b61b-4743-496c-9215-cd6766e296f1\") " pod="openshift-ingress-canary/ingress-canary-hkmvc" Oct 31 04:04:22.115295 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:22.115274 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Oct 31 04:04:22.115408 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:22.115296 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Oct 31 04:04:22.115408 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:22.115316 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 31 04:04:22.115408 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:22.115360 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-qzkqw\"" Oct 31 04:04:22.115408 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:22.115371 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 31 04:04:22.115408 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:22.115366 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-gw6dk\"" Oct 31 04:04:22.115408 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:22.115404 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Oct 31 04:04:22.115680 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:22.115669 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-8w2pm\"" Oct 31 04:04:22.115770 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:22.115760 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-gmtbq\"" Oct 31 04:04:22.213996 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:22.213957 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3ae04333-9598-482c-ae38-8de983d7a48f-metrics-tls\") pod \"dns-default-rv7hd\" (UID: \"3ae04333-9598-482c-ae38-8de983d7a48f\") " pod="openshift-dns/dns-default-rv7hd" Oct 31 04:04:22.213996 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:22.213999 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/a6c57686-4e0f-43ed-8ff0-843f3d9b3310-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-tj4g2\" (UID: \"a6c57686-4e0f-43ed-8ff0-843f3d9b3310\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-tj4g2" Oct 31 04:04:22.214206 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:22.214019 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7ae6197e-7c25-4b01-a0f2-d70621c8ad50-crio-socket\") pod \"insights-runtime-extractor-fdtl5\" (UID: \"7ae6197e-7c25-4b01-a0f2-d70621c8ad50\") " pod="openshift-insights/insights-runtime-extractor-fdtl5" Oct 31 04:04:22.214206 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:22.214071 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b4d3b61b-4743-496c-9215-cd6766e296f1-cert\") pod \"ingress-canary-hkmvc\" (UID: \"b4d3b61b-4743-496c-9215-cd6766e296f1\") " pod="openshift-ingress-canary/ingress-canary-hkmvc" Oct 31 04:04:22.214206 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:22.214155 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7ae6197e-7c25-4b01-a0f2-d70621c8ad50-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fdtl5\" (UID: \"7ae6197e-7c25-4b01-a0f2-d70621c8ad50\") " pod="openshift-insights/insights-runtime-extractor-fdtl5" Oct 31 04:04:22.214313 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:22.214243 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7ae6197e-7c25-4b01-a0f2-d70621c8ad50-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fdtl5\" (UID: \"7ae6197e-7c25-4b01-a0f2-d70621c8ad50\") " pod="openshift-insights/insights-runtime-extractor-fdtl5" Oct 31 04:04:22.214313 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:22.214274 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3ae04333-9598-482c-ae38-8de983d7a48f-config-volume\") pod \"dns-default-rv7hd\" (UID: \"3ae04333-9598-482c-ae38-8de983d7a48f\") " pod="openshift-dns/dns-default-rv7hd" Oct 31 04:04:22.214313 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:22.214293 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3e67fa28-1a34-48bd-be16-373044df6f41-registry-certificates\") pod \"image-registry-799cd67bd8-fwz76\" (UID: \"3e67fa28-1a34-48bd-be16-373044df6f41\") " pod="openshift-image-registry/image-registry-799cd67bd8-fwz76" Oct 31 04:04:22.214313 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:22.214311 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3e67fa28-1a34-48bd-be16-373044df6f41-bound-sa-token\") pod \"image-registry-799cd67bd8-fwz76\" (UID: \"3e67fa28-1a34-48bd-be16-373044df6f41\") " pod="openshift-image-registry/image-registry-799cd67bd8-fwz76" Oct 31 04:04:22.214506 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:22.214418 4376 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 31 04:04:22.214506 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:22.214490 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/43d6c532-a6b9-4f34-8d70-8bab1a8c3fd9-tmp\") pod \"downloads-6dd789c8b7-74d6m\" (UID: \"43d6c532-a6b9-4f34-8d70-8bab1a8c3fd9\") " pod="openshift-console/downloads-6dd789c8b7-74d6m" Oct 31 04:04:22.214593 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:22.214526 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3e67fa28-1a34-48bd-be16-373044df6f41-trusted-ca\") pod \"image-registry-799cd67bd8-fwz76\" (UID: \"3e67fa28-1a34-48bd-be16-373044df6f41\") " pod="openshift-image-registry/image-registry-799cd67bd8-fwz76" Oct 31 04:04:22.214593 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:22.214558 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwzl6\" (UniqueName: \"kubernetes.io/projected/43d6c532-a6b9-4f34-8d70-8bab1a8c3fd9-kube-api-access-bwzl6\") pod \"downloads-6dd789c8b7-74d6m\" (UID: \"43d6c532-a6b9-4f34-8d70-8bab1a8c3fd9\") " pod="openshift-console/downloads-6dd789c8b7-74d6m" Oct 31 04:04:22.214696 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:22.214613 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3e67fa28-1a34-48bd-be16-373044df6f41-ca-trust-extracted\") pod \"image-registry-799cd67bd8-fwz76\" (UID: \"3e67fa28-1a34-48bd-be16-373044df6f41\") " pod="openshift-image-registry/image-registry-799cd67bd8-fwz76" Oct 31 04:04:22.214696 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:22.214648 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2tnv\" (UniqueName: \"kubernetes.io/projected/3e67fa28-1a34-48bd-be16-373044df6f41-kube-api-access-c2tnv\") pod \"image-registry-799cd67bd8-fwz76\" (UID: \"3e67fa28-1a34-48bd-be16-373044df6f41\") " pod="openshift-image-registry/image-registry-799cd67bd8-fwz76" Oct 31 04:04:22.214793 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:22.214710 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwg7k\" (UniqueName: \"kubernetes.io/projected/b4d3b61b-4743-496c-9215-cd6766e296f1-kube-api-access-pwg7k\") pod \"ingress-canary-hkmvc\" (UID: \"b4d3b61b-4743-496c-9215-cd6766e296f1\") " pod="openshift-ingress-canary/ingress-canary-hkmvc" Oct 31 04:04:22.214793 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:22.214745 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2wx8\" (UniqueName: \"kubernetes.io/projected/3ae04333-9598-482c-ae38-8de983d7a48f-kube-api-access-f2wx8\") pod \"dns-default-rv7hd\" (UID: \"3ae04333-9598-482c-ae38-8de983d7a48f\") " pod="openshift-dns/dns-default-rv7hd" Oct 31 04:04:22.214900 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:22.214878 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3e67fa28-1a34-48bd-be16-373044df6f41-installation-pull-secrets\") pod \"image-registry-799cd67bd8-fwz76\" (UID: \"3e67fa28-1a34-48bd-be16-373044df6f41\") " pod="openshift-image-registry/image-registry-799cd67bd8-fwz76" Oct 31 04:04:22.214952 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:22.214905 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/43d6c532-a6b9-4f34-8d70-8bab1a8c3fd9-tmp\") pod \"downloads-6dd789c8b7-74d6m\" (UID: \"43d6c532-a6b9-4f34-8d70-8bab1a8c3fd9\") " pod="openshift-console/downloads-6dd789c8b7-74d6m" Oct 31 04:04:22.214952 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:22.214920 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ae04333-9598-482c-ae38-8de983d7a48f-tmp-dir\") pod \"dns-default-rv7hd\" (UID: \"3ae04333-9598-482c-ae38-8de983d7a48f\") " pod="openshift-dns/dns-default-rv7hd" Oct 31 04:04:22.215031 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:22.214999 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3e67fa28-1a34-48bd-be16-373044df6f41-registry-tls\") pod \"image-registry-799cd67bd8-fwz76\" (UID: \"3e67fa28-1a34-48bd-be16-373044df6f41\") " pod="openshift-image-registry/image-registry-799cd67bd8-fwz76" Oct 31 04:04:22.215071 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:22.215043 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7ae6197e-7c25-4b01-a0f2-d70621c8ad50-data-volume\") pod \"insights-runtime-extractor-fdtl5\" (UID: \"7ae6197e-7c25-4b01-a0f2-d70621c8ad50\") " pod="openshift-insights/insights-runtime-extractor-fdtl5" Oct 31 04:04:22.215110 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:22.215072 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qc28z\" (UniqueName: \"kubernetes.io/projected/7ae6197e-7c25-4b01-a0f2-d70621c8ad50-kube-api-access-qc28z\") pod \"insights-runtime-extractor-fdtl5\" (UID: \"7ae6197e-7c25-4b01-a0f2-d70621c8ad50\") " pod="openshift-insights/insights-runtime-extractor-fdtl5" Oct 31 04:04:22.215110 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:22.215099 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3e67fa28-1a34-48bd-be16-373044df6f41-image-registry-private-configuration\") pod \"image-registry-799cd67bd8-fwz76\" (UID: \"3e67fa28-1a34-48bd-be16-373044df6f41\") " pod="openshift-image-registry/image-registry-799cd67bd8-fwz76" Oct 31 04:04:22.216643 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:22.216618 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b4d3b61b-4743-496c-9215-cd6766e296f1-cert\") pod \"ingress-canary-hkmvc\" (UID: \"b4d3b61b-4743-496c-9215-cd6766e296f1\") " pod="openshift-ingress-canary/ingress-canary-hkmvc" Oct 31 04:04:22.216779 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:22.216762 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/a6c57686-4e0f-43ed-8ff0-843f3d9b3310-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-tj4g2\" (UID: \"a6c57686-4e0f-43ed-8ff0-843f3d9b3310\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-tj4g2" Oct 31 04:04:22.242921 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:22.242845 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwzl6\" (UniqueName: \"kubernetes.io/projected/43d6c532-a6b9-4f34-8d70-8bab1a8c3fd9-kube-api-access-bwzl6\") pod \"downloads-6dd789c8b7-74d6m\" (UID: \"43d6c532-a6b9-4f34-8d70-8bab1a8c3fd9\") " pod="openshift-console/downloads-6dd789c8b7-74d6m" Oct 31 04:04:22.244057 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:22.244038 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwg7k\" (UniqueName: \"kubernetes.io/projected/b4d3b61b-4743-496c-9215-cd6766e296f1-kube-api-access-pwg7k\") pod \"ingress-canary-hkmvc\" (UID: \"b4d3b61b-4743-496c-9215-cd6766e296f1\") " pod="openshift-ingress-canary/ingress-canary-hkmvc" Oct 31 04:04:22.289708 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:22.289679 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-tj4g2" Oct 31 04:04:22.294799 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:22.294775 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-74d6m" Oct 31 04:04:22.315771 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:22.315726 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2wx8\" (UniqueName: \"kubernetes.io/projected/3ae04333-9598-482c-ae38-8de983d7a48f-kube-api-access-f2wx8\") pod \"dns-default-rv7hd\" (UID: \"3ae04333-9598-482c-ae38-8de983d7a48f\") " pod="openshift-dns/dns-default-rv7hd" Oct 31 04:04:22.315941 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:22.315793 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3e67fa28-1a34-48bd-be16-373044df6f41-installation-pull-secrets\") pod \"image-registry-799cd67bd8-fwz76\" (UID: \"3e67fa28-1a34-48bd-be16-373044df6f41\") " pod="openshift-image-registry/image-registry-799cd67bd8-fwz76" Oct 31 04:04:22.315941 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:22.315836 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ae04333-9598-482c-ae38-8de983d7a48f-tmp-dir\") pod \"dns-default-rv7hd\" (UID: \"3ae04333-9598-482c-ae38-8de983d7a48f\") " pod="openshift-dns/dns-default-rv7hd" Oct 31 04:04:22.315941 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:22.315871 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3e67fa28-1a34-48bd-be16-373044df6f41-registry-tls\") pod \"image-registry-799cd67bd8-fwz76\" (UID: \"3e67fa28-1a34-48bd-be16-373044df6f41\") " pod="openshift-image-registry/image-registry-799cd67bd8-fwz76" Oct 31 04:04:22.315941 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:22.315904 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7ae6197e-7c25-4b01-a0f2-d70621c8ad50-data-volume\") pod \"insights-runtime-extractor-fdtl5\" (UID: \"7ae6197e-7c25-4b01-a0f2-d70621c8ad50\") " pod="openshift-insights/insights-runtime-extractor-fdtl5" Oct 31 04:04:22.315941 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:22.315930 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qc28z\" (UniqueName: \"kubernetes.io/projected/7ae6197e-7c25-4b01-a0f2-d70621c8ad50-kube-api-access-qc28z\") pod \"insights-runtime-extractor-fdtl5\" (UID: \"7ae6197e-7c25-4b01-a0f2-d70621c8ad50\") " pod="openshift-insights/insights-runtime-extractor-fdtl5" Oct 31 04:04:22.316217 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:22.315958 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3e67fa28-1a34-48bd-be16-373044df6f41-image-registry-private-configuration\") pod \"image-registry-799cd67bd8-fwz76\" (UID: \"3e67fa28-1a34-48bd-be16-373044df6f41\") " pod="openshift-image-registry/image-registry-799cd67bd8-fwz76" Oct 31 04:04:22.316217 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:22.315991 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3ae04333-9598-482c-ae38-8de983d7a48f-metrics-tls\") pod \"dns-default-rv7hd\" (UID: \"3ae04333-9598-482c-ae38-8de983d7a48f\") " pod="openshift-dns/dns-default-rv7hd" Oct 31 04:04:22.316217 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:22.316020 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7ae6197e-7c25-4b01-a0f2-d70621c8ad50-crio-socket\") pod \"insights-runtime-extractor-fdtl5\" (UID: \"7ae6197e-7c25-4b01-a0f2-d70621c8ad50\") " pod="openshift-insights/insights-runtime-extractor-fdtl5" Oct 31 04:04:22.316217 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:22.316046 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7ae6197e-7c25-4b01-a0f2-d70621c8ad50-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fdtl5\" (UID: \"7ae6197e-7c25-4b01-a0f2-d70621c8ad50\") " pod="openshift-insights/insights-runtime-extractor-fdtl5" Oct 31 04:04:22.316217 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:22.316071 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7ae6197e-7c25-4b01-a0f2-d70621c8ad50-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fdtl5\" (UID: \"7ae6197e-7c25-4b01-a0f2-d70621c8ad50\") " pod="openshift-insights/insights-runtime-extractor-fdtl5" Oct 31 04:04:22.316217 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:22.316096 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3ae04333-9598-482c-ae38-8de983d7a48f-config-volume\") pod \"dns-default-rv7hd\" (UID: \"3ae04333-9598-482c-ae38-8de983d7a48f\") " pod="openshift-dns/dns-default-rv7hd" Oct 31 04:04:22.316217 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:22.316119 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3e67fa28-1a34-48bd-be16-373044df6f41-registry-certificates\") pod \"image-registry-799cd67bd8-fwz76\" (UID: \"3e67fa28-1a34-48bd-be16-373044df6f41\") " pod="openshift-image-registry/image-registry-799cd67bd8-fwz76" Oct 31 04:04:22.316217 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:22.316146 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3e67fa28-1a34-48bd-be16-373044df6f41-bound-sa-token\") pod \"image-registry-799cd67bd8-fwz76\" (UID: \"3e67fa28-1a34-48bd-be16-373044df6f41\") " pod="openshift-image-registry/image-registry-799cd67bd8-fwz76" Oct 31 04:04:22.316217 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:22.316197 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3e67fa28-1a34-48bd-be16-373044df6f41-trusted-ca\") pod \"image-registry-799cd67bd8-fwz76\" (UID: \"3e67fa28-1a34-48bd-be16-373044df6f41\") " pod="openshift-image-registry/image-registry-799cd67bd8-fwz76" Oct 31 04:04:22.317785 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:22.316223 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3e67fa28-1a34-48bd-be16-373044df6f41-ca-trust-extracted\") pod \"image-registry-799cd67bd8-fwz76\" (UID: \"3e67fa28-1a34-48bd-be16-373044df6f41\") " pod="openshift-image-registry/image-registry-799cd67bd8-fwz76" Oct 31 04:04:22.317785 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:22.316235 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ae04333-9598-482c-ae38-8de983d7a48f-tmp-dir\") pod \"dns-default-rv7hd\" (UID: \"3ae04333-9598-482c-ae38-8de983d7a48f\") " pod="openshift-dns/dns-default-rv7hd" Oct 31 04:04:22.317785 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:22.316248 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2tnv\" (UniqueName: \"kubernetes.io/projected/3e67fa28-1a34-48bd-be16-373044df6f41-kube-api-access-c2tnv\") pod \"image-registry-799cd67bd8-fwz76\" (UID: \"3e67fa28-1a34-48bd-be16-373044df6f41\") " pod="openshift-image-registry/image-registry-799cd67bd8-fwz76" Oct 31 04:04:22.317785 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:22.316770 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3e67fa28-1a34-48bd-be16-373044df6f41-ca-trust-extracted\") pod \"image-registry-799cd67bd8-fwz76\" (UID: \"3e67fa28-1a34-48bd-be16-373044df6f41\") " pod="openshift-image-registry/image-registry-799cd67bd8-fwz76" Oct 31 04:04:22.317785 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:22.316855 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3ae04333-9598-482c-ae38-8de983d7a48f-config-volume\") pod \"dns-default-rv7hd\" (UID: \"3ae04333-9598-482c-ae38-8de983d7a48f\") " pod="openshift-dns/dns-default-rv7hd" Oct 31 04:04:22.317785 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:22.316925 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7ae6197e-7c25-4b01-a0f2-d70621c8ad50-crio-socket\") pod \"insights-runtime-extractor-fdtl5\" (UID: \"7ae6197e-7c25-4b01-a0f2-d70621c8ad50\") " pod="openshift-insights/insights-runtime-extractor-fdtl5" Oct 31 04:04:22.317785 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:22.316995 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7ae6197e-7c25-4b01-a0f2-d70621c8ad50-data-volume\") pod \"insights-runtime-extractor-fdtl5\" (UID: \"7ae6197e-7c25-4b01-a0f2-d70621c8ad50\") " pod="openshift-insights/insights-runtime-extractor-fdtl5" Oct 31 04:04:22.317785 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:22.317201 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7ae6197e-7c25-4b01-a0f2-d70621c8ad50-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fdtl5\" (UID: \"7ae6197e-7c25-4b01-a0f2-d70621c8ad50\") " pod="openshift-insights/insights-runtime-extractor-fdtl5" Oct 31 04:04:22.317785 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:22.317589 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3e67fa28-1a34-48bd-be16-373044df6f41-registry-certificates\") pod \"image-registry-799cd67bd8-fwz76\" (UID: \"3e67fa28-1a34-48bd-be16-373044df6f41\") " pod="openshift-image-registry/image-registry-799cd67bd8-fwz76" Oct 31 04:04:22.318166 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:22.318030 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3e67fa28-1a34-48bd-be16-373044df6f41-trusted-ca\") pod \"image-registry-799cd67bd8-fwz76\" (UID: \"3e67fa28-1a34-48bd-be16-373044df6f41\") " pod="openshift-image-registry/image-registry-799cd67bd8-fwz76" Oct 31 04:04:22.319275 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:22.319251 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3ae04333-9598-482c-ae38-8de983d7a48f-metrics-tls\") pod \"dns-default-rv7hd\" (UID: \"3ae04333-9598-482c-ae38-8de983d7a48f\") " pod="openshift-dns/dns-default-rv7hd" Oct 31 04:04:22.319668 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:22.319652 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3e67fa28-1a34-48bd-be16-373044df6f41-installation-pull-secrets\") pod \"image-registry-799cd67bd8-fwz76\" (UID: \"3e67fa28-1a34-48bd-be16-373044df6f41\") " pod="openshift-image-registry/image-registry-799cd67bd8-fwz76" Oct 31 04:04:22.320065 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:22.320044 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7ae6197e-7c25-4b01-a0f2-d70621c8ad50-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fdtl5\" (UID: \"7ae6197e-7c25-4b01-a0f2-d70621c8ad50\") " pod="openshift-insights/insights-runtime-extractor-fdtl5" Oct 31 04:04:22.321153 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:22.321128 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3e67fa28-1a34-48bd-be16-373044df6f41-image-registry-private-configuration\") pod \"image-registry-799cd67bd8-fwz76\" (UID: \"3e67fa28-1a34-48bd-be16-373044df6f41\") " pod="openshift-image-registry/image-registry-799cd67bd8-fwz76" Oct 31 04:04:22.321487 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:22.321467 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3e67fa28-1a34-48bd-be16-373044df6f41-registry-tls\") pod \"image-registry-799cd67bd8-fwz76\" (UID: \"3e67fa28-1a34-48bd-be16-373044df6f41\") " pod="openshift-image-registry/image-registry-799cd67bd8-fwz76" Oct 31 04:04:22.341492 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:22.340371 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-hkmvc" Oct 31 04:04:22.344407 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:22.343652 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3e67fa28-1a34-48bd-be16-373044df6f41-bound-sa-token\") pod \"image-registry-799cd67bd8-fwz76\" (UID: \"3e67fa28-1a34-48bd-be16-373044df6f41\") " pod="openshift-image-registry/image-registry-799cd67bd8-fwz76" Oct 31 04:04:22.344407 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:22.343847 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2tnv\" (UniqueName: \"kubernetes.io/projected/3e67fa28-1a34-48bd-be16-373044df6f41-kube-api-access-c2tnv\") pod \"image-registry-799cd67bd8-fwz76\" (UID: \"3e67fa28-1a34-48bd-be16-373044df6f41\") " pod="openshift-image-registry/image-registry-799cd67bd8-fwz76" Oct 31 04:04:22.344407 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:22.344139 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qc28z\" (UniqueName: \"kubernetes.io/projected/7ae6197e-7c25-4b01-a0f2-d70621c8ad50-kube-api-access-qc28z\") pod \"insights-runtime-extractor-fdtl5\" (UID: \"7ae6197e-7c25-4b01-a0f2-d70621c8ad50\") " pod="openshift-insights/insights-runtime-extractor-fdtl5" Oct 31 04:04:22.346558 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:22.346077 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2wx8\" (UniqueName: \"kubernetes.io/projected/3ae04333-9598-482c-ae38-8de983d7a48f-kube-api-access-f2wx8\") pod \"dns-default-rv7hd\" (UID: \"3ae04333-9598-482c-ae38-8de983d7a48f\") " pod="openshift-dns/dns-default-rv7hd" Oct 31 04:04:22.383046 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:22.382968 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-fdtl5" Oct 31 04:04:22.388117 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:22.388097 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-rv7hd" Oct 31 04:04:22.393512 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:22.392816 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-799cd67bd8-fwz76" Oct 31 04:04:22.582880 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:22.582853 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-tj4g2"] Oct 31 04:04:22.583190 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:22.583166 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6c57686_4e0f_43ed_8ff0_843f3d9b3310.slice/crio-de7cf33c53b636996e836e41f17cddb04cebf682a9d5f32854ad02a4724c2bcc WatchSource:0}: Error finding container de7cf33c53b636996e836e41f17cddb04cebf682a9d5f32854ad02a4724c2bcc: Status 404 returned error can't find the container with id de7cf33c53b636996e836e41f17cddb04cebf682a9d5f32854ad02a4724c2bcc Oct 31 04:04:22.585066 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:22.585051 4376 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 04:04:22.594216 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:22.594193 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-74d6m"] Oct 31 04:04:22.595613 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:22.595592 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43d6c532_a6b9_4f34_8d70_8bab1a8c3fd9.slice/crio-d8ce77977a8a40d43412273131e20973e64b7edef01167e675cb165613af8f4e WatchSource:0}: Error finding container d8ce77977a8a40d43412273131e20973e64b7edef01167e675cb165613af8f4e: Status 404 returned error can't find the container with id d8ce77977a8a40d43412273131e20973e64b7edef01167e675cb165613af8f4e Oct 31 04:04:22.638987 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:22.638958 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-hkmvc"] Oct 31 04:04:22.639511 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:22.639479 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4d3b61b_4743_496c_9215_cd6766e296f1.slice/crio-0640f76c3e2c27bc85974c19d105e5d31b673b6d876bc6a48105697bc283535c WatchSource:0}: Error finding container 0640f76c3e2c27bc85974c19d105e5d31b673b6d876bc6a48105697bc283535c: Status 404 returned error can't find the container with id 0640f76c3e2c27bc85974c19d105e5d31b673b6d876bc6a48105697bc283535c Oct 31 04:04:22.710361 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:22.710269 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-rv7hd"] Oct 31 04:04:22.713705 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:22.713680 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-fdtl5"] Oct 31 04:04:22.714163 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:22.714142 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-799cd67bd8-fwz76"] Oct 31 04:04:22.714759 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:22.714738 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e67fa28_1a34_48bd_be16_373044df6f41.slice/crio-50a6fb3e70ad013ac4712e78b217929f745fdfbcbaa2689c0013fc5dab502e48 WatchSource:0}: Error finding container 50a6fb3e70ad013ac4712e78b217929f745fdfbcbaa2689c0013fc5dab502e48: Status 404 returned error can't find the container with id 50a6fb3e70ad013ac4712e78b217929f745fdfbcbaa2689c0013fc5dab502e48 Oct 31 04:04:22.715239 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:22.715206 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ae6197e_7c25_4b01_a0f2_d70621c8ad50.slice/crio-61dd62fdb04bc81ec6aab9d460cf3f412e29b33b9146f716c24eb1566f5bd840 WatchSource:0}: Error finding container 61dd62fdb04bc81ec6aab9d460cf3f412e29b33b9146f716c24eb1566f5bd840: Status 404 returned error can't find the container with id 61dd62fdb04bc81ec6aab9d460cf3f412e29b33b9146f716c24eb1566f5bd840 Oct 31 04:04:23.345245 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:23.344392 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-w9sdm" Oct 31 04:04:23.499916 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:23.499873 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rv7hd" event={"ID":"3ae04333-9598-482c-ae38-8de983d7a48f","Type":"ContainerStarted","Data":"8b5f6c1e391e64bce2a81fc78ad20816ad93fbd08998967ff53a01fe23aa7cda"} Oct 31 04:04:23.503205 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:23.503170 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-799cd67bd8-fwz76" event={"ID":"3e67fa28-1a34-48bd-be16-373044df6f41","Type":"ContainerStarted","Data":"944a3889d62c1ed125af846a661f10521f9d092cc0a43837cc0ce4ebf7263f1c"} Oct 31 04:04:23.503321 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:23.503231 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-799cd67bd8-fwz76" event={"ID":"3e67fa28-1a34-48bd-be16-373044df6f41","Type":"ContainerStarted","Data":"50a6fb3e70ad013ac4712e78b217929f745fdfbcbaa2689c0013fc5dab502e48"} Oct 31 04:04:23.507716 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:23.507606 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-fdtl5" event={"ID":"7ae6197e-7c25-4b01-a0f2-d70621c8ad50","Type":"ContainerStarted","Data":"101ba948c491456fae50012ec3fff2ab26f5fd35f562d3ea991411fd9dd80b24"} Oct 31 04:04:23.507716 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:23.507661 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-fdtl5" event={"ID":"7ae6197e-7c25-4b01-a0f2-d70621c8ad50","Type":"ContainerStarted","Data":"61dd62fdb04bc81ec6aab9d460cf3f412e29b33b9146f716c24eb1566f5bd840"} Oct 31 04:04:23.509520 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:23.509456 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-hkmvc" event={"ID":"b4d3b61b-4743-496c-9215-cd6766e296f1","Type":"ContainerStarted","Data":"0640f76c3e2c27bc85974c19d105e5d31b673b6d876bc6a48105697bc283535c"} Oct 31 04:04:23.512602 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:23.512576 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-74d6m" event={"ID":"43d6c532-a6b9-4f34-8d70-8bab1a8c3fd9","Type":"ContainerStarted","Data":"d8ce77977a8a40d43412273131e20973e64b7edef01167e675cb165613af8f4e"} Oct 31 04:04:23.515089 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:23.515061 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-tj4g2" event={"ID":"a6c57686-4e0f-43ed-8ff0-843f3d9b3310","Type":"ContainerStarted","Data":"de7cf33c53b636996e836e41f17cddb04cebf682a9d5f32854ad02a4724c2bcc"} Oct 31 04:04:23.549557 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:23.549484 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-799cd67bd8-fwz76" podStartSLOduration=1.549466526 podStartE2EDuration="1.549466526s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:04:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:04:23.549285589 +0000 UTC m=+11.263620206" watchObservedRunningTime="2026-10-31 04:04:23.549466526 +0000 UTC m=+11.263801136" Oct 31 04:04:23.690052 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:23.690019 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zbh9t" Oct 31 04:04:23.691665 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:23.690278 4376 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 31 04:04:24.522784 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:24.522690 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-fdtl5" event={"ID":"7ae6197e-7c25-4b01-a0f2-d70621c8ad50","Type":"ContainerStarted","Data":"107e9e24142402f9d2f37bddc2dca6a3cb7d82c62472110b4a26c93b6d3ccc9b"} Oct 31 04:04:24.524613 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:24.524587 4376 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 31 04:04:24.525432 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:24.525402 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-tj4g2" event={"ID":"a6c57686-4e0f-43ed-8ff0-843f3d9b3310","Type":"ContainerStarted","Data":"700d2ceb53081033bda998d22659655709b0e0c1659fe6f0e5b4fb845928f688"} Oct 31 04:04:24.525516 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:24.525460 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-tj4g2" Oct 31 04:04:24.531882 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:24.531857 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-tj4g2" Oct 31 04:04:24.607842 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:24.607778 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-tj4g2" podStartSLOduration=2.038663099 podStartE2EDuration="3.607762148s" totalImagesPullingTime="1.569099049s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:04:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:04:24.566840117 +0000 UTC m=+12.281174744" watchObservedRunningTime="2026-10-31 04:04:24.607762148 +0000 UTC m=+12.322096765" Oct 31 04:04:25.409501 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:25.409469 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-dn78l"] Oct 31 04:04:25.412912 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:25.412889 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-dn78l" Oct 31 04:04:25.444445 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:25.444402 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-dn78l"] Oct 31 04:04:25.444614 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:25.444551 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Oct 31 04:04:25.444675 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:25.444641 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 31 04:04:25.444796 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:25.444780 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 31 04:04:25.444933 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:25.444919 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 31 04:04:25.445053 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:25.445038 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Oct 31 04:04:25.456697 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:25.456650 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-hrgj9\"" Oct 31 04:04:25.534548 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:25.534508 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rv7hd" event={"ID":"3ae04333-9598-482c-ae38-8de983d7a48f","Type":"ContainerStarted","Data":"8cb0b162c38c90df8c1233b2d0fe7e1c499a7a1a74ca62aa30621276c527ea7f"} Oct 31 04:04:25.534993 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:25.534559 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rv7hd" event={"ID":"3ae04333-9598-482c-ae38-8de983d7a48f","Type":"ContainerStarted","Data":"0022fd4d6fddb265bf22d0d3954645191396109c7f836f47537026e08e7f0a39"} Oct 31 04:04:25.534993 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:25.534606 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-rv7hd" Oct 31 04:04:25.560636 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:25.560565 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/55a51982-5071-4b52-9ce3-a6eadc5505cc-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-dn78l\" (UID: \"55a51982-5071-4b52-9ce3-a6eadc5505cc\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-dn78l" Oct 31 04:04:25.560636 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:25.560608 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/55a51982-5071-4b52-9ce3-a6eadc5505cc-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-dn78l\" (UID: \"55a51982-5071-4b52-9ce3-a6eadc5505cc\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-dn78l" Oct 31 04:04:25.560831 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:25.560703 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76mjd\" (UniqueName: \"kubernetes.io/projected/55a51982-5071-4b52-9ce3-a6eadc5505cc-kube-api-access-76mjd\") pod \"prometheus-operator-9cd89b74c-dn78l\" (UID: \"55a51982-5071-4b52-9ce3-a6eadc5505cc\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-dn78l" Oct 31 04:04:25.560831 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:25.560759 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/55a51982-5071-4b52-9ce3-a6eadc5505cc-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-dn78l\" (UID: \"55a51982-5071-4b52-9ce3-a6eadc5505cc\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-dn78l" Oct 31 04:04:25.577757 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:25.577163 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-rv7hd" podStartSLOduration=1.2434228 podStartE2EDuration="3.577151566s" totalImagesPullingTime="2.333728766s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:04:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:04:25.576872603 +0000 UTC m=+13.291207220" watchObservedRunningTime="2026-10-31 04:04:25.577151566 +0000 UTC m=+13.291486180" Oct 31 04:04:25.662122 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:25.662082 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/55a51982-5071-4b52-9ce3-a6eadc5505cc-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-dn78l\" (UID: \"55a51982-5071-4b52-9ce3-a6eadc5505cc\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-dn78l" Oct 31 04:04:25.662291 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:25.662221 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/55a51982-5071-4b52-9ce3-a6eadc5505cc-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-dn78l\" (UID: \"55a51982-5071-4b52-9ce3-a6eadc5505cc\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-dn78l" Oct 31 04:04:25.662291 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:25.662251 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/55a51982-5071-4b52-9ce3-a6eadc5505cc-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-dn78l\" (UID: \"55a51982-5071-4b52-9ce3-a6eadc5505cc\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-dn78l" Oct 31 04:04:25.662448 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:25.662291 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76mjd\" (UniqueName: \"kubernetes.io/projected/55a51982-5071-4b52-9ce3-a6eadc5505cc-kube-api-access-76mjd\") pod \"prometheus-operator-9cd89b74c-dn78l\" (UID: \"55a51982-5071-4b52-9ce3-a6eadc5505cc\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-dn78l" Oct 31 04:04:25.663635 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:25.663609 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/55a51982-5071-4b52-9ce3-a6eadc5505cc-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-dn78l\" (UID: \"55a51982-5071-4b52-9ce3-a6eadc5505cc\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-dn78l" Oct 31 04:04:25.668857 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:25.668833 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/55a51982-5071-4b52-9ce3-a6eadc5505cc-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-dn78l\" (UID: \"55a51982-5071-4b52-9ce3-a6eadc5505cc\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-dn78l" Oct 31 04:04:25.677961 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:25.677409 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-ntbm9" Oct 31 04:04:25.681693 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:25.681672 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/55a51982-5071-4b52-9ce3-a6eadc5505cc-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-dn78l\" (UID: \"55a51982-5071-4b52-9ce3-a6eadc5505cc\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-dn78l" Oct 31 04:04:25.685079 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:25.685058 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-5f76c7d5dc-bbl9h"] Oct 31 04:04:25.688083 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:25.688062 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f76c7d5dc-bbl9h" Oct 31 04:04:25.688804 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:25.688773 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76mjd\" (UniqueName: \"kubernetes.io/projected/55a51982-5071-4b52-9ce3-a6eadc5505cc-kube-api-access-76mjd\") pod \"prometheus-operator-9cd89b74c-dn78l\" (UID: \"55a51982-5071-4b52-9ce3-a6eadc5505cc\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-dn78l" Oct 31 04:04:25.708281 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:25.708257 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5f76c7d5dc-bbl9h"] Oct 31 04:04:25.709157 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:25.709127 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Oct 31 04:04:25.709272 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:25.709251 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Oct 31 04:04:25.709363 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:25.709302 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Oct 31 04:04:25.709363 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:25.709303 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-tsb5s\"" Oct 31 04:04:25.709456 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:25.709425 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Oct 31 04:04:25.709456 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:25.709433 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Oct 31 04:04:25.730760 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:25.730728 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-dn78l" Oct 31 04:04:25.864264 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:25.864228 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e1605166-1167-4857-bf3a-1ab0b7f7ea77-service-ca\") pod \"console-5f76c7d5dc-bbl9h\" (UID: \"e1605166-1167-4857-bf3a-1ab0b7f7ea77\") " pod="openshift-console/console-5f76c7d5dc-bbl9h" Oct 31 04:04:25.864416 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:25.864282 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e1605166-1167-4857-bf3a-1ab0b7f7ea77-console-oauth-config\") pod \"console-5f76c7d5dc-bbl9h\" (UID: \"e1605166-1167-4857-bf3a-1ab0b7f7ea77\") " pod="openshift-console/console-5f76c7d5dc-bbl9h" Oct 31 04:04:25.864416 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:25.864376 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9ldj\" (UniqueName: \"kubernetes.io/projected/e1605166-1167-4857-bf3a-1ab0b7f7ea77-kube-api-access-x9ldj\") pod \"console-5f76c7d5dc-bbl9h\" (UID: \"e1605166-1167-4857-bf3a-1ab0b7f7ea77\") " pod="openshift-console/console-5f76c7d5dc-bbl9h" Oct 31 04:04:25.864499 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:25.864420 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e1605166-1167-4857-bf3a-1ab0b7f7ea77-oauth-serving-cert\") pod \"console-5f76c7d5dc-bbl9h\" (UID: \"e1605166-1167-4857-bf3a-1ab0b7f7ea77\") " pod="openshift-console/console-5f76c7d5dc-bbl9h" Oct 31 04:04:25.864499 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:25.864456 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e1605166-1167-4857-bf3a-1ab0b7f7ea77-console-config\") pod \"console-5f76c7d5dc-bbl9h\" (UID: \"e1605166-1167-4857-bf3a-1ab0b7f7ea77\") " pod="openshift-console/console-5f76c7d5dc-bbl9h" Oct 31 04:04:25.864499 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:25.864475 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e1605166-1167-4857-bf3a-1ab0b7f7ea77-console-serving-cert\") pod \"console-5f76c7d5dc-bbl9h\" (UID: \"e1605166-1167-4857-bf3a-1ab0b7f7ea77\") " pod="openshift-console/console-5f76c7d5dc-bbl9h" Oct 31 04:04:25.864499 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:25.864494 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e1605166-1167-4857-bf3a-1ab0b7f7ea77-tmp\") pod \"console-5f76c7d5dc-bbl9h\" (UID: \"e1605166-1167-4857-bf3a-1ab0b7f7ea77\") " pod="openshift-console/console-5f76c7d5dc-bbl9h" Oct 31 04:04:25.869207 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:25.869174 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-799cd67bd8-fwz76" Oct 31 04:04:25.965342 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:25.965293 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e1605166-1167-4857-bf3a-1ab0b7f7ea77-service-ca\") pod \"console-5f76c7d5dc-bbl9h\" (UID: \"e1605166-1167-4857-bf3a-1ab0b7f7ea77\") " pod="openshift-console/console-5f76c7d5dc-bbl9h" Oct 31 04:04:25.965342 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:25.965344 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e1605166-1167-4857-bf3a-1ab0b7f7ea77-console-oauth-config\") pod \"console-5f76c7d5dc-bbl9h\" (UID: \"e1605166-1167-4857-bf3a-1ab0b7f7ea77\") " pod="openshift-console/console-5f76c7d5dc-bbl9h" Oct 31 04:04:25.965579 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:25.965379 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9ldj\" (UniqueName: \"kubernetes.io/projected/e1605166-1167-4857-bf3a-1ab0b7f7ea77-kube-api-access-x9ldj\") pod \"console-5f76c7d5dc-bbl9h\" (UID: \"e1605166-1167-4857-bf3a-1ab0b7f7ea77\") " pod="openshift-console/console-5f76c7d5dc-bbl9h" Oct 31 04:04:25.965579 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:25.965484 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e1605166-1167-4857-bf3a-1ab0b7f7ea77-oauth-serving-cert\") pod \"console-5f76c7d5dc-bbl9h\" (UID: \"e1605166-1167-4857-bf3a-1ab0b7f7ea77\") " pod="openshift-console/console-5f76c7d5dc-bbl9h" Oct 31 04:04:25.965579 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:25.965529 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e1605166-1167-4857-bf3a-1ab0b7f7ea77-console-config\") pod \"console-5f76c7d5dc-bbl9h\" (UID: \"e1605166-1167-4857-bf3a-1ab0b7f7ea77\") " pod="openshift-console/console-5f76c7d5dc-bbl9h" Oct 31 04:04:25.965579 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:25.965571 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e1605166-1167-4857-bf3a-1ab0b7f7ea77-console-serving-cert\") pod \"console-5f76c7d5dc-bbl9h\" (UID: \"e1605166-1167-4857-bf3a-1ab0b7f7ea77\") " pod="openshift-console/console-5f76c7d5dc-bbl9h" Oct 31 04:04:25.965786 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:25.965616 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e1605166-1167-4857-bf3a-1ab0b7f7ea77-tmp\") pod \"console-5f76c7d5dc-bbl9h\" (UID: \"e1605166-1167-4857-bf3a-1ab0b7f7ea77\") " pod="openshift-console/console-5f76c7d5dc-bbl9h" Oct 31 04:04:25.966382 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:25.966352 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e1605166-1167-4857-bf3a-1ab0b7f7ea77-console-config\") pod \"console-5f76c7d5dc-bbl9h\" (UID: \"e1605166-1167-4857-bf3a-1ab0b7f7ea77\") " pod="openshift-console/console-5f76c7d5dc-bbl9h" Oct 31 04:04:25.966498 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:25.966351 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e1605166-1167-4857-bf3a-1ab0b7f7ea77-oauth-serving-cert\") pod \"console-5f76c7d5dc-bbl9h\" (UID: \"e1605166-1167-4857-bf3a-1ab0b7f7ea77\") " pod="openshift-console/console-5f76c7d5dc-bbl9h" Oct 31 04:04:25.966498 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:25.966449 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e1605166-1167-4857-bf3a-1ab0b7f7ea77-tmp\") pod \"console-5f76c7d5dc-bbl9h\" (UID: \"e1605166-1167-4857-bf3a-1ab0b7f7ea77\") " pod="openshift-console/console-5f76c7d5dc-bbl9h" Oct 31 04:04:25.966857 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:25.966832 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e1605166-1167-4857-bf3a-1ab0b7f7ea77-service-ca\") pod \"console-5f76c7d5dc-bbl9h\" (UID: \"e1605166-1167-4857-bf3a-1ab0b7f7ea77\") " pod="openshift-console/console-5f76c7d5dc-bbl9h" Oct 31 04:04:25.968238 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:25.968214 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e1605166-1167-4857-bf3a-1ab0b7f7ea77-console-oauth-config\") pod \"console-5f76c7d5dc-bbl9h\" (UID: \"e1605166-1167-4857-bf3a-1ab0b7f7ea77\") " pod="openshift-console/console-5f76c7d5dc-bbl9h" Oct 31 04:04:25.968616 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:25.968591 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e1605166-1167-4857-bf3a-1ab0b7f7ea77-console-serving-cert\") pod \"console-5f76c7d5dc-bbl9h\" (UID: \"e1605166-1167-4857-bf3a-1ab0b7f7ea77\") " pod="openshift-console/console-5f76c7d5dc-bbl9h" Oct 31 04:04:25.989880 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:25.989858 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9ldj\" (UniqueName: \"kubernetes.io/projected/e1605166-1167-4857-bf3a-1ab0b7f7ea77-kube-api-access-x9ldj\") pod \"console-5f76c7d5dc-bbl9h\" (UID: \"e1605166-1167-4857-bf3a-1ab0b7f7ea77\") " pod="openshift-console/console-5f76c7d5dc-bbl9h" Oct 31 04:04:26.011670 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:26.011637 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f76c7d5dc-bbl9h" Oct 31 04:04:26.043032 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:26.042771 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-dn78l"] Oct 31 04:04:26.046239 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:26.046210 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55a51982_5071_4b52_9ce3_a6eadc5505cc.slice/crio-36c16db6e7a66f0130d157173d07052d1031727944d7b242f0e8fefa377a9f9a WatchSource:0}: Error finding container 36c16db6e7a66f0130d157173d07052d1031727944d7b242f0e8fefa377a9f9a: Status 404 returned error can't find the container with id 36c16db6e7a66f0130d157173d07052d1031727944d7b242f0e8fefa377a9f9a Oct 31 04:04:26.326777 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:26.326728 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5f76c7d5dc-bbl9h"] Oct 31 04:04:26.328599 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:26.328571 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1605166_1167_4857_bf3a_1ab0b7f7ea77.slice/crio-d14af442ec3b50c7bb220199e6fb210fd1749ebe04cb056661cb8f0d2b7048ae WatchSource:0}: Error finding container d14af442ec3b50c7bb220199e6fb210fd1749ebe04cb056661cb8f0d2b7048ae: Status 404 returned error can't find the container with id d14af442ec3b50c7bb220199e6fb210fd1749ebe04cb056661cb8f0d2b7048ae Oct 31 04:04:26.538723 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:26.538684 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f76c7d5dc-bbl9h" event={"ID":"e1605166-1167-4857-bf3a-1ab0b7f7ea77","Type":"ContainerStarted","Data":"d14af442ec3b50c7bb220199e6fb210fd1749ebe04cb056661cb8f0d2b7048ae"} Oct 31 04:04:26.540039 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:26.540014 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-dn78l" event={"ID":"55a51982-5071-4b52-9ce3-a6eadc5505cc","Type":"ContainerStarted","Data":"36c16db6e7a66f0130d157173d07052d1031727944d7b242f0e8fefa377a9f9a"} Oct 31 04:04:26.543143 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:26.542636 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-fdtl5" event={"ID":"7ae6197e-7c25-4b01-a0f2-d70621c8ad50","Type":"ContainerStarted","Data":"6656cb141a4cfb9c2326d00a26f8fe60dd2237785c70e7de605e73a3d038e751"} Oct 31 04:04:26.544288 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:26.544261 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-hkmvc" event={"ID":"b4d3b61b-4743-496c-9215-cd6766e296f1","Type":"ContainerStarted","Data":"93e9d44eb3de963e2c82a178d2f6f6ea575401138e3498a769131efe2ab2db34"} Oct 31 04:04:26.587249 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:26.587164 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-fdtl5" podStartSLOduration=1.707380794 podStartE2EDuration="4.587150951s" totalImagesPullingTime="2.879770157s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 04:04:22 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:04:26.586697257 +0000 UTC m=+14.301031910" watchObservedRunningTime="2026-10-31 04:04:26.587150951 +0000 UTC m=+14.301485562" Oct 31 04:04:26.628287 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:26.628231 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-hkmvc" podStartSLOduration=2.008942272 podStartE2EDuration="5.628212774s" totalImagesPullingTime="3.619270502s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:04:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:04:26.627610064 +0000 UTC m=+14.341944683" watchObservedRunningTime="2026-10-31 04:04:26.628212774 +0000 UTC m=+14.342547392" Oct 31 04:04:26.659461 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:26.659429 4376 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-rv7hd_3ae04333-9598-482c-ae38-8de983d7a48f/dns/0.log" Oct 31 04:04:26.796773 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:26.796742 4376 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-rv7hd_3ae04333-9598-482c-ae38-8de983d7a48f/kube-rbac-proxy/0.log" Oct 31 04:04:27.395728 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:27.395525 4376 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-77z7p_1f2543cc-b528-41be-92e0-9d654405a723/dns-node-resolver/0.log" Oct 31 04:04:28.202644 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:28.202569 4376 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-799cd67bd8-fwz76_3e67fa28-1a34-48bd-be16-373044df6f41/registry/0.log" Oct 31 04:04:28.554874 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:28.554797 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-dn78l" event={"ID":"55a51982-5071-4b52-9ce3-a6eadc5505cc","Type":"ContainerStarted","Data":"6e9334065e6825fe8ffd69ce8e0d8f01d64f7a36a74eb2617713e401e5d58a0f"} Oct 31 04:04:28.555183 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:28.555162 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-dn78l" event={"ID":"55a51982-5071-4b52-9ce3-a6eadc5505cc","Type":"ContainerStarted","Data":"eb7ce79ce5d093269853bba5f7b1469ab89215cef8794a6006ffa2f237f34ebd"} Oct 31 04:04:28.598067 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:28.598009 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-dn78l" podStartSLOduration=2.137853304 podStartE2EDuration="3.597995496s" totalImagesPullingTime="1.460142192s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:04:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:04:28.597250192 +0000 UTC m=+16.311584829" watchObservedRunningTime="2026-10-31 04:04:28.597995496 +0000 UTC m=+16.312330113" Oct 31 04:04:28.795947 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:28.795921 4376 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-xmbxc_a90157bb-46c0-4afa-82b1-9f6819518744/node-ca/0.log" Oct 31 04:04:29.396420 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:29.396388 4376 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-hkmvc_b4d3b61b-4743-496c-9215-cd6766e296f1/serve-healthcheck-canary/0.log" Oct 31 04:04:30.564394 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:30.564353 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f76c7d5dc-bbl9h" event={"ID":"e1605166-1167-4857-bf3a-1ab0b7f7ea77","Type":"ContainerStarted","Data":"9c7d87e7ab24866fa9cb2f658d086e6e4a5eead43dfdc919d0cb144ec0977c2b"} Oct 31 04:04:30.607606 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:30.607559 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-5f76c7d5dc-bbl9h" podStartSLOduration=2.400735 podStartE2EDuration="5.607545424s" totalImagesPullingTime="3.206810424s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:04:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:04:30.607419637 +0000 UTC m=+18.321754282" watchObservedRunningTime="2026-10-31 04:04:30.607545424 +0000 UTC m=+18.321880037" Oct 31 04:04:31.062871 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:31.062839 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-fg4ft"] Oct 31 04:04:31.066566 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:31.066547 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-fg4ft" Oct 31 04:04:31.128958 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:31.128924 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 31 04:04:31.129123 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:31.129101 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 31 04:04:31.129209 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:31.129101 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-mfl64\"" Oct 31 04:04:31.132784 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:31.132759 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 31 04:04:31.212610 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:31.212581 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7b35c4cc-25db-4bf8-8b1b-37b8a45b46cf-node-exporter-tls\") pod \"node-exporter-fg4ft\" (UID: \"7b35c4cc-25db-4bf8-8b1b-37b8a45b46cf\") " pod="openshift-monitoring/node-exporter-fg4ft" Oct 31 04:04:31.212763 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:31.212629 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7b35c4cc-25db-4bf8-8b1b-37b8a45b46cf-sys\") pod \"node-exporter-fg4ft\" (UID: \"7b35c4cc-25db-4bf8-8b1b-37b8a45b46cf\") " pod="openshift-monitoring/node-exporter-fg4ft" Oct 31 04:04:31.212763 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:31.212648 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnlfx\" (UniqueName: \"kubernetes.io/projected/7b35c4cc-25db-4bf8-8b1b-37b8a45b46cf-kube-api-access-pnlfx\") pod \"node-exporter-fg4ft\" (UID: \"7b35c4cc-25db-4bf8-8b1b-37b8a45b46cf\") " pod="openshift-monitoring/node-exporter-fg4ft" Oct 31 04:04:31.212763 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:31.212719 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7b35c4cc-25db-4bf8-8b1b-37b8a45b46cf-root\") pod \"node-exporter-fg4ft\" (UID: \"7b35c4cc-25db-4bf8-8b1b-37b8a45b46cf\") " pod="openshift-monitoring/node-exporter-fg4ft" Oct 31 04:04:31.212921 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:31.212889 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7b35c4cc-25db-4bf8-8b1b-37b8a45b46cf-node-exporter-wtmp\") pod \"node-exporter-fg4ft\" (UID: \"7b35c4cc-25db-4bf8-8b1b-37b8a45b46cf\") " pod="openshift-monitoring/node-exporter-fg4ft" Oct 31 04:04:31.212972 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:31.212934 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7b35c4cc-25db-4bf8-8b1b-37b8a45b46cf-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-fg4ft\" (UID: \"7b35c4cc-25db-4bf8-8b1b-37b8a45b46cf\") " pod="openshift-monitoring/node-exporter-fg4ft" Oct 31 04:04:31.213016 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:31.212973 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7b35c4cc-25db-4bf8-8b1b-37b8a45b46cf-node-exporter-accelerators-collector-config\") pod \"node-exporter-fg4ft\" (UID: \"7b35c4cc-25db-4bf8-8b1b-37b8a45b46cf\") " pod="openshift-monitoring/node-exporter-fg4ft" Oct 31 04:04:31.213083 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:31.213065 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7b35c4cc-25db-4bf8-8b1b-37b8a45b46cf-metrics-client-ca\") pod \"node-exporter-fg4ft\" (UID: \"7b35c4cc-25db-4bf8-8b1b-37b8a45b46cf\") " pod="openshift-monitoring/node-exporter-fg4ft" Oct 31 04:04:31.213114 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:31.213101 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7b35c4cc-25db-4bf8-8b1b-37b8a45b46cf-node-exporter-textfile\") pod \"node-exporter-fg4ft\" (UID: \"7b35c4cc-25db-4bf8-8b1b-37b8a45b46cf\") " pod="openshift-monitoring/node-exporter-fg4ft" Oct 31 04:04:31.313678 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:31.313607 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7b35c4cc-25db-4bf8-8b1b-37b8a45b46cf-node-exporter-wtmp\") pod \"node-exporter-fg4ft\" (UID: \"7b35c4cc-25db-4bf8-8b1b-37b8a45b46cf\") " pod="openshift-monitoring/node-exporter-fg4ft" Oct 31 04:04:31.313678 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:31.313658 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7b35c4cc-25db-4bf8-8b1b-37b8a45b46cf-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-fg4ft\" (UID: \"7b35c4cc-25db-4bf8-8b1b-37b8a45b46cf\") " pod="openshift-monitoring/node-exporter-fg4ft" Oct 31 04:04:31.313871 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:31.313692 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7b35c4cc-25db-4bf8-8b1b-37b8a45b46cf-node-exporter-accelerators-collector-config\") pod \"node-exporter-fg4ft\" (UID: \"7b35c4cc-25db-4bf8-8b1b-37b8a45b46cf\") " pod="openshift-monitoring/node-exporter-fg4ft" Oct 31 04:04:31.313871 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:31.313727 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7b35c4cc-25db-4bf8-8b1b-37b8a45b46cf-metrics-client-ca\") pod \"node-exporter-fg4ft\" (UID: \"7b35c4cc-25db-4bf8-8b1b-37b8a45b46cf\") " pod="openshift-monitoring/node-exporter-fg4ft" Oct 31 04:04:31.313871 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:31.313758 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7b35c4cc-25db-4bf8-8b1b-37b8a45b46cf-node-exporter-textfile\") pod \"node-exporter-fg4ft\" (UID: \"7b35c4cc-25db-4bf8-8b1b-37b8a45b46cf\") " pod="openshift-monitoring/node-exporter-fg4ft" Oct 31 04:04:31.313871 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:31.313796 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7b35c4cc-25db-4bf8-8b1b-37b8a45b46cf-node-exporter-tls\") pod \"node-exporter-fg4ft\" (UID: \"7b35c4cc-25db-4bf8-8b1b-37b8a45b46cf\") " pod="openshift-monitoring/node-exporter-fg4ft" Oct 31 04:04:31.313871 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:31.313814 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7b35c4cc-25db-4bf8-8b1b-37b8a45b46cf-node-exporter-wtmp\") pod \"node-exporter-fg4ft\" (UID: \"7b35c4cc-25db-4bf8-8b1b-37b8a45b46cf\") " pod="openshift-monitoring/node-exporter-fg4ft" Oct 31 04:04:31.313871 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:31.313826 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7b35c4cc-25db-4bf8-8b1b-37b8a45b46cf-sys\") pod \"node-exporter-fg4ft\" (UID: \"7b35c4cc-25db-4bf8-8b1b-37b8a45b46cf\") " pod="openshift-monitoring/node-exporter-fg4ft" Oct 31 04:04:31.313871 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:31.313864 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnlfx\" (UniqueName: \"kubernetes.io/projected/7b35c4cc-25db-4bf8-8b1b-37b8a45b46cf-kube-api-access-pnlfx\") pod \"node-exporter-fg4ft\" (UID: \"7b35c4cc-25db-4bf8-8b1b-37b8a45b46cf\") " pod="openshift-monitoring/node-exporter-fg4ft" Oct 31 04:04:31.314184 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:31.313879 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7b35c4cc-25db-4bf8-8b1b-37b8a45b46cf-root\") pod \"node-exporter-fg4ft\" (UID: \"7b35c4cc-25db-4bf8-8b1b-37b8a45b46cf\") " pod="openshift-monitoring/node-exporter-fg4ft" Oct 31 04:04:31.314184 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:31.313938 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7b35c4cc-25db-4bf8-8b1b-37b8a45b46cf-root\") pod \"node-exporter-fg4ft\" (UID: \"7b35c4cc-25db-4bf8-8b1b-37b8a45b46cf\") " pod="openshift-monitoring/node-exporter-fg4ft" Oct 31 04:04:31.314289 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:31.314192 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7b35c4cc-25db-4bf8-8b1b-37b8a45b46cf-sys\") pod \"node-exporter-fg4ft\" (UID: \"7b35c4cc-25db-4bf8-8b1b-37b8a45b46cf\") " pod="openshift-monitoring/node-exporter-fg4ft" Oct 31 04:04:31.314289 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:31.314247 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7b35c4cc-25db-4bf8-8b1b-37b8a45b46cf-node-exporter-textfile\") pod \"node-exporter-fg4ft\" (UID: \"7b35c4cc-25db-4bf8-8b1b-37b8a45b46cf\") " pod="openshift-monitoring/node-exporter-fg4ft" Oct 31 04:04:31.314412 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:31.314344 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7b35c4cc-25db-4bf8-8b1b-37b8a45b46cf-node-exporter-accelerators-collector-config\") pod \"node-exporter-fg4ft\" (UID: \"7b35c4cc-25db-4bf8-8b1b-37b8a45b46cf\") " pod="openshift-monitoring/node-exporter-fg4ft" Oct 31 04:04:31.314685 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:31.314666 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7b35c4cc-25db-4bf8-8b1b-37b8a45b46cf-metrics-client-ca\") pod \"node-exporter-fg4ft\" (UID: \"7b35c4cc-25db-4bf8-8b1b-37b8a45b46cf\") " pod="openshift-monitoring/node-exporter-fg4ft" Oct 31 04:04:31.316104 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:31.316084 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7b35c4cc-25db-4bf8-8b1b-37b8a45b46cf-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-fg4ft\" (UID: \"7b35c4cc-25db-4bf8-8b1b-37b8a45b46cf\") " pod="openshift-monitoring/node-exporter-fg4ft" Oct 31 04:04:31.316344 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:31.316306 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7b35c4cc-25db-4bf8-8b1b-37b8a45b46cf-node-exporter-tls\") pod \"node-exporter-fg4ft\" (UID: \"7b35c4cc-25db-4bf8-8b1b-37b8a45b46cf\") " pod="openshift-monitoring/node-exporter-fg4ft" Oct 31 04:04:31.340386 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:31.340361 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnlfx\" (UniqueName: \"kubernetes.io/projected/7b35c4cc-25db-4bf8-8b1b-37b8a45b46cf-kube-api-access-pnlfx\") pod \"node-exporter-fg4ft\" (UID: \"7b35c4cc-25db-4bf8-8b1b-37b8a45b46cf\") " pod="openshift-monitoring/node-exporter-fg4ft" Oct 31 04:04:31.384421 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:31.384389 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-fg4ft" Oct 31 04:04:31.397304 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:04:31.397275 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b35c4cc_25db_4bf8_8b1b_37b8a45b46cf.slice/crio-569873eed82dc42368c2581be0d749467e57875053f17cba5d87a5b9b5e742bf WatchSource:0}: Error finding container 569873eed82dc42368c2581be0d749467e57875053f17cba5d87a5b9b5e742bf: Status 404 returned error can't find the container with id 569873eed82dc42368c2581be0d749467e57875053f17cba5d87a5b9b5e742bf Oct 31 04:04:31.569345 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:31.569242 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fg4ft" event={"ID":"7b35c4cc-25db-4bf8-8b1b-37b8a45b46cf","Type":"ContainerStarted","Data":"569873eed82dc42368c2581be0d749467e57875053f17cba5d87a5b9b5e742bf"} Oct 31 04:04:32.577934 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:32.577890 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fg4ft" event={"ID":"7b35c4cc-25db-4bf8-8b1b-37b8a45b46cf","Type":"ContainerStarted","Data":"a9ad4d7e503c88816c9cd7c6fa0a3bd8ec93a405e0b02da75b5c04a09f38f2c9"} Oct 31 04:04:33.583317 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:33.583278 4376 generic.go:356] "Generic (PLEG): container finished" podID="7b35c4cc-25db-4bf8-8b1b-37b8a45b46cf" containerID="a9ad4d7e503c88816c9cd7c6fa0a3bd8ec93a405e0b02da75b5c04a09f38f2c9" exitCode=0 Oct 31 04:04:33.583743 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:33.583372 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fg4ft" event={"ID":"7b35c4cc-25db-4bf8-8b1b-37b8a45b46cf","Type":"ContainerDied","Data":"a9ad4d7e503c88816c9cd7c6fa0a3bd8ec93a405e0b02da75b5c04a09f38f2c9"} Oct 31 04:04:34.572426 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:34.572393 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zbh9t" Oct 31 04:04:34.589796 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:34.589760 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fg4ft" event={"ID":"7b35c4cc-25db-4bf8-8b1b-37b8a45b46cf","Type":"ContainerStarted","Data":"aae93a076d0faf62b61920e290a845b6d5bb96f7a377e9ab727ad8a921d7f768"} Oct 31 04:04:34.590205 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:34.589822 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fg4ft" event={"ID":"7b35c4cc-25db-4bf8-8b1b-37b8a45b46cf","Type":"ContainerStarted","Data":"eedd219026181f68b9fb25283cb2e884868f793f229634d5cca83b7e19d8eed9"} Oct 31 04:04:34.700202 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:34.700149 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-fg4ft" podStartSLOduration=2.581939111 podStartE2EDuration="3.700133183s" totalImagesPullingTime="1.028049442s" totalInitContainerRuntime="90.14463ms" isStatefulPod=true podCreationTimestamp="2026-10-31 04:04:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:04:34.699565421 +0000 UTC m=+22.413900042" watchObservedRunningTime="2026-10-31 04:04:34.700133183 +0000 UTC m=+22.414467802" Oct 31 04:04:36.012572 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:36.012531 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5f76c7d5dc-bbl9h" Oct 31 04:04:36.013124 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:36.012601 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5f76c7d5dc-bbl9h" Oct 31 04:04:36.018356 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:36.018306 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5f76c7d5dc-bbl9h" Oct 31 04:04:36.551996 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:36.551966 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-rv7hd" Oct 31 04:04:36.601191 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:36.601164 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5f76c7d5dc-bbl9h" Oct 31 04:04:40.625848 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:40.625805 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-74d6m" event={"ID":"43d6c532-a6b9-4f34-8d70-8bab1a8c3fd9","Type":"ContainerStarted","Data":"0182bb6aefb7dd43db758aa061b90fa2923004d99fbd8853204d69810c790991"} Oct 31 04:04:40.626317 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:40.626080 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-74d6m" Oct 31 04:04:40.647273 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:40.647187 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-74d6m" Oct 31 04:04:40.695345 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:40.695261 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-74d6m" podStartSLOduration=1.994696231 podStartE2EDuration="19.695248318s" totalImagesPullingTime="17.700552087s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:04:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:04:40.682095021 +0000 UTC m=+28.396429638" watchObservedRunningTime="2026-10-31 04:04:40.695248318 +0000 UTC m=+28.409582929" Oct 31 04:04:41.025403 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:04:41.021601 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 04:04:43.347757 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:43.347730 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-w9sdm" Oct 31 04:04:45.874436 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:45.874409 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-799cd67bd8-fwz76" Oct 31 04:04:46.346821 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:46.346729 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-557794d575-dkt97"] Oct 31 04:04:46.371619 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:46.371586 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-557794d575-dkt97" Oct 31 04:04:46.373757 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:46.373727 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-557794d575-dkt97"] Oct 31 04:04:46.415883 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:46.415843 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Oct 31 04:04:46.459721 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:46.459688 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/00a54582-1572-4f7e-a6ba-784bb56e026a-trusted-ca-bundle\") pod \"console-557794d575-dkt97\" (UID: \"00a54582-1572-4f7e-a6ba-784bb56e026a\") " pod="openshift-console/console-557794d575-dkt97" Oct 31 04:04:46.459898 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:46.459752 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/00a54582-1572-4f7e-a6ba-784bb56e026a-console-serving-cert\") pod \"console-557794d575-dkt97\" (UID: \"00a54582-1572-4f7e-a6ba-784bb56e026a\") " pod="openshift-console/console-557794d575-dkt97" Oct 31 04:04:46.459898 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:46.459856 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mf9p\" (UniqueName: \"kubernetes.io/projected/00a54582-1572-4f7e-a6ba-784bb56e026a-kube-api-access-8mf9p\") pod \"console-557794d575-dkt97\" (UID: \"00a54582-1572-4f7e-a6ba-784bb56e026a\") " pod="openshift-console/console-557794d575-dkt97" Oct 31 04:04:46.460028 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:46.459899 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/00a54582-1572-4f7e-a6ba-784bb56e026a-service-ca\") pod \"console-557794d575-dkt97\" (UID: \"00a54582-1572-4f7e-a6ba-784bb56e026a\") " pod="openshift-console/console-557794d575-dkt97" Oct 31 04:04:46.460028 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:46.460005 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/00a54582-1572-4f7e-a6ba-784bb56e026a-console-oauth-config\") pod \"console-557794d575-dkt97\" (UID: \"00a54582-1572-4f7e-a6ba-784bb56e026a\") " pod="openshift-console/console-557794d575-dkt97" Oct 31 04:04:46.460136 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:46.460034 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/00a54582-1572-4f7e-a6ba-784bb56e026a-console-config\") pod \"console-557794d575-dkt97\" (UID: \"00a54582-1572-4f7e-a6ba-784bb56e026a\") " pod="openshift-console/console-557794d575-dkt97" Oct 31 04:04:46.460136 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:46.460066 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/00a54582-1572-4f7e-a6ba-784bb56e026a-oauth-serving-cert\") pod \"console-557794d575-dkt97\" (UID: \"00a54582-1572-4f7e-a6ba-784bb56e026a\") " pod="openshift-console/console-557794d575-dkt97" Oct 31 04:04:46.460234 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:46.460196 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/00a54582-1572-4f7e-a6ba-784bb56e026a-tmp\") pod \"console-557794d575-dkt97\" (UID: \"00a54582-1572-4f7e-a6ba-784bb56e026a\") " pod="openshift-console/console-557794d575-dkt97" Oct 31 04:04:46.561033 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:46.560992 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/00a54582-1572-4f7e-a6ba-784bb56e026a-tmp\") pod \"console-557794d575-dkt97\" (UID: \"00a54582-1572-4f7e-a6ba-784bb56e026a\") " pod="openshift-console/console-557794d575-dkt97" Oct 31 04:04:46.561212 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:46.561040 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/00a54582-1572-4f7e-a6ba-784bb56e026a-trusted-ca-bundle\") pod \"console-557794d575-dkt97\" (UID: \"00a54582-1572-4f7e-a6ba-784bb56e026a\") " pod="openshift-console/console-557794d575-dkt97" Oct 31 04:04:46.561212 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:46.561092 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/00a54582-1572-4f7e-a6ba-784bb56e026a-console-serving-cert\") pod \"console-557794d575-dkt97\" (UID: \"00a54582-1572-4f7e-a6ba-784bb56e026a\") " pod="openshift-console/console-557794d575-dkt97" Oct 31 04:04:46.561212 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:46.561121 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mf9p\" (UniqueName: \"kubernetes.io/projected/00a54582-1572-4f7e-a6ba-784bb56e026a-kube-api-access-8mf9p\") pod \"console-557794d575-dkt97\" (UID: \"00a54582-1572-4f7e-a6ba-784bb56e026a\") " pod="openshift-console/console-557794d575-dkt97" Oct 31 04:04:46.561212 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:46.561147 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/00a54582-1572-4f7e-a6ba-784bb56e026a-service-ca\") pod \"console-557794d575-dkt97\" (UID: \"00a54582-1572-4f7e-a6ba-784bb56e026a\") " pod="openshift-console/console-557794d575-dkt97" Oct 31 04:04:46.561212 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:46.561181 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/00a54582-1572-4f7e-a6ba-784bb56e026a-console-oauth-config\") pod \"console-557794d575-dkt97\" (UID: \"00a54582-1572-4f7e-a6ba-784bb56e026a\") " pod="openshift-console/console-557794d575-dkt97" Oct 31 04:04:46.561212 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:46.561207 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/00a54582-1572-4f7e-a6ba-784bb56e026a-console-config\") pod \"console-557794d575-dkt97\" (UID: \"00a54582-1572-4f7e-a6ba-784bb56e026a\") " pod="openshift-console/console-557794d575-dkt97" Oct 31 04:04:46.561557 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:46.561234 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/00a54582-1572-4f7e-a6ba-784bb56e026a-oauth-serving-cert\") pod \"console-557794d575-dkt97\" (UID: \"00a54582-1572-4f7e-a6ba-784bb56e026a\") " pod="openshift-console/console-557794d575-dkt97" Oct 31 04:04:46.561557 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:46.561514 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/00a54582-1572-4f7e-a6ba-784bb56e026a-tmp\") pod \"console-557794d575-dkt97\" (UID: \"00a54582-1572-4f7e-a6ba-784bb56e026a\") " pod="openshift-console/console-557794d575-dkt97" Oct 31 04:04:46.562012 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:46.561985 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/00a54582-1572-4f7e-a6ba-784bb56e026a-oauth-serving-cert\") pod \"console-557794d575-dkt97\" (UID: \"00a54582-1572-4f7e-a6ba-784bb56e026a\") " pod="openshift-console/console-557794d575-dkt97" Oct 31 04:04:46.562224 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:46.562199 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/00a54582-1572-4f7e-a6ba-784bb56e026a-trusted-ca-bundle\") pod \"console-557794d575-dkt97\" (UID: \"00a54582-1572-4f7e-a6ba-784bb56e026a\") " pod="openshift-console/console-557794d575-dkt97" Oct 31 04:04:46.562351 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:46.562272 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/00a54582-1572-4f7e-a6ba-784bb56e026a-service-ca\") pod \"console-557794d575-dkt97\" (UID: \"00a54582-1572-4f7e-a6ba-784bb56e026a\") " pod="openshift-console/console-557794d575-dkt97" Oct 31 04:04:46.562717 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:46.562697 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/00a54582-1572-4f7e-a6ba-784bb56e026a-console-config\") pod \"console-557794d575-dkt97\" (UID: \"00a54582-1572-4f7e-a6ba-784bb56e026a\") " pod="openshift-console/console-557794d575-dkt97" Oct 31 04:04:46.564106 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:46.564085 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/00a54582-1572-4f7e-a6ba-784bb56e026a-console-serving-cert\") pod \"console-557794d575-dkt97\" (UID: \"00a54582-1572-4f7e-a6ba-784bb56e026a\") " pod="openshift-console/console-557794d575-dkt97" Oct 31 04:04:46.564753 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:46.564730 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/00a54582-1572-4f7e-a6ba-784bb56e026a-console-oauth-config\") pod \"console-557794d575-dkt97\" (UID: \"00a54582-1572-4f7e-a6ba-784bb56e026a\") " pod="openshift-console/console-557794d575-dkt97" Oct 31 04:04:46.585339 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:46.585306 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mf9p\" (UniqueName: \"kubernetes.io/projected/00a54582-1572-4f7e-a6ba-784bb56e026a-kube-api-access-8mf9p\") pod \"console-557794d575-dkt97\" (UID: \"00a54582-1572-4f7e-a6ba-784bb56e026a\") " pod="openshift-console/console-557794d575-dkt97" Oct 31 04:04:46.706311 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:46.706279 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-557794d575-dkt97" Oct 31 04:04:47.054468 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:47.054299 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-557794d575-dkt97"] Oct 31 04:04:47.655025 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:47.654984 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-557794d575-dkt97" event={"ID":"00a54582-1572-4f7e-a6ba-784bb56e026a","Type":"ContainerStarted","Data":"399a966be242c408afdb74d7fb4c4eeaa828dcf140dc1c42d2a6a61ae871eccb"} Oct 31 04:04:47.655214 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:47.655051 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-557794d575-dkt97" event={"ID":"00a54582-1572-4f7e-a6ba-784bb56e026a","Type":"ContainerStarted","Data":"1299f8c4b8a7306144b4d0092bf1809ae16b7a3a5fd595fa4b7384f595b0b5d3"} Oct 31 04:04:47.700559 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:47.700501 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-557794d575-dkt97" podStartSLOduration=1.7004851840000001 podStartE2EDuration="1.700485184s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:04:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:04:47.699538329 +0000 UTC m=+35.413872965" watchObservedRunningTime="2026-10-31 04:04:47.700485184 +0000 UTC m=+35.414819800" Oct 31 04:04:56.706921 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:56.706840 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-557794d575-dkt97" Oct 31 04:04:56.706921 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:56.706904 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-557794d575-dkt97" Oct 31 04:04:56.711635 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:56.711616 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-557794d575-dkt97" Oct 31 04:04:57.689132 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:57.689106 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-557794d575-dkt97" Oct 31 04:04:57.762694 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:04:57.762667 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5f76c7d5dc-bbl9h"] Oct 31 04:05:10.997867 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:05:10.997828 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 04:05:22.782405 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:05:22.782351 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-5f76c7d5dc-bbl9h" podUID="e1605166-1167-4857-bf3a-1ab0b7f7ea77" containerName="console" containerID="cri-o://9c7d87e7ab24866fa9cb2f658d086e6e4a5eead43dfdc919d0cb144ec0977c2b" gracePeriod=15 Oct 31 04:05:23.051444 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:05:23.051382 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f76c7d5dc-bbl9h" Oct 31 04:05:23.215320 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:05:23.215291 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1605166-1167-4857-bf3a-1ab0b7f7ea77-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e1605166-1167-4857-bf3a-1ab0b7f7ea77-oauth-serving-cert\") pod \"e1605166-1167-4857-bf3a-1ab0b7f7ea77\" (UID: \"e1605166-1167-4857-bf3a-1ab0b7f7ea77\") " Oct 31 04:05:23.215459 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:05:23.215347 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e1605166-1167-4857-bf3a-1ab0b7f7ea77-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e1605166-1167-4857-bf3a-1ab0b7f7ea77-console-serving-cert\") pod \"e1605166-1167-4857-bf3a-1ab0b7f7ea77\" (UID: \"e1605166-1167-4857-bf3a-1ab0b7f7ea77\") " Oct 31 04:05:23.215459 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:05:23.215376 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1605166-1167-4857-bf3a-1ab0b7f7ea77-tmp\" (UniqueName: \"kubernetes.io/empty-dir/e1605166-1167-4857-bf3a-1ab0b7f7ea77-tmp\") pod \"e1605166-1167-4857-bf3a-1ab0b7f7ea77\" (UID: \"e1605166-1167-4857-bf3a-1ab0b7f7ea77\") " Oct 31 04:05:23.215459 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:05:23.215410 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e1605166-1167-4857-bf3a-1ab0b7f7ea77-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e1605166-1167-4857-bf3a-1ab0b7f7ea77-console-oauth-config\") pod \"e1605166-1167-4857-bf3a-1ab0b7f7ea77\" (UID: \"e1605166-1167-4857-bf3a-1ab0b7f7ea77\") " Oct 31 04:05:23.215459 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:05:23.215434 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1605166-1167-4857-bf3a-1ab0b7f7ea77-kube-api-access-x9ldj\" (UniqueName: \"kubernetes.io/projected/e1605166-1167-4857-bf3a-1ab0b7f7ea77-kube-api-access-x9ldj\") pod \"e1605166-1167-4857-bf3a-1ab0b7f7ea77\" (UID: \"e1605166-1167-4857-bf3a-1ab0b7f7ea77\") " Oct 31 04:05:23.215661 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:05:23.215536 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1605166-1167-4857-bf3a-1ab0b7f7ea77-console-config\" (UniqueName: \"kubernetes.io/configmap/e1605166-1167-4857-bf3a-1ab0b7f7ea77-console-config\") pod \"e1605166-1167-4857-bf3a-1ab0b7f7ea77\" (UID: \"e1605166-1167-4857-bf3a-1ab0b7f7ea77\") " Oct 31 04:05:23.215661 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:05:23.215588 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1605166-1167-4857-bf3a-1ab0b7f7ea77-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1605166-1167-4857-bf3a-1ab0b7f7ea77-service-ca\") pod \"e1605166-1167-4857-bf3a-1ab0b7f7ea77\" (UID: \"e1605166-1167-4857-bf3a-1ab0b7f7ea77\") " Oct 31 04:05:23.215747 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:05:23.215700 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1605166-1167-4857-bf3a-1ab0b7f7ea77-oauth-serving-cert" pod "e1605166-1167-4857-bf3a-1ab0b7f7ea77" (UID: "e1605166-1167-4857-bf3a-1ab0b7f7ea77"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:05:23.215798 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:05:23.215727 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1605166-1167-4857-bf3a-1ab0b7f7ea77-tmp" pod "e1605166-1167-4857-bf3a-1ab0b7f7ea77" (UID: "e1605166-1167-4857-bf3a-1ab0b7f7ea77"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:05:23.215865 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:05:23.215848 4376 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e1605166-1167-4857-bf3a-1ab0b7f7ea77-oauth-serving-cert\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:05:23.215914 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:05:23.215872 4376 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e1605166-1167-4857-bf3a-1ab0b7f7ea77-tmp\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:05:23.215914 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:05:23.215896 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1605166-1167-4857-bf3a-1ab0b7f7ea77-console-config" pod "e1605166-1167-4857-bf3a-1ab0b7f7ea77" (UID: "e1605166-1167-4857-bf3a-1ab0b7f7ea77"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:05:23.216085 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:05:23.216064 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1605166-1167-4857-bf3a-1ab0b7f7ea77-service-ca" pod "e1605166-1167-4857-bf3a-1ab0b7f7ea77" (UID: "e1605166-1167-4857-bf3a-1ab0b7f7ea77"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:05:23.217664 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:05:23.217635 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1605166-1167-4857-bf3a-1ab0b7f7ea77-console-oauth-config" pod "e1605166-1167-4857-bf3a-1ab0b7f7ea77" (UID: "e1605166-1167-4857-bf3a-1ab0b7f7ea77"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:05:23.217763 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:05:23.217689 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1605166-1167-4857-bf3a-1ab0b7f7ea77-kube-api-access-x9ldj" pod "e1605166-1167-4857-bf3a-1ab0b7f7ea77" (UID: "e1605166-1167-4857-bf3a-1ab0b7f7ea77"). InnerVolumeSpecName "kube-api-access-x9ldj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:05:23.217763 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:05:23.217699 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1605166-1167-4857-bf3a-1ab0b7f7ea77-console-serving-cert" pod "e1605166-1167-4857-bf3a-1ab0b7f7ea77" (UID: "e1605166-1167-4857-bf3a-1ab0b7f7ea77"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:05:23.246336 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:05:23.246314 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-7b87c4b57-nxzpf"] Oct 31 04:05:23.246555 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:05:23.246543 4376 cpu_manager.go:414] "containerMap: removing container" podUID="e1605166-1167-4857-bf3a-1ab0b7f7ea77" containerName="console" Oct 31 04:05:23.246594 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:05:23.246556 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1605166-1167-4857-bf3a-1ab0b7f7ea77" containerName="console" Oct 31 04:05:23.246628 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:05:23.246617 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1605166-1167-4857-bf3a-1ab0b7f7ea77" containerName="console" Oct 31 04:05:23.265235 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:05:23.265217 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b87c4b57-nxzpf" Oct 31 04:05:23.271675 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:05:23.271180 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7b87c4b57-nxzpf"] Oct 31 04:05:23.317160 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:05:23.317111 4376 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e1605166-1167-4857-bf3a-1ab0b7f7ea77-console-serving-cert\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:05:23.317160 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:05:23.317130 4376 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e1605166-1167-4857-bf3a-1ab0b7f7ea77-console-oauth-config\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:05:23.317160 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:05:23.317140 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9ldj\" (UniqueName: \"kubernetes.io/projected/e1605166-1167-4857-bf3a-1ab0b7f7ea77-kube-api-access-x9ldj\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:05:23.317160 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:05:23.317149 4376 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e1605166-1167-4857-bf3a-1ab0b7f7ea77-console-config\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:05:23.317160 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:05:23.317157 4376 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e1605166-1167-4857-bf3a-1ab0b7f7ea77-service-ca\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:05:23.417748 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:05:23.417725 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/92740329-2888-4011-b817-1c0d74fd1c21-console-serving-cert\") pod \"console-7b87c4b57-nxzpf\" (UID: \"92740329-2888-4011-b817-1c0d74fd1c21\") " pod="openshift-console/console-7b87c4b57-nxzpf" Oct 31 04:05:23.417850 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:05:23.417768 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/92740329-2888-4011-b817-1c0d74fd1c21-trusted-ca-bundle\") pod \"console-7b87c4b57-nxzpf\" (UID: \"92740329-2888-4011-b817-1c0d74fd1c21\") " pod="openshift-console/console-7b87c4b57-nxzpf" Oct 31 04:05:23.417850 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:05:23.417792 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/92740329-2888-4011-b817-1c0d74fd1c21-tmp\") pod \"console-7b87c4b57-nxzpf\" (UID: \"92740329-2888-4011-b817-1c0d74fd1c21\") " pod="openshift-console/console-7b87c4b57-nxzpf" Oct 31 04:05:23.417935 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:05:23.417872 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/92740329-2888-4011-b817-1c0d74fd1c21-console-oauth-config\") pod \"console-7b87c4b57-nxzpf\" (UID: \"92740329-2888-4011-b817-1c0d74fd1c21\") " pod="openshift-console/console-7b87c4b57-nxzpf" Oct 31 04:05:23.417935 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:05:23.417914 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rv645\" (UniqueName: \"kubernetes.io/projected/92740329-2888-4011-b817-1c0d74fd1c21-kube-api-access-rv645\") pod \"console-7b87c4b57-nxzpf\" (UID: \"92740329-2888-4011-b817-1c0d74fd1c21\") " pod="openshift-console/console-7b87c4b57-nxzpf" Oct 31 04:05:23.417935 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:05:23.417931 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/92740329-2888-4011-b817-1c0d74fd1c21-console-config\") pod \"console-7b87c4b57-nxzpf\" (UID: \"92740329-2888-4011-b817-1c0d74fd1c21\") " pod="openshift-console/console-7b87c4b57-nxzpf" Oct 31 04:05:23.418078 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:05:23.417950 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/92740329-2888-4011-b817-1c0d74fd1c21-service-ca\") pod \"console-7b87c4b57-nxzpf\" (UID: \"92740329-2888-4011-b817-1c0d74fd1c21\") " pod="openshift-console/console-7b87c4b57-nxzpf" Oct 31 04:05:23.418078 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:05:23.417979 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/92740329-2888-4011-b817-1c0d74fd1c21-oauth-serving-cert\") pod \"console-7b87c4b57-nxzpf\" (UID: \"92740329-2888-4011-b817-1c0d74fd1c21\") " pod="openshift-console/console-7b87c4b57-nxzpf" Oct 31 04:05:23.518710 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:05:23.518687 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/92740329-2888-4011-b817-1c0d74fd1c21-trusted-ca-bundle\") pod \"console-7b87c4b57-nxzpf\" (UID: \"92740329-2888-4011-b817-1c0d74fd1c21\") " pod="openshift-console/console-7b87c4b57-nxzpf" Oct 31 04:05:23.518817 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:05:23.518718 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/92740329-2888-4011-b817-1c0d74fd1c21-tmp\") pod \"console-7b87c4b57-nxzpf\" (UID: \"92740329-2888-4011-b817-1c0d74fd1c21\") " pod="openshift-console/console-7b87c4b57-nxzpf" Oct 31 04:05:23.518817 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:05:23.518736 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/92740329-2888-4011-b817-1c0d74fd1c21-console-oauth-config\") pod \"console-7b87c4b57-nxzpf\" (UID: \"92740329-2888-4011-b817-1c0d74fd1c21\") " pod="openshift-console/console-7b87c4b57-nxzpf" Oct 31 04:05:23.518817 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:05:23.518752 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rv645\" (UniqueName: \"kubernetes.io/projected/92740329-2888-4011-b817-1c0d74fd1c21-kube-api-access-rv645\") pod \"console-7b87c4b57-nxzpf\" (UID: \"92740329-2888-4011-b817-1c0d74fd1c21\") " pod="openshift-console/console-7b87c4b57-nxzpf" Oct 31 04:05:23.518817 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:05:23.518773 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/92740329-2888-4011-b817-1c0d74fd1c21-console-config\") pod \"console-7b87c4b57-nxzpf\" (UID: \"92740329-2888-4011-b817-1c0d74fd1c21\") " pod="openshift-console/console-7b87c4b57-nxzpf" Oct 31 04:05:23.518817 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:05:23.518801 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/92740329-2888-4011-b817-1c0d74fd1c21-service-ca\") pod \"console-7b87c4b57-nxzpf\" (UID: \"92740329-2888-4011-b817-1c0d74fd1c21\") " pod="openshift-console/console-7b87c4b57-nxzpf" Oct 31 04:05:23.519063 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:05:23.518834 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/92740329-2888-4011-b817-1c0d74fd1c21-oauth-serving-cert\") pod \"console-7b87c4b57-nxzpf\" (UID: \"92740329-2888-4011-b817-1c0d74fd1c21\") " pod="openshift-console/console-7b87c4b57-nxzpf" Oct 31 04:05:23.519063 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:05:23.518872 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/92740329-2888-4011-b817-1c0d74fd1c21-console-serving-cert\") pod \"console-7b87c4b57-nxzpf\" (UID: \"92740329-2888-4011-b817-1c0d74fd1c21\") " pod="openshift-console/console-7b87c4b57-nxzpf" Oct 31 04:05:23.523638 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:05:23.523618 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/92740329-2888-4011-b817-1c0d74fd1c21-tmp\") pod \"console-7b87c4b57-nxzpf\" (UID: \"92740329-2888-4011-b817-1c0d74fd1c21\") " pod="openshift-console/console-7b87c4b57-nxzpf" Oct 31 04:05:23.525107 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:05:23.525087 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/92740329-2888-4011-b817-1c0d74fd1c21-console-oauth-config\") pod \"console-7b87c4b57-nxzpf\" (UID: \"92740329-2888-4011-b817-1c0d74fd1c21\") " pod="openshift-console/console-7b87c4b57-nxzpf" Oct 31 04:05:23.525162 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:05:23.525106 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/92740329-2888-4011-b817-1c0d74fd1c21-trusted-ca-bundle\") pod \"console-7b87c4b57-nxzpf\" (UID: \"92740329-2888-4011-b817-1c0d74fd1c21\") " pod="openshift-console/console-7b87c4b57-nxzpf" Oct 31 04:05:23.525162 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:05:23.525142 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/92740329-2888-4011-b817-1c0d74fd1c21-service-ca\") pod \"console-7b87c4b57-nxzpf\" (UID: \"92740329-2888-4011-b817-1c0d74fd1c21\") " pod="openshift-console/console-7b87c4b57-nxzpf" Oct 31 04:05:23.529613 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:05:23.529591 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/92740329-2888-4011-b817-1c0d74fd1c21-console-config\") pod \"console-7b87c4b57-nxzpf\" (UID: \"92740329-2888-4011-b817-1c0d74fd1c21\") " pod="openshift-console/console-7b87c4b57-nxzpf" Oct 31 04:05:23.529680 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:05:23.529612 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/92740329-2888-4011-b817-1c0d74fd1c21-console-serving-cert\") pod \"console-7b87c4b57-nxzpf\" (UID: \"92740329-2888-4011-b817-1c0d74fd1c21\") " pod="openshift-console/console-7b87c4b57-nxzpf" Oct 31 04:05:23.529680 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:05:23.529630 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/92740329-2888-4011-b817-1c0d74fd1c21-oauth-serving-cert\") pod \"console-7b87c4b57-nxzpf\" (UID: \"92740329-2888-4011-b817-1c0d74fd1c21\") " pod="openshift-console/console-7b87c4b57-nxzpf" Oct 31 04:05:23.546313 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:05:23.546289 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rv645\" (UniqueName: \"kubernetes.io/projected/92740329-2888-4011-b817-1c0d74fd1c21-kube-api-access-rv645\") pod \"console-7b87c4b57-nxzpf\" (UID: \"92740329-2888-4011-b817-1c0d74fd1c21\") " pod="openshift-console/console-7b87c4b57-nxzpf" Oct 31 04:05:23.595470 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:05:23.595427 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b87c4b57-nxzpf" Oct 31 04:05:23.769203 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:05:23.769171 4376 generic.go:356] "Generic (PLEG): container finished" podID="e1605166-1167-4857-bf3a-1ab0b7f7ea77" containerID="9c7d87e7ab24866fa9cb2f658d086e6e4a5eead43dfdc919d0cb144ec0977c2b" exitCode=0 Oct 31 04:05:23.769368 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:05:23.769215 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f76c7d5dc-bbl9h" event={"ID":"e1605166-1167-4857-bf3a-1ab0b7f7ea77","Type":"ContainerDied","Data":"9c7d87e7ab24866fa9cb2f658d086e6e4a5eead43dfdc919d0cb144ec0977c2b"} Oct 31 04:05:23.769368 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:05:23.769241 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f76c7d5dc-bbl9h" Oct 31 04:05:23.769368 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:05:23.769252 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f76c7d5dc-bbl9h" event={"ID":"e1605166-1167-4857-bf3a-1ab0b7f7ea77","Type":"ContainerDied","Data":"d14af442ec3b50c7bb220199e6fb210fd1749ebe04cb056661cb8f0d2b7048ae"} Oct 31 04:05:23.769368 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:05:23.769271 4376 scope.go:122] "RemoveContainer" containerID="9c7d87e7ab24866fa9cb2f658d086e6e4a5eead43dfdc919d0cb144ec0977c2b" Oct 31 04:05:23.776639 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:05:23.776525 4376 scope.go:122] "RemoveContainer" containerID="9c7d87e7ab24866fa9cb2f658d086e6e4a5eead43dfdc919d0cb144ec0977c2b" Oct 31 04:05:23.776854 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:05:23.776817 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c7d87e7ab24866fa9cb2f658d086e6e4a5eead43dfdc919d0cb144ec0977c2b\": container with ID starting with 9c7d87e7ab24866fa9cb2f658d086e6e4a5eead43dfdc919d0cb144ec0977c2b not found: ID does not exist" containerID="9c7d87e7ab24866fa9cb2f658d086e6e4a5eead43dfdc919d0cb144ec0977c2b" Oct 31 04:05:23.776958 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:05:23.776861 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c7d87e7ab24866fa9cb2f658d086e6e4a5eead43dfdc919d0cb144ec0977c2b"} err="failed to get container status \"9c7d87e7ab24866fa9cb2f658d086e6e4a5eead43dfdc919d0cb144ec0977c2b\": rpc error: code = NotFound desc = could not find container \"9c7d87e7ab24866fa9cb2f658d086e6e4a5eead43dfdc919d0cb144ec0977c2b\": container with ID starting with 9c7d87e7ab24866fa9cb2f658d086e6e4a5eead43dfdc919d0cb144ec0977c2b not found: ID does not exist" Oct 31 04:05:23.822081 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:05:23.822060 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5f76c7d5dc-bbl9h"] Oct 31 04:05:23.834321 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:05:23.834303 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5f76c7d5dc-bbl9h"] Oct 31 04:05:23.903084 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:05:23.903065 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7b87c4b57-nxzpf"] Oct 31 04:05:23.903740 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:05:23.903721 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92740329_2888_4011_b817_1c0d74fd1c21.slice/crio-e49ae44583d15d08770f31e923b28ca0c860533dcdd5245d5f5446591a7c1cd1 WatchSource:0}: Error finding container e49ae44583d15d08770f31e923b28ca0c860533dcdd5245d5f5446591a7c1cd1: Status 404 returned error can't find the container with id e49ae44583d15d08770f31e923b28ca0c860533dcdd5245d5f5446591a7c1cd1 Oct 31 04:05:24.440175 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:05:24.440133 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1605166-1167-4857-bf3a-1ab0b7f7ea77" path="/var/lib/kubelet/pods/e1605166-1167-4857-bf3a-1ab0b7f7ea77/volumes" Oct 31 04:05:24.773692 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:05:24.773594 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b87c4b57-nxzpf" event={"ID":"92740329-2888-4011-b817-1c0d74fd1c21","Type":"ContainerStarted","Data":"29883f25a0981aa212d36e8d3c9c82f67f1d627a7b99d2c2b82031243dc5f172"} Oct 31 04:05:24.773692 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:05:24.773653 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b87c4b57-nxzpf" event={"ID":"92740329-2888-4011-b817-1c0d74fd1c21","Type":"ContainerStarted","Data":"e49ae44583d15d08770f31e923b28ca0c860533dcdd5245d5f5446591a7c1cd1"} Oct 31 04:05:24.817619 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:05:24.817573 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-7b87c4b57-nxzpf" podStartSLOduration=1.8175623509999999 podStartE2EDuration="1.817562351s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:05:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:05:24.817298022 +0000 UTC m=+72.531632659" watchObservedRunningTime="2026-10-31 04:05:24.817562351 +0000 UTC m=+72.531896967" Oct 31 04:05:33.596196 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:05:33.596167 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7b87c4b57-nxzpf" Oct 31 04:05:33.596666 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:05:33.596214 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7b87c4b57-nxzpf" Oct 31 04:05:33.600474 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:05:33.600452 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7b87c4b57-nxzpf" Oct 31 04:05:33.806440 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:05:33.806416 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7b87c4b57-nxzpf" Oct 31 04:05:33.879574 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:05:33.879549 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-557794d575-dkt97"] Oct 31 04:05:40.997675 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:05:40.997624 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 04:05:58.902391 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:05:58.902353 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-557794d575-dkt97" podUID="00a54582-1572-4f7e-a6ba-784bb56e026a" containerName="console" containerID="cri-o://399a966be242c408afdb74d7fb4c4eeaa828dcf140dc1c42d2a6a61ae871eccb" gracePeriod=15 Oct 31 04:05:59.134152 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:05:59.134130 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-557794d575-dkt97" Oct 31 04:05:59.168402 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:05:59.168340 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/00a54582-1572-4f7e-a6ba-784bb56e026a-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/00a54582-1572-4f7e-a6ba-784bb56e026a-console-oauth-config\") pod \"00a54582-1572-4f7e-a6ba-784bb56e026a\" (UID: \"00a54582-1572-4f7e-a6ba-784bb56e026a\") " Oct 31 04:05:59.168402 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:05:59.168381 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00a54582-1572-4f7e-a6ba-784bb56e026a-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/00a54582-1572-4f7e-a6ba-784bb56e026a-oauth-serving-cert\") pod \"00a54582-1572-4f7e-a6ba-784bb56e026a\" (UID: \"00a54582-1572-4f7e-a6ba-784bb56e026a\") " Oct 31 04:05:59.168576 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:05:59.168418 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00a54582-1572-4f7e-a6ba-784bb56e026a-service-ca\" (UniqueName: \"kubernetes.io/configmap/00a54582-1572-4f7e-a6ba-784bb56e026a-service-ca\") pod \"00a54582-1572-4f7e-a6ba-784bb56e026a\" (UID: \"00a54582-1572-4f7e-a6ba-784bb56e026a\") " Oct 31 04:05:59.168576 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:05:59.168442 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/00a54582-1572-4f7e-a6ba-784bb56e026a-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/00a54582-1572-4f7e-a6ba-784bb56e026a-console-serving-cert\") pod \"00a54582-1572-4f7e-a6ba-784bb56e026a\" (UID: \"00a54582-1572-4f7e-a6ba-784bb56e026a\") " Oct 31 04:05:59.168576 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:05:59.168467 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00a54582-1572-4f7e-a6ba-784bb56e026a-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/00a54582-1572-4f7e-a6ba-784bb56e026a-trusted-ca-bundle\") pod \"00a54582-1572-4f7e-a6ba-784bb56e026a\" (UID: \"00a54582-1572-4f7e-a6ba-784bb56e026a\") " Oct 31 04:05:59.168576 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:05:59.168491 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00a54582-1572-4f7e-a6ba-784bb56e026a-kube-api-access-8mf9p\" (UniqueName: \"kubernetes.io/projected/00a54582-1572-4f7e-a6ba-784bb56e026a-kube-api-access-8mf9p\") pod \"00a54582-1572-4f7e-a6ba-784bb56e026a\" (UID: \"00a54582-1572-4f7e-a6ba-784bb56e026a\") " Oct 31 04:05:59.168576 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:05:59.168522 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00a54582-1572-4f7e-a6ba-784bb56e026a-tmp\" (UniqueName: \"kubernetes.io/empty-dir/00a54582-1572-4f7e-a6ba-784bb56e026a-tmp\") pod \"00a54582-1572-4f7e-a6ba-784bb56e026a\" (UID: \"00a54582-1572-4f7e-a6ba-784bb56e026a\") " Oct 31 04:05:59.168576 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:05:59.168554 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00a54582-1572-4f7e-a6ba-784bb56e026a-console-config\" (UniqueName: \"kubernetes.io/configmap/00a54582-1572-4f7e-a6ba-784bb56e026a-console-config\") pod \"00a54582-1572-4f7e-a6ba-784bb56e026a\" (UID: \"00a54582-1572-4f7e-a6ba-784bb56e026a\") " Oct 31 04:05:59.168857 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:05:59.168820 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00a54582-1572-4f7e-a6ba-784bb56e026a-oauth-serving-cert" pod "00a54582-1572-4f7e-a6ba-784bb56e026a" (UID: "00a54582-1572-4f7e-a6ba-784bb56e026a"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:05:59.168994 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:05:59.168963 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00a54582-1572-4f7e-a6ba-784bb56e026a-tmp" pod "00a54582-1572-4f7e-a6ba-784bb56e026a" (UID: "00a54582-1572-4f7e-a6ba-784bb56e026a"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:05:59.169093 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:05:59.168995 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00a54582-1572-4f7e-a6ba-784bb56e026a-service-ca" pod "00a54582-1572-4f7e-a6ba-784bb56e026a" (UID: "00a54582-1572-4f7e-a6ba-784bb56e026a"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:05:59.169093 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:05:59.169068 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00a54582-1572-4f7e-a6ba-784bb56e026a-console-config" pod "00a54582-1572-4f7e-a6ba-784bb56e026a" (UID: "00a54582-1572-4f7e-a6ba-784bb56e026a"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:05:59.169230 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:05:59.169200 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00a54582-1572-4f7e-a6ba-784bb56e026a-trusted-ca-bundle" pod "00a54582-1572-4f7e-a6ba-784bb56e026a" (UID: "00a54582-1572-4f7e-a6ba-784bb56e026a"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:05:59.170490 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:05:59.170460 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00a54582-1572-4f7e-a6ba-784bb56e026a-console-oauth-config" pod "00a54582-1572-4f7e-a6ba-784bb56e026a" (UID: "00a54582-1572-4f7e-a6ba-784bb56e026a"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:05:59.170619 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:05:59.170596 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00a54582-1572-4f7e-a6ba-784bb56e026a-console-serving-cert" pod "00a54582-1572-4f7e-a6ba-784bb56e026a" (UID: "00a54582-1572-4f7e-a6ba-784bb56e026a"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:05:59.170834 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:05:59.170817 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00a54582-1572-4f7e-a6ba-784bb56e026a-kube-api-access-8mf9p" pod "00a54582-1572-4f7e-a6ba-784bb56e026a" (UID: "00a54582-1572-4f7e-a6ba-784bb56e026a"). InnerVolumeSpecName "kube-api-access-8mf9p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:05:59.269617 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:05:59.269577 4376 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/00a54582-1572-4f7e-a6ba-784bb56e026a-console-config\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:05:59.269617 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:05:59.269610 4376 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/00a54582-1572-4f7e-a6ba-784bb56e026a-console-oauth-config\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:05:59.269617 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:05:59.269621 4376 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/00a54582-1572-4f7e-a6ba-784bb56e026a-oauth-serving-cert\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:05:59.269855 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:05:59.269629 4376 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/00a54582-1572-4f7e-a6ba-784bb56e026a-service-ca\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:05:59.269855 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:05:59.269638 4376 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/00a54582-1572-4f7e-a6ba-784bb56e026a-console-serving-cert\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:05:59.269855 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:05:59.269646 4376 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/00a54582-1572-4f7e-a6ba-784bb56e026a-trusted-ca-bundle\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:05:59.269855 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:05:59.269655 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mf9p\" (UniqueName: \"kubernetes.io/projected/00a54582-1572-4f7e-a6ba-784bb56e026a-kube-api-access-8mf9p\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:05:59.269855 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:05:59.269664 4376 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/00a54582-1572-4f7e-a6ba-784bb56e026a-tmp\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:05:59.883985 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:05:59.883950 4376 generic.go:356] "Generic (PLEG): container finished" podID="00a54582-1572-4f7e-a6ba-784bb56e026a" containerID="399a966be242c408afdb74d7fb4c4eeaa828dcf140dc1c42d2a6a61ae871eccb" exitCode=0 Oct 31 04:05:59.884151 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:05:59.883997 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-557794d575-dkt97" Oct 31 04:05:59.884151 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:05:59.884013 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-557794d575-dkt97" event={"ID":"00a54582-1572-4f7e-a6ba-784bb56e026a","Type":"ContainerDied","Data":"399a966be242c408afdb74d7fb4c4eeaa828dcf140dc1c42d2a6a61ae871eccb"} Oct 31 04:05:59.884151 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:05:59.884054 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-557794d575-dkt97" event={"ID":"00a54582-1572-4f7e-a6ba-784bb56e026a","Type":"ContainerDied","Data":"1299f8c4b8a7306144b4d0092bf1809ae16b7a3a5fd595fa4b7384f595b0b5d3"} Oct 31 04:05:59.884151 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:05:59.884073 4376 scope.go:122] "RemoveContainer" containerID="399a966be242c408afdb74d7fb4c4eeaa828dcf140dc1c42d2a6a61ae871eccb" Oct 31 04:05:59.891847 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:05:59.891760 4376 scope.go:122] "RemoveContainer" containerID="399a966be242c408afdb74d7fb4c4eeaa828dcf140dc1c42d2a6a61ae871eccb" Oct 31 04:05:59.892092 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:05:59.892070 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"399a966be242c408afdb74d7fb4c4eeaa828dcf140dc1c42d2a6a61ae871eccb\": container with ID starting with 399a966be242c408afdb74d7fb4c4eeaa828dcf140dc1c42d2a6a61ae871eccb not found: ID does not exist" containerID="399a966be242c408afdb74d7fb4c4eeaa828dcf140dc1c42d2a6a61ae871eccb" Oct 31 04:05:59.892162 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:05:59.892107 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"399a966be242c408afdb74d7fb4c4eeaa828dcf140dc1c42d2a6a61ae871eccb"} err="failed to get container status \"399a966be242c408afdb74d7fb4c4eeaa828dcf140dc1c42d2a6a61ae871eccb\": rpc error: code = NotFound desc = could not find container \"399a966be242c408afdb74d7fb4c4eeaa828dcf140dc1c42d2a6a61ae871eccb\": container with ID starting with 399a966be242c408afdb74d7fb4c4eeaa828dcf140dc1c42d2a6a61ae871eccb not found: ID does not exist" Oct 31 04:05:59.940251 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:05:59.940206 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-557794d575-dkt97"] Oct 31 04:05:59.953421 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:05:59.953387 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-557794d575-dkt97"] Oct 31 04:06:00.443815 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:06:00.443786 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00a54582-1572-4f7e-a6ba-784bb56e026a" path="/var/lib/kubelet/pods/00a54582-1572-4f7e-a6ba-784bb56e026a/volumes" Oct 31 04:06:10.999678 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:06:10.999633 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 04:06:40.998023 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:06:40.997988 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 04:07:10.996958 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:07:10.996913 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 04:07:40.997014 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:07:40.996979 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 04:08:10.997317 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:08:10.997274 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 04:08:16.763117 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:08:16.763083 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cl5nkn"] Oct 31 04:08:16.763771 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:08:16.763489 4376 cpu_manager.go:414] "containerMap: removing container" podUID="00a54582-1572-4f7e-a6ba-784bb56e026a" containerName="console" Oct 31 04:08:16.763771 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:08:16.763507 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00a54582-1572-4f7e-a6ba-784bb56e026a" containerName="console" Oct 31 04:08:16.763771 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:08:16.763573 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="00a54582-1572-4f7e-a6ba-784bb56e026a" containerName="console" Oct 31 04:08:16.765414 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:08:16.765395 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cl5nkn" Oct 31 04:08:16.779556 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:08:16.779536 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2ce891d9-fcb5-4125-82f0-d462edaf6b73-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cl5nkn\" (UID: \"2ce891d9-fcb5-4125-82f0-d462edaf6b73\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cl5nkn" Oct 31 04:08:16.779655 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:08:16.779564 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvqp4\" (UniqueName: \"kubernetes.io/projected/2ce891d9-fcb5-4125-82f0-d462edaf6b73-kube-api-access-tvqp4\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cl5nkn\" (UID: \"2ce891d9-fcb5-4125-82f0-d462edaf6b73\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cl5nkn" Oct 31 04:08:16.779655 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:08:16.779597 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2ce891d9-fcb5-4125-82f0-d462edaf6b73-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cl5nkn\" (UID: \"2ce891d9-fcb5-4125-82f0-d462edaf6b73\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cl5nkn" Oct 31 04:08:16.796095 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:08:16.796074 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Oct 31 04:08:16.796164 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:08:16.796091 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Oct 31 04:08:16.796164 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:08:16.796091 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-tp9sg\"" Oct 31 04:08:16.880171 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:08:16.880139 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2ce891d9-fcb5-4125-82f0-d462edaf6b73-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cl5nkn\" (UID: \"2ce891d9-fcb5-4125-82f0-d462edaf6b73\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cl5nkn" Oct 31 04:08:16.880171 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:08:16.880173 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvqp4\" (UniqueName: \"kubernetes.io/projected/2ce891d9-fcb5-4125-82f0-d462edaf6b73-kube-api-access-tvqp4\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cl5nkn\" (UID: \"2ce891d9-fcb5-4125-82f0-d462edaf6b73\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cl5nkn" Oct 31 04:08:16.880412 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:08:16.880222 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2ce891d9-fcb5-4125-82f0-d462edaf6b73-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cl5nkn\" (UID: \"2ce891d9-fcb5-4125-82f0-d462edaf6b73\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cl5nkn" Oct 31 04:08:16.880537 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:08:16.880518 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2ce891d9-fcb5-4125-82f0-d462edaf6b73-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cl5nkn\" (UID: \"2ce891d9-fcb5-4125-82f0-d462edaf6b73\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cl5nkn" Oct 31 04:08:16.880585 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:08:16.880568 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2ce891d9-fcb5-4125-82f0-d462edaf6b73-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cl5nkn\" (UID: \"2ce891d9-fcb5-4125-82f0-d462edaf6b73\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cl5nkn" Oct 31 04:08:16.884924 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:08:16.884900 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cl5nkn"] Oct 31 04:08:16.903965 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:08:16.903937 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvqp4\" (UniqueName: \"kubernetes.io/projected/2ce891d9-fcb5-4125-82f0-d462edaf6b73-kube-api-access-tvqp4\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cl5nkn\" (UID: \"2ce891d9-fcb5-4125-82f0-d462edaf6b73\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cl5nkn" Oct 31 04:08:16.925339 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:08:16.925303 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jpmnr"] Oct 31 04:08:16.927298 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:08:16.927284 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jpmnr" Oct 31 04:08:16.942976 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:08:16.942960 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-zvgb6\"" Oct 31 04:08:16.949961 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:08:16.949941 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jpmnr"] Oct 31 04:08:16.980986 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:08:16.980968 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/32706069-0f3b-4e47-b883-e94ca32b14d1-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jpmnr\" (UID: \"32706069-0f3b-4e47-b883-e94ca32b14d1\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jpmnr" Oct 31 04:08:16.981075 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:08:16.981005 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5lm2\" (UniqueName: \"kubernetes.io/projected/32706069-0f3b-4e47-b883-e94ca32b14d1-kube-api-access-b5lm2\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jpmnr\" (UID: \"32706069-0f3b-4e47-b883-e94ca32b14d1\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jpmnr" Oct 31 04:08:16.981116 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:08:16.981085 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/32706069-0f3b-4e47-b883-e94ca32b14d1-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jpmnr\" (UID: \"32706069-0f3b-4e47-b883-e94ca32b14d1\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jpmnr" Oct 31 04:08:17.077149 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:08:17.077082 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cl5nkn" Oct 31 04:08:17.081868 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:08:17.081850 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/32706069-0f3b-4e47-b883-e94ca32b14d1-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jpmnr\" (UID: \"32706069-0f3b-4e47-b883-e94ca32b14d1\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jpmnr" Oct 31 04:08:17.081935 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:08:17.081886 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/32706069-0f3b-4e47-b883-e94ca32b14d1-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jpmnr\" (UID: \"32706069-0f3b-4e47-b883-e94ca32b14d1\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jpmnr" Oct 31 04:08:17.081935 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:08:17.081916 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5lm2\" (UniqueName: \"kubernetes.io/projected/32706069-0f3b-4e47-b883-e94ca32b14d1-kube-api-access-b5lm2\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jpmnr\" (UID: \"32706069-0f3b-4e47-b883-e94ca32b14d1\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jpmnr" Oct 31 04:08:17.082260 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:08:17.082231 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/32706069-0f3b-4e47-b883-e94ca32b14d1-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jpmnr\" (UID: \"32706069-0f3b-4e47-b883-e94ca32b14d1\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jpmnr" Oct 31 04:08:17.082260 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:08:17.082244 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/32706069-0f3b-4e47-b883-e94ca32b14d1-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jpmnr\" (UID: \"32706069-0f3b-4e47-b883-e94ca32b14d1\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jpmnr" Oct 31 04:08:17.105663 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:08:17.105640 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5lm2\" (UniqueName: \"kubernetes.io/projected/32706069-0f3b-4e47-b883-e94ca32b14d1-kube-api-access-b5lm2\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jpmnr\" (UID: \"32706069-0f3b-4e47-b883-e94ca32b14d1\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jpmnr" Oct 31 04:08:17.238913 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:08:17.238883 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jpmnr" Oct 31 04:08:17.369238 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:08:17.369215 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cl5nkn"] Oct 31 04:08:17.370018 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:08:17.369994 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ce891d9_fcb5_4125_82f0_d462edaf6b73.slice/crio-17160fa5ab5b372341e8e9fdd3d7f59f66638ae29dbca84b80881b546f51d3da WatchSource:0}: Error finding container 17160fa5ab5b372341e8e9fdd3d7f59f66638ae29dbca84b80881b546f51d3da: Status 404 returned error can't find the container with id 17160fa5ab5b372341e8e9fdd3d7f59f66638ae29dbca84b80881b546f51d3da Oct 31 04:08:17.383952 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:08:17.383931 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-bvgmx"] Oct 31 04:08:17.389824 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:08:17.389795 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-bvgmx" Oct 31 04:08:17.407488 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:08:17.407472 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 31 04:08:17.408708 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:08:17.408676 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 31 04:08:17.408815 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:08:17.408772 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-bvgmx"] Oct 31 04:08:17.409382 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:08:17.409359 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 31 04:08:17.409472 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:08:17.409442 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 31 04:08:17.409518 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:08:17.409492 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 31 04:08:17.485416 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:08:17.485380 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/375f24d5-f66f-4a0f-be70-59bc85fa0352-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-bvgmx\" (UID: \"375f24d5-f66f-4a0f-be70-59bc85fa0352\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-bvgmx" Oct 31 04:08:17.485695 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:08:17.485427 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/375f24d5-f66f-4a0f-be70-59bc85fa0352-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-bvgmx\" (UID: \"375f24d5-f66f-4a0f-be70-59bc85fa0352\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-bvgmx" Oct 31 04:08:17.485695 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:08:17.485450 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwmnr\" (UniqueName: \"kubernetes.io/projected/375f24d5-f66f-4a0f-be70-59bc85fa0352-kube-api-access-xwmnr\") pod \"istiod-openshift-gateway-7876bfcb74-bvgmx\" (UID: \"375f24d5-f66f-4a0f-be70-59bc85fa0352\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-bvgmx" Oct 31 04:08:17.485695 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:08:17.485670 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/375f24d5-f66f-4a0f-be70-59bc85fa0352-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-bvgmx\" (UID: \"375f24d5-f66f-4a0f-be70-59bc85fa0352\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-bvgmx" Oct 31 04:08:17.485695 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:08:17.485694 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/375f24d5-f66f-4a0f-be70-59bc85fa0352-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-bvgmx\" (UID: \"375f24d5-f66f-4a0f-be70-59bc85fa0352\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-bvgmx" Oct 31 04:08:17.485859 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:08:17.485724 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/375f24d5-f66f-4a0f-be70-59bc85fa0352-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-bvgmx\" (UID: \"375f24d5-f66f-4a0f-be70-59bc85fa0352\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-bvgmx" Oct 31 04:08:17.485859 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:08:17.485775 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/375f24d5-f66f-4a0f-be70-59bc85fa0352-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-bvgmx\" (UID: \"375f24d5-f66f-4a0f-be70-59bc85fa0352\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-bvgmx" Oct 31 04:08:17.507653 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:08:17.507628 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 31 04:08:17.507773 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:08:17.507709 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-m4kc9\"" Oct 31 04:08:17.527536 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:08:17.527519 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jpmnr"] Oct 31 04:08:17.528270 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:08:17.528251 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32706069_0f3b_4e47_b883_e94ca32b14d1.slice/crio-92a02b88c38e8b012ab44b6faf1a2e418e38c64728ac9869d439f267e21590b3 WatchSource:0}: Error finding container 92a02b88c38e8b012ab44b6faf1a2e418e38c64728ac9869d439f267e21590b3: Status 404 returned error can't find the container with id 92a02b88c38e8b012ab44b6faf1a2e418e38c64728ac9869d439f267e21590b3 Oct 31 04:08:17.587174 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:08:17.587151 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/375f24d5-f66f-4a0f-be70-59bc85fa0352-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-bvgmx\" (UID: \"375f24d5-f66f-4a0f-be70-59bc85fa0352\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-bvgmx" Oct 31 04:08:17.587282 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:08:17.587179 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/375f24d5-f66f-4a0f-be70-59bc85fa0352-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-bvgmx\" (UID: \"375f24d5-f66f-4a0f-be70-59bc85fa0352\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-bvgmx" Oct 31 04:08:17.587282 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:08:17.587199 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/375f24d5-f66f-4a0f-be70-59bc85fa0352-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-bvgmx\" (UID: \"375f24d5-f66f-4a0f-be70-59bc85fa0352\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-bvgmx" Oct 31 04:08:17.587282 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:08:17.587217 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/375f24d5-f66f-4a0f-be70-59bc85fa0352-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-bvgmx\" (UID: \"375f24d5-f66f-4a0f-be70-59bc85fa0352\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-bvgmx" Oct 31 04:08:17.587282 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:08:17.587263 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/375f24d5-f66f-4a0f-be70-59bc85fa0352-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-bvgmx\" (UID: \"375f24d5-f66f-4a0f-be70-59bc85fa0352\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-bvgmx" Oct 31 04:08:17.587490 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:08:17.587291 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/375f24d5-f66f-4a0f-be70-59bc85fa0352-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-bvgmx\" (UID: \"375f24d5-f66f-4a0f-be70-59bc85fa0352\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-bvgmx" Oct 31 04:08:17.587490 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:08:17.587313 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwmnr\" (UniqueName: \"kubernetes.io/projected/375f24d5-f66f-4a0f-be70-59bc85fa0352-kube-api-access-xwmnr\") pod \"istiod-openshift-gateway-7876bfcb74-bvgmx\" (UID: \"375f24d5-f66f-4a0f-be70-59bc85fa0352\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-bvgmx" Oct 31 04:08:17.587996 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:08:17.587972 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/375f24d5-f66f-4a0f-be70-59bc85fa0352-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-bvgmx\" (UID: \"375f24d5-f66f-4a0f-be70-59bc85fa0352\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-bvgmx" Oct 31 04:08:17.589405 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:08:17.589382 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/375f24d5-f66f-4a0f-be70-59bc85fa0352-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-bvgmx\" (UID: \"375f24d5-f66f-4a0f-be70-59bc85fa0352\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-bvgmx" Oct 31 04:08:17.589526 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:08:17.589510 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/375f24d5-f66f-4a0f-be70-59bc85fa0352-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-bvgmx\" (UID: \"375f24d5-f66f-4a0f-be70-59bc85fa0352\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-bvgmx" Oct 31 04:08:17.589600 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:08:17.589553 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/375f24d5-f66f-4a0f-be70-59bc85fa0352-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-bvgmx\" (UID: \"375f24d5-f66f-4a0f-be70-59bc85fa0352\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-bvgmx" Oct 31 04:08:17.589674 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:08:17.589652 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/375f24d5-f66f-4a0f-be70-59bc85fa0352-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-bvgmx\" (UID: \"375f24d5-f66f-4a0f-be70-59bc85fa0352\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-bvgmx" Oct 31 04:08:17.610860 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:08:17.610840 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/375f24d5-f66f-4a0f-be70-59bc85fa0352-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-bvgmx\" (UID: \"375f24d5-f66f-4a0f-be70-59bc85fa0352\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-bvgmx" Oct 31 04:08:17.611026 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:08:17.611010 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwmnr\" (UniqueName: \"kubernetes.io/projected/375f24d5-f66f-4a0f-be70-59bc85fa0352-kube-api-access-xwmnr\") pod \"istiod-openshift-gateway-7876bfcb74-bvgmx\" (UID: \"375f24d5-f66f-4a0f-be70-59bc85fa0352\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-bvgmx" Oct 31 04:08:17.700303 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:08:17.700240 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-bvgmx" Oct 31 04:08:18.003983 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:08:18.003808 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-bvgmx"] Oct 31 04:08:18.007134 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:08:18.007106 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod375f24d5_f66f_4a0f_be70_59bc85fa0352.slice/crio-b56b7ed148389508bde058592ebc838dac508e30e657afb18eb9876e916a8c92 WatchSource:0}: Error finding container b56b7ed148389508bde058592ebc838dac508e30e657afb18eb9876e916a8c92: Status 404 returned error can't find the container with id b56b7ed148389508bde058592ebc838dac508e30e657afb18eb9876e916a8c92 Oct 31 04:08:18.305544 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:08:18.305460 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jpmnr" event={"ID":"32706069-0f3b-4e47-b883-e94ca32b14d1","Type":"ContainerStarted","Data":"92a02b88c38e8b012ab44b6faf1a2e418e38c64728ac9869d439f267e21590b3"} Oct 31 04:08:18.307045 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:08:18.306989 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cl5nkn" event={"ID":"2ce891d9-fcb5-4125-82f0-d462edaf6b73","Type":"ContainerStarted","Data":"17160fa5ab5b372341e8e9fdd3d7f59f66638ae29dbca84b80881b546f51d3da"} Oct 31 04:08:18.308418 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:08:18.308389 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-bvgmx" event={"ID":"375f24d5-f66f-4a0f-be70-59bc85fa0352","Type":"ContainerStarted","Data":"b56b7ed148389508bde058592ebc838dac508e30e657afb18eb9876e916a8c92"} Oct 31 04:08:20.987917 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:08:20.987884 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:08:20.988264 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:08:20.987959 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:08:21.322866 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:08:21.322755 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-bvgmx" event={"ID":"375f24d5-f66f-4a0f-be70-59bc85fa0352","Type":"ContainerStarted","Data":"5000d073035f05154ea361e364716cb81b60de25091e9c70df21f229de9b1625"} Oct 31 04:08:21.323021 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:08:21.322873 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-bvgmx" Oct 31 04:08:21.804992 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:08:21.804927 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-bvgmx" podStartSLOduration=1.8277114920000002 podStartE2EDuration="4.804908366s" totalImagesPullingTime="2.977196874s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:08:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:08:21.381294393 +0000 UTC m=+249.095629011" watchObservedRunningTime="2026-10-31 04:08:21.804908366 +0000 UTC m=+249.519242983" Oct 31 04:08:22.328900 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:08:22.328862 4376 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-bvgmx container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 04:08:22.329491 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:08:22.328921 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-bvgmx" podUID="375f24d5-f66f-4a0f-be70-59bc85fa0352" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 04:08:24.335531 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:08:24.335442 4376 generic.go:356] "Generic (PLEG): container finished" podID="32706069-0f3b-4e47-b883-e94ca32b14d1" containerID="66a026201f61a832a7161995fc7c7f35e1cea7a18f44d7e27f85261a1ee2bac5" exitCode=0 Oct 31 04:08:24.335531 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:08:24.335520 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jpmnr" event={"ID":"32706069-0f3b-4e47-b883-e94ca32b14d1","Type":"ContainerDied","Data":"66a026201f61a832a7161995fc7c7f35e1cea7a18f44d7e27f85261a1ee2bac5"} Oct 31 04:08:24.337107 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:08:24.337080 4376 generic.go:356] "Generic (PLEG): container finished" podID="2ce891d9-fcb5-4125-82f0-d462edaf6b73" containerID="742ba7cada2095d51e69cc14f9dfbc6848aa9a3194f49c713910e90c9a399562" exitCode=0 Oct 31 04:08:24.337203 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:08:24.337135 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cl5nkn" event={"ID":"2ce891d9-fcb5-4125-82f0-d462edaf6b73","Type":"ContainerDied","Data":"742ba7cada2095d51e69cc14f9dfbc6848aa9a3194f49c713910e90c9a399562"} Oct 31 04:08:25.327957 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:08:25.327880 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-bvgmx" Oct 31 04:08:25.342136 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:08:25.342106 4376 generic.go:356] "Generic (PLEG): container finished" podID="32706069-0f3b-4e47-b883-e94ca32b14d1" containerID="362757bb6cac518140bfda97b51ecd1fd9f806a22773b8131c785e83480e1c7f" exitCode=0 Oct 31 04:08:25.342504 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:08:25.342195 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jpmnr" event={"ID":"32706069-0f3b-4e47-b883-e94ca32b14d1","Type":"ContainerDied","Data":"362757bb6cac518140bfda97b51ecd1fd9f806a22773b8131c785e83480e1c7f"} Oct 31 04:08:27.353761 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:08:27.353730 4376 generic.go:356] "Generic (PLEG): container finished" podID="2ce891d9-fcb5-4125-82f0-d462edaf6b73" containerID="5ca6d5527372b62f0c119d901e441a3c4c0a37c4b5718fd0fc2cf573bc0bdd62" exitCode=0 Oct 31 04:08:27.354138 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:08:27.353797 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cl5nkn" event={"ID":"2ce891d9-fcb5-4125-82f0-d462edaf6b73","Type":"ContainerDied","Data":"5ca6d5527372b62f0c119d901e441a3c4c0a37c4b5718fd0fc2cf573bc0bdd62"} Oct 31 04:08:32.379173 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:08:32.379100 4376 generic.go:356] "Generic (PLEG): container finished" podID="32706069-0f3b-4e47-b883-e94ca32b14d1" containerID="70f31c56dbb54eb25b15de4a3b30614ed0cfdad7cf5adc7b5e3813a57b93b3fc" exitCode=0 Oct 31 04:08:32.379173 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:08:32.379166 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jpmnr" event={"ID":"32706069-0f3b-4e47-b883-e94ca32b14d1","Type":"ContainerDied","Data":"70f31c56dbb54eb25b15de4a3b30614ed0cfdad7cf5adc7b5e3813a57b93b3fc"} Oct 31 04:08:33.510407 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:08:33.510380 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jpmnr" Oct 31 04:08:33.626986 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:08:33.626958 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32706069-0f3b-4e47-b883-e94ca32b14d1-kube-api-access-b5lm2\" (UniqueName: \"kubernetes.io/projected/32706069-0f3b-4e47-b883-e94ca32b14d1-kube-api-access-b5lm2\") pod \"32706069-0f3b-4e47-b883-e94ca32b14d1\" (UID: \"32706069-0f3b-4e47-b883-e94ca32b14d1\") " Oct 31 04:08:33.627126 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:08:33.626994 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32706069-0f3b-4e47-b883-e94ca32b14d1-bundle\" (UniqueName: \"kubernetes.io/empty-dir/32706069-0f3b-4e47-b883-e94ca32b14d1-bundle\") pod \"32706069-0f3b-4e47-b883-e94ca32b14d1\" (UID: \"32706069-0f3b-4e47-b883-e94ca32b14d1\") " Oct 31 04:08:33.627126 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:08:33.627044 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32706069-0f3b-4e47-b883-e94ca32b14d1-util\" (UniqueName: \"kubernetes.io/empty-dir/32706069-0f3b-4e47-b883-e94ca32b14d1-util\") pod \"32706069-0f3b-4e47-b883-e94ca32b14d1\" (UID: \"32706069-0f3b-4e47-b883-e94ca32b14d1\") " Oct 31 04:08:33.627604 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:08:33.627573 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32706069-0f3b-4e47-b883-e94ca32b14d1-bundle" pod "32706069-0f3b-4e47-b883-e94ca32b14d1" (UID: "32706069-0f3b-4e47-b883-e94ca32b14d1"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:08:33.629084 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:08:33.629051 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32706069-0f3b-4e47-b883-e94ca32b14d1-kube-api-access-b5lm2" pod "32706069-0f3b-4e47-b883-e94ca32b14d1" (UID: "32706069-0f3b-4e47-b883-e94ca32b14d1"). InnerVolumeSpecName "kube-api-access-b5lm2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:08:33.631715 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:08:33.631689 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32706069-0f3b-4e47-b883-e94ca32b14d1-util" pod "32706069-0f3b-4e47-b883-e94ca32b14d1" (UID: "32706069-0f3b-4e47-b883-e94ca32b14d1"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:08:33.728398 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:08:33.728366 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5lm2\" (UniqueName: \"kubernetes.io/projected/32706069-0f3b-4e47-b883-e94ca32b14d1-kube-api-access-b5lm2\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:08:33.728555 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:08:33.728417 4376 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/32706069-0f3b-4e47-b883-e94ca32b14d1-bundle\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:08:33.728555 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:08:33.728427 4376 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/32706069-0f3b-4e47-b883-e94ca32b14d1-util\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:08:34.387427 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:08:34.387389 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jpmnr" event={"ID":"32706069-0f3b-4e47-b883-e94ca32b14d1","Type":"ContainerDied","Data":"92a02b88c38e8b012ab44b6faf1a2e418e38c64728ac9869d439f267e21590b3"} Oct 31 04:08:34.387608 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:08:34.387465 4376 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="92a02b88c38e8b012ab44b6faf1a2e418e38c64728ac9869d439f267e21590b3" Oct 31 04:08:34.387608 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:08:34.387407 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jpmnr" Oct 31 04:08:34.389627 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:08:34.389602 4376 generic.go:356] "Generic (PLEG): container finished" podID="2ce891d9-fcb5-4125-82f0-d462edaf6b73" containerID="ecde4f41b374c1926943a4f6c1150c02afe85929f9eb2601d7aaef723b93cd5a" exitCode=0 Oct 31 04:08:34.389739 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:08:34.389653 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cl5nkn" event={"ID":"2ce891d9-fcb5-4125-82f0-d462edaf6b73","Type":"ContainerDied","Data":"ecde4f41b374c1926943a4f6c1150c02afe85929f9eb2601d7aaef723b93cd5a"} Oct 31 04:08:35.512925 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:08:35.512903 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cl5nkn" Oct 31 04:08:35.641046 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:08:35.641016 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ce891d9-fcb5-4125-82f0-d462edaf6b73-util\" (UniqueName: \"kubernetes.io/empty-dir/2ce891d9-fcb5-4125-82f0-d462edaf6b73-util\") pod \"2ce891d9-fcb5-4125-82f0-d462edaf6b73\" (UID: \"2ce891d9-fcb5-4125-82f0-d462edaf6b73\") " Oct 31 04:08:35.641204 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:08:35.641063 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ce891d9-fcb5-4125-82f0-d462edaf6b73-bundle\" (UniqueName: \"kubernetes.io/empty-dir/2ce891d9-fcb5-4125-82f0-d462edaf6b73-bundle\") pod \"2ce891d9-fcb5-4125-82f0-d462edaf6b73\" (UID: \"2ce891d9-fcb5-4125-82f0-d462edaf6b73\") " Oct 31 04:08:35.641204 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:08:35.641110 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ce891d9-fcb5-4125-82f0-d462edaf6b73-kube-api-access-tvqp4\" (UniqueName: \"kubernetes.io/projected/2ce891d9-fcb5-4125-82f0-d462edaf6b73-kube-api-access-tvqp4\") pod \"2ce891d9-fcb5-4125-82f0-d462edaf6b73\" (UID: \"2ce891d9-fcb5-4125-82f0-d462edaf6b73\") " Oct 31 04:08:35.641711 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:08:35.641680 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ce891d9-fcb5-4125-82f0-d462edaf6b73-bundle" pod "2ce891d9-fcb5-4125-82f0-d462edaf6b73" (UID: "2ce891d9-fcb5-4125-82f0-d462edaf6b73"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:08:35.643138 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:08:35.643105 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ce891d9-fcb5-4125-82f0-d462edaf6b73-kube-api-access-tvqp4" pod "2ce891d9-fcb5-4125-82f0-d462edaf6b73" (UID: "2ce891d9-fcb5-4125-82f0-d462edaf6b73"). InnerVolumeSpecName "kube-api-access-tvqp4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:08:35.645293 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:08:35.645251 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ce891d9-fcb5-4125-82f0-d462edaf6b73-util" pod "2ce891d9-fcb5-4125-82f0-d462edaf6b73" (UID: "2ce891d9-fcb5-4125-82f0-d462edaf6b73"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:08:35.742387 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:08:35.742364 4376 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2ce891d9-fcb5-4125-82f0-d462edaf6b73-util\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:08:35.742387 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:08:35.742384 4376 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2ce891d9-fcb5-4125-82f0-d462edaf6b73-bundle\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:08:35.742506 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:08:35.742393 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvqp4\" (UniqueName: \"kubernetes.io/projected/2ce891d9-fcb5-4125-82f0-d462edaf6b73-kube-api-access-tvqp4\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:08:36.397586 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:08:36.397557 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cl5nkn" Oct 31 04:08:36.397757 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:08:36.397557 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cl5nkn" event={"ID":"2ce891d9-fcb5-4125-82f0-d462edaf6b73","Type":"ContainerDied","Data":"17160fa5ab5b372341e8e9fdd3d7f59f66638ae29dbca84b80881b546f51d3da"} Oct 31 04:08:36.397757 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:08:36.397673 4376 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="17160fa5ab5b372341e8e9fdd3d7f59f66638ae29dbca84b80881b546f51d3da" Oct 31 04:08:40.997221 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:08:40.997183 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 04:08:51.318107 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:08:51.318075 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-79b6bb9d79-gcpjq"] Oct 31 04:08:51.318723 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:08:51.318447 4376 cpu_manager.go:414] "containerMap: removing container" podUID="32706069-0f3b-4e47-b883-e94ca32b14d1" containerName="extract" Oct 31 04:08:51.318723 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:08:51.318465 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32706069-0f3b-4e47-b883-e94ca32b14d1" containerName="extract" Oct 31 04:08:51.318723 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:08:51.318486 4376 cpu_manager.go:414] "containerMap: removing container" podUID="32706069-0f3b-4e47-b883-e94ca32b14d1" containerName="util" Oct 31 04:08:51.318723 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:08:51.318493 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32706069-0f3b-4e47-b883-e94ca32b14d1" containerName="util" Oct 31 04:08:51.318723 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:08:51.318511 4376 cpu_manager.go:414] "containerMap: removing container" podUID="2ce891d9-fcb5-4125-82f0-d462edaf6b73" containerName="pull" Oct 31 04:08:51.318723 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:08:51.318517 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ce891d9-fcb5-4125-82f0-d462edaf6b73" containerName="pull" Oct 31 04:08:51.318723 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:08:51.318526 4376 cpu_manager.go:414] "containerMap: removing container" podUID="2ce891d9-fcb5-4125-82f0-d462edaf6b73" containerName="extract" Oct 31 04:08:51.318723 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:08:51.318532 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ce891d9-fcb5-4125-82f0-d462edaf6b73" containerName="extract" Oct 31 04:08:51.318723 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:08:51.318543 4376 cpu_manager.go:414] "containerMap: removing container" podUID="2ce891d9-fcb5-4125-82f0-d462edaf6b73" containerName="util" Oct 31 04:08:51.318723 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:08:51.318552 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ce891d9-fcb5-4125-82f0-d462edaf6b73" containerName="util" Oct 31 04:08:51.318723 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:08:51.318560 4376 cpu_manager.go:414] "containerMap: removing container" podUID="32706069-0f3b-4e47-b883-e94ca32b14d1" containerName="pull" Oct 31 04:08:51.318723 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:08:51.318566 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32706069-0f3b-4e47-b883-e94ca32b14d1" containerName="pull" Oct 31 04:08:51.318723 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:08:51.318626 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="32706069-0f3b-4e47-b883-e94ca32b14d1" containerName="extract" Oct 31 04:08:51.318723 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:08:51.318640 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ce891d9-fcb5-4125-82f0-d462edaf6b73" containerName="extract" Oct 31 04:08:51.322225 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:08:51.322205 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79b6bb9d79-gcpjq" Oct 31 04:08:51.357893 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:08:51.357869 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-79b6bb9d79-gcpjq"] Oct 31 04:08:51.413987 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:08:51.413960 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 31 04:08:51.413987 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:08:51.413975 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Oct 31 04:08:51.414165 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:08:51.414055 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Oct 31 04:08:51.414165 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:08:51.414073 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 31 04:08:51.414165 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:08:51.414097 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Oct 31 04:08:51.414262 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:08:51.414200 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-cbq7v\"" Oct 31 04:08:51.451230 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:08:51.451207 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/f77f79ec-3717-4a05-9d82-2b5e448ac98f-metrics-cert\") pod \"opendatahub-operator-controller-manager-79b6bb9d79-gcpjq\" (UID: \"f77f79ec-3717-4a05-9d82-2b5e448ac98f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79b6bb9d79-gcpjq" Oct 31 04:08:51.451350 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:08:51.451238 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jcw7\" (UniqueName: \"kubernetes.io/projected/f77f79ec-3717-4a05-9d82-2b5e448ac98f-kube-api-access-5jcw7\") pod \"opendatahub-operator-controller-manager-79b6bb9d79-gcpjq\" (UID: \"f77f79ec-3717-4a05-9d82-2b5e448ac98f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79b6bb9d79-gcpjq" Oct 31 04:08:51.451350 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:08:51.451259 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f77f79ec-3717-4a05-9d82-2b5e448ac98f-tmp\") pod \"opendatahub-operator-controller-manager-79b6bb9d79-gcpjq\" (UID: \"f77f79ec-3717-4a05-9d82-2b5e448ac98f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79b6bb9d79-gcpjq" Oct 31 04:08:51.451350 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:08:51.451275 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f77f79ec-3717-4a05-9d82-2b5e448ac98f-webhook-cert\") pod \"opendatahub-operator-controller-manager-79b6bb9d79-gcpjq\" (UID: \"f77f79ec-3717-4a05-9d82-2b5e448ac98f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79b6bb9d79-gcpjq" Oct 31 04:08:51.451478 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:08:51.451403 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f77f79ec-3717-4a05-9d82-2b5e448ac98f-manifests\") pod \"opendatahub-operator-controller-manager-79b6bb9d79-gcpjq\" (UID: \"f77f79ec-3717-4a05-9d82-2b5e448ac98f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79b6bb9d79-gcpjq" Oct 31 04:08:51.451514 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:08:51.451488 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f77f79ec-3717-4a05-9d82-2b5e448ac98f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-79b6bb9d79-gcpjq\" (UID: \"f77f79ec-3717-4a05-9d82-2b5e448ac98f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79b6bb9d79-gcpjq" Oct 31 04:08:51.551744 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:08:51.551721 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/f77f79ec-3717-4a05-9d82-2b5e448ac98f-metrics-cert\") pod \"opendatahub-operator-controller-manager-79b6bb9d79-gcpjq\" (UID: \"f77f79ec-3717-4a05-9d82-2b5e448ac98f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79b6bb9d79-gcpjq" Oct 31 04:08:51.551837 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:08:51.551751 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jcw7\" (UniqueName: \"kubernetes.io/projected/f77f79ec-3717-4a05-9d82-2b5e448ac98f-kube-api-access-5jcw7\") pod \"opendatahub-operator-controller-manager-79b6bb9d79-gcpjq\" (UID: \"f77f79ec-3717-4a05-9d82-2b5e448ac98f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79b6bb9d79-gcpjq" Oct 31 04:08:51.551837 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:08:51.551773 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f77f79ec-3717-4a05-9d82-2b5e448ac98f-tmp\") pod \"opendatahub-operator-controller-manager-79b6bb9d79-gcpjq\" (UID: \"f77f79ec-3717-4a05-9d82-2b5e448ac98f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79b6bb9d79-gcpjq" Oct 31 04:08:51.551837 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:08:51.551793 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f77f79ec-3717-4a05-9d82-2b5e448ac98f-webhook-cert\") pod \"opendatahub-operator-controller-manager-79b6bb9d79-gcpjq\" (UID: \"f77f79ec-3717-4a05-9d82-2b5e448ac98f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79b6bb9d79-gcpjq" Oct 31 04:08:51.552010 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:08:51.551987 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f77f79ec-3717-4a05-9d82-2b5e448ac98f-manifests\") pod \"opendatahub-operator-controller-manager-79b6bb9d79-gcpjq\" (UID: \"f77f79ec-3717-4a05-9d82-2b5e448ac98f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79b6bb9d79-gcpjq" Oct 31 04:08:51.552073 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:08:51.552056 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f77f79ec-3717-4a05-9d82-2b5e448ac98f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-79b6bb9d79-gcpjq\" (UID: \"f77f79ec-3717-4a05-9d82-2b5e448ac98f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79b6bb9d79-gcpjq" Oct 31 04:08:51.552420 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:08:51.552388 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f77f79ec-3717-4a05-9d82-2b5e448ac98f-manifests\") pod \"opendatahub-operator-controller-manager-79b6bb9d79-gcpjq\" (UID: \"f77f79ec-3717-4a05-9d82-2b5e448ac98f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79b6bb9d79-gcpjq" Oct 31 04:08:51.554074 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:08:51.554052 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f77f79ec-3717-4a05-9d82-2b5e448ac98f-tmp\") pod \"opendatahub-operator-controller-manager-79b6bb9d79-gcpjq\" (UID: \"f77f79ec-3717-4a05-9d82-2b5e448ac98f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79b6bb9d79-gcpjq" Oct 31 04:08:51.554436 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:08:51.554414 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f77f79ec-3717-4a05-9d82-2b5e448ac98f-webhook-cert\") pod \"opendatahub-operator-controller-manager-79b6bb9d79-gcpjq\" (UID: \"f77f79ec-3717-4a05-9d82-2b5e448ac98f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79b6bb9d79-gcpjq" Oct 31 04:08:51.554486 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:08:51.554422 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f77f79ec-3717-4a05-9d82-2b5e448ac98f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-79b6bb9d79-gcpjq\" (UID: \"f77f79ec-3717-4a05-9d82-2b5e448ac98f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79b6bb9d79-gcpjq" Oct 31 04:08:51.554792 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:08:51.554769 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/f77f79ec-3717-4a05-9d82-2b5e448ac98f-metrics-cert\") pod \"opendatahub-operator-controller-manager-79b6bb9d79-gcpjq\" (UID: \"f77f79ec-3717-4a05-9d82-2b5e448ac98f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79b6bb9d79-gcpjq" Oct 31 04:08:51.576072 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:08:51.576023 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jcw7\" (UniqueName: \"kubernetes.io/projected/f77f79ec-3717-4a05-9d82-2b5e448ac98f-kube-api-access-5jcw7\") pod \"opendatahub-operator-controller-manager-79b6bb9d79-gcpjq\" (UID: \"f77f79ec-3717-4a05-9d82-2b5e448ac98f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79b6bb9d79-gcpjq" Oct 31 04:08:51.633576 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:08:51.633556 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79b6bb9d79-gcpjq" Oct 31 04:08:51.982387 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:08:51.982363 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-79b6bb9d79-gcpjq"] Oct 31 04:08:51.983085 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:08:51.983061 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf77f79ec_3717_4a05_9d82_2b5e448ac98f.slice/crio-74e74c40c7788d353aa626801e7413719610b25b416327576d2f7641c0a27df5 WatchSource:0}: Error finding container 74e74c40c7788d353aa626801e7413719610b25b416327576d2f7641c0a27df5: Status 404 returned error can't find the container with id 74e74c40c7788d353aa626801e7413719610b25b416327576d2f7641c0a27df5 Oct 31 04:08:52.454605 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:08:52.454573 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79b6bb9d79-gcpjq" event={"ID":"f77f79ec-3717-4a05-9d82-2b5e448ac98f","Type":"ContainerStarted","Data":"74e74c40c7788d353aa626801e7413719610b25b416327576d2f7641c0a27df5"} Oct 31 04:08:55.467019 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:08:55.466986 4376 generic.go:356] "Generic (PLEG): container finished" podID="f77f79ec-3717-4a05-9d82-2b5e448ac98f" containerID="b401ef9cd62b514166ad48969acef7ee2ad6e106e93cbcb234d0f09417fc1443" exitCode=0 Oct 31 04:08:55.467470 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:08:55.467073 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79b6bb9d79-gcpjq" event={"ID":"f77f79ec-3717-4a05-9d82-2b5e448ac98f","Type":"ContainerDied","Data":"b401ef9cd62b514166ad48969acef7ee2ad6e106e93cbcb234d0f09417fc1443"} Oct 31 04:08:56.474766 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:08:56.474720 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79b6bb9d79-gcpjq" event={"ID":"f77f79ec-3717-4a05-9d82-2b5e448ac98f","Type":"ContainerStarted","Data":"951034526f75d31d71112274139b977b998d22ad014bf3fc03817dbfc549f981"} Oct 31 04:08:56.475199 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:08:56.475044 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79b6bb9d79-gcpjq" Oct 31 04:08:56.563848 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:08:56.563788 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79b6bb9d79-gcpjq" podStartSLOduration=1.97416847 podStartE2EDuration="5.563774682s" totalImagesPullingTime="3.130603032s" totalInitContainerRuntime="459.00318ms" isStatefulPod=false podCreationTimestamp="2026-10-31 04:08:51 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:08:56.550128542 +0000 UTC m=+284.264463161" watchObservedRunningTime="2026-10-31 04:08:56.563774682 +0000 UTC m=+284.278109299" Oct 31 04:09:07.480302 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:09:07.480272 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79b6bb9d79-gcpjq" Oct 31 04:09:10.997653 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:09:10.997618 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 04:09:19.533472 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:09:19.533433 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-58666cb9df-chk6f"] Oct 31 04:09:19.539841 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:09:19.539821 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-58666cb9df-chk6f" Oct 31 04:09:19.560543 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:09:19.560513 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-58666cb9df-chk6f"] Oct 31 04:09:19.561087 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:09:19.561060 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f22948af-f85d-4360-afde-86b2a8c542d2-tmp\") pod \"kube-auth-proxy-58666cb9df-chk6f\" (UID: \"f22948af-f85d-4360-afde-86b2a8c542d2\") " pod="openshift-ingress/kube-auth-proxy-58666cb9df-chk6f" Oct 31 04:09:19.561214 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:09:19.561123 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4h9r4\" (UniqueName: \"kubernetes.io/projected/f22948af-f85d-4360-afde-86b2a8c542d2-kube-api-access-4h9r4\") pod \"kube-auth-proxy-58666cb9df-chk6f\" (UID: \"f22948af-f85d-4360-afde-86b2a8c542d2\") " pod="openshift-ingress/kube-auth-proxy-58666cb9df-chk6f" Oct 31 04:09:19.561214 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:09:19.561166 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f22948af-f85d-4360-afde-86b2a8c542d2-tls-certs\") pod \"kube-auth-proxy-58666cb9df-chk6f\" (UID: \"f22948af-f85d-4360-afde-86b2a8c542d2\") " pod="openshift-ingress/kube-auth-proxy-58666cb9df-chk6f" Oct 31 04:09:19.574361 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:09:19.573516 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Oct 31 04:09:19.574361 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:09:19.573551 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-flbxz\"" Oct 31 04:09:19.574361 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:09:19.573571 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Oct 31 04:09:19.662533 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:09:19.662496 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f22948af-f85d-4360-afde-86b2a8c542d2-tmp\") pod \"kube-auth-proxy-58666cb9df-chk6f\" (UID: \"f22948af-f85d-4360-afde-86b2a8c542d2\") " pod="openshift-ingress/kube-auth-proxy-58666cb9df-chk6f" Oct 31 04:09:19.662720 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:09:19.662551 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4h9r4\" (UniqueName: \"kubernetes.io/projected/f22948af-f85d-4360-afde-86b2a8c542d2-kube-api-access-4h9r4\") pod \"kube-auth-proxy-58666cb9df-chk6f\" (UID: \"f22948af-f85d-4360-afde-86b2a8c542d2\") " pod="openshift-ingress/kube-auth-proxy-58666cb9df-chk6f" Oct 31 04:09:19.662720 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:09:19.662582 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f22948af-f85d-4360-afde-86b2a8c542d2-tls-certs\") pod \"kube-auth-proxy-58666cb9df-chk6f\" (UID: \"f22948af-f85d-4360-afde-86b2a8c542d2\") " pod="openshift-ingress/kube-auth-proxy-58666cb9df-chk6f" Oct 31 04:09:19.664864 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:09:19.664834 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f22948af-f85d-4360-afde-86b2a8c542d2-tmp\") pod \"kube-auth-proxy-58666cb9df-chk6f\" (UID: \"f22948af-f85d-4360-afde-86b2a8c542d2\") " pod="openshift-ingress/kube-auth-proxy-58666cb9df-chk6f" Oct 31 04:09:19.665017 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:09:19.664997 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f22948af-f85d-4360-afde-86b2a8c542d2-tls-certs\") pod \"kube-auth-proxy-58666cb9df-chk6f\" (UID: \"f22948af-f85d-4360-afde-86b2a8c542d2\") " pod="openshift-ingress/kube-auth-proxy-58666cb9df-chk6f" Oct 31 04:09:19.687808 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:09:19.687786 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4h9r4\" (UniqueName: \"kubernetes.io/projected/f22948af-f85d-4360-afde-86b2a8c542d2-kube-api-access-4h9r4\") pod \"kube-auth-proxy-58666cb9df-chk6f\" (UID: \"f22948af-f85d-4360-afde-86b2a8c542d2\") " pod="openshift-ingress/kube-auth-proxy-58666cb9df-chk6f" Oct 31 04:09:19.852024 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:09:19.851945 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-58666cb9df-chk6f" Oct 31 04:09:20.146343 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:09:20.146297 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-58666cb9df-chk6f"] Oct 31 04:09:20.555385 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:09:20.555296 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-58666cb9df-chk6f" event={"ID":"f22948af-f85d-4360-afde-86b2a8c542d2","Type":"ContainerStarted","Data":"9e2db5cc63c8137d07af9a2b822bc42873a561e9a2c9e288e6294fb4fcfa488d"} Oct 31 04:09:23.571959 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:09:23.571924 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-58666cb9df-chk6f" event={"ID":"f22948af-f85d-4360-afde-86b2a8c542d2","Type":"ContainerStarted","Data":"c74685fe0cbfa4fa00fc9306f4e608fde9a9463f5ea55822bedfc489f6a790a0"} Oct 31 04:09:23.614195 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:09:23.614146 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-58666cb9df-chk6f" podStartSLOduration=2.251421446 podStartE2EDuration="4.614134268s" totalImagesPullingTime="2.362712822s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:09:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:09:23.613676172 +0000 UTC m=+311.328010809" watchObservedRunningTime="2026-10-31 04:09:23.614134268 +0000 UTC m=+311.328468882" Oct 31 04:09:27.751416 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:09:27.751321 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-mjffb"] Oct 31 04:09:27.754757 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:09:27.754741 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-mjffb" Oct 31 04:09:27.786575 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:09:27.786533 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 31 04:09:27.786793 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:09:27.786545 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-dqzgj\"" Oct 31 04:09:27.786793 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:09:27.786581 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 31 04:09:27.788222 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:09:27.788200 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-mjffb"] Oct 31 04:09:27.813056 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:09:27.813028 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\"" Oct 31 04:09:27.826645 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:09:27.826624 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c3ae653d-3b01-495e-bc64-3466a1f9f14f-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-mjffb\" (UID: \"c3ae653d-3b01-495e-bc64-3466a1f9f14f\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-mjffb" Oct 31 04:09:27.826754 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:09:27.826657 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnl66\" (UniqueName: \"kubernetes.io/projected/c3ae653d-3b01-495e-bc64-3466a1f9f14f-kube-api-access-gnl66\") pod \"kserve-module-controller-manager-7566cf5fc8-mjffb\" (UID: \"c3ae653d-3b01-495e-bc64-3466a1f9f14f\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-mjffb" Oct 31 04:09:27.826799 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:09:27.826746 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c3ae653d-3b01-495e-bc64-3466a1f9f14f-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-mjffb\" (UID: \"c3ae653d-3b01-495e-bc64-3466a1f9f14f\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-mjffb" Oct 31 04:09:27.826866 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:09:27.826849 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c3ae653d-3b01-495e-bc64-3466a1f9f14f-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-mjffb\" (UID: \"c3ae653d-3b01-495e-bc64-3466a1f9f14f\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-mjffb" Oct 31 04:09:27.927945 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:09:27.927912 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c3ae653d-3b01-495e-bc64-3466a1f9f14f-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-mjffb\" (UID: \"c3ae653d-3b01-495e-bc64-3466a1f9f14f\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-mjffb" Oct 31 04:09:27.928081 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:09:27.927962 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c3ae653d-3b01-495e-bc64-3466a1f9f14f-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-mjffb\" (UID: \"c3ae653d-3b01-495e-bc64-3466a1f9f14f\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-mjffb" Oct 31 04:09:27.928081 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:09:27.927983 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnl66\" (UniqueName: \"kubernetes.io/projected/c3ae653d-3b01-495e-bc64-3466a1f9f14f-kube-api-access-gnl66\") pod \"kserve-module-controller-manager-7566cf5fc8-mjffb\" (UID: \"c3ae653d-3b01-495e-bc64-3466a1f9f14f\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-mjffb" Oct 31 04:09:27.928081 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:09:27.928004 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c3ae653d-3b01-495e-bc64-3466a1f9f14f-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-mjffb\" (UID: \"c3ae653d-3b01-495e-bc64-3466a1f9f14f\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-mjffb" Oct 31 04:09:27.928320 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:09:27.928299 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c3ae653d-3b01-495e-bc64-3466a1f9f14f-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-mjffb\" (UID: \"c3ae653d-3b01-495e-bc64-3466a1f9f14f\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-mjffb" Oct 31 04:09:27.928320 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:09:27.928317 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c3ae653d-3b01-495e-bc64-3466a1f9f14f-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-mjffb\" (UID: \"c3ae653d-3b01-495e-bc64-3466a1f9f14f\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-mjffb" Oct 31 04:09:27.928751 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:09:27.928716 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c3ae653d-3b01-495e-bc64-3466a1f9f14f-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-mjffb\" (UID: \"c3ae653d-3b01-495e-bc64-3466a1f9f14f\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-mjffb" Oct 31 04:09:27.953539 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:09:27.953512 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnl66\" (UniqueName: \"kubernetes.io/projected/c3ae653d-3b01-495e-bc64-3466a1f9f14f-kube-api-access-gnl66\") pod \"kserve-module-controller-manager-7566cf5fc8-mjffb\" (UID: \"c3ae653d-3b01-495e-bc64-3466a1f9f14f\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-mjffb" Oct 31 04:09:28.067715 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:09:28.067645 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-mjffb" Oct 31 04:09:28.370912 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:09:28.370886 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-mjffb"] Oct 31 04:09:28.372795 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:09:28.372767 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3ae653d_3b01_495e_bc64_3466a1f9f14f.slice/crio-e4abb917c3ae062b7f7423dc916abf35e8d1946e3b4a98760417f6dc0ebd2182 WatchSource:0}: Error finding container e4abb917c3ae062b7f7423dc916abf35e8d1946e3b4a98760417f6dc0ebd2182: Status 404 returned error can't find the container with id e4abb917c3ae062b7f7423dc916abf35e8d1946e3b4a98760417f6dc0ebd2182 Oct 31 04:09:28.375222 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:09:28.375206 4376 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 04:09:28.589970 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:09:28.589934 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-mjffb" event={"ID":"c3ae653d-3b01-495e-bc64-3466a1f9f14f","Type":"ContainerStarted","Data":"e4abb917c3ae062b7f7423dc916abf35e8d1946e3b4a98760417f6dc0ebd2182"} Oct 31 04:09:30.598840 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:09:30.598802 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-mjffb" event={"ID":"c3ae653d-3b01-495e-bc64-3466a1f9f14f","Type":"ContainerStarted","Data":"34258f68cdf4e7f0b51e275c1b735289e8e698e71387b8d6d7b6b373d3b78794"} Oct 31 04:09:30.599255 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:09:30.598925 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-mjffb" Oct 31 04:09:30.658171 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:09:30.658130 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-mjffb" podStartSLOduration=2.07886321 podStartE2EDuration="3.658118422s" totalImagesPullingTime="1.579255212s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:09:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:09:30.657102462 +0000 UTC m=+318.371437099" watchObservedRunningTime="2026-10-31 04:09:30.658118422 +0000 UTC m=+318.372453053" Oct 31 04:09:40.998008 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:09:40.997965 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 04:09:51.605146 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:09:51.605112 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-mjffb" Oct 31 04:10:00.955517 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:10:00.955483 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 04:10:00.961902 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:10:00.961880 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-54z7t" Oct 31 04:10:00.976514 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:10:00.976492 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 04:10:00.988262 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:10:00.988239 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Oct 31 04:10:00.988391 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:10:00.988283 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-86txz\"" Oct 31 04:10:00.989677 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:10:00.989661 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 31 04:10:00.989729 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:10:00.989674 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 31 04:10:01.056181 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:10:01.056152 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 04:10:01.093430 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:10:01.093401 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnk5l\" (UniqueName: \"kubernetes.io/projected/d9f625f9-0628-476d-8235-9afc0a939cc1-kube-api-access-bnk5l\") pod \"limitador-limitador-7c9b87d574-54z7t\" (UID: \"d9f625f9-0628-476d-8235-9afc0a939cc1\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-54z7t" Oct 31 04:10:01.093559 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:10:01.093467 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d9f625f9-0628-476d-8235-9afc0a939cc1-config-file\") pod \"limitador-limitador-7c9b87d574-54z7t\" (UID: \"d9f625f9-0628-476d-8235-9afc0a939cc1\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-54z7t" Oct 31 04:10:01.194235 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:10:01.194205 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnk5l\" (UniqueName: \"kubernetes.io/projected/d9f625f9-0628-476d-8235-9afc0a939cc1-kube-api-access-bnk5l\") pod \"limitador-limitador-7c9b87d574-54z7t\" (UID: \"d9f625f9-0628-476d-8235-9afc0a939cc1\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-54z7t" Oct 31 04:10:01.194409 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:10:01.194273 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d9f625f9-0628-476d-8235-9afc0a939cc1-config-file\") pod \"limitador-limitador-7c9b87d574-54z7t\" (UID: \"d9f625f9-0628-476d-8235-9afc0a939cc1\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-54z7t" Oct 31 04:10:01.194906 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:10:01.194885 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d9f625f9-0628-476d-8235-9afc0a939cc1-config-file\") pod \"limitador-limitador-7c9b87d574-54z7t\" (UID: \"d9f625f9-0628-476d-8235-9afc0a939cc1\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-54z7t" Oct 31 04:10:01.218103 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:10:01.218040 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnk5l\" (UniqueName: \"kubernetes.io/projected/d9f625f9-0628-476d-8235-9afc0a939cc1-kube-api-access-bnk5l\") pod \"limitador-limitador-7c9b87d574-54z7t\" (UID: \"d9f625f9-0628-476d-8235-9afc0a939cc1\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-54z7t" Oct 31 04:10:01.275892 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:10:01.275865 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-54z7t" Oct 31 04:10:01.577228 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:10:01.577195 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 04:10:01.577633 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:10:01.577612 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9f625f9_0628_476d_8235_9afc0a939cc1.slice/crio-66d193bb446f57bcc27fb22da31a05eed63cf0c197b9c5533cbf1f73f7719ddc WatchSource:0}: Error finding container 66d193bb446f57bcc27fb22da31a05eed63cf0c197b9c5533cbf1f73f7719ddc: Status 404 returned error can't find the container with id 66d193bb446f57bcc27fb22da31a05eed63cf0c197b9c5533cbf1f73f7719ddc Oct 31 04:10:01.715131 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:10:01.715100 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-54z7t" event={"ID":"d9f625f9-0628-476d-8235-9afc0a939cc1","Type":"ContainerStarted","Data":"66d193bb446f57bcc27fb22da31a05eed63cf0c197b9c5533cbf1f73f7719ddc"} Oct 31 04:10:04.728650 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:10:04.728611 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-54z7t" event={"ID":"d9f625f9-0628-476d-8235-9afc0a939cc1","Type":"ContainerStarted","Data":"8c97ce80af78b409b600001ba1137518fd473e436386600255a6d3e42e926a5f"} Oct 31 04:10:04.729048 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:10:04.728692 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-54z7t" Oct 31 04:10:04.770720 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:10:04.770673 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-54z7t" podStartSLOduration=2.617243337 podStartE2EDuration="4.770660337s" totalImagesPullingTime="2.153417s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:10:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:10:04.769684964 +0000 UTC m=+352.484019580" watchObservedRunningTime="2026-10-31 04:10:04.770660337 +0000 UTC m=+352.484994954" Oct 31 04:10:10.998943 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:10:10.998901 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 04:10:15.732601 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:10:15.732568 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-54z7t" Oct 31 04:10:21.960569 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:10:21.960532 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7ltxr"] Oct 31 04:10:21.967236 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:10:21.967212 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7ltxr" Oct 31 04:10:21.989014 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:10:21.988987 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7ltxr"] Oct 31 04:10:22.052510 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:10:22.052470 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/df76adb1-33a3-4ddc-8192-0e1b3697626b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7ltxr\" (UID: \"df76adb1-33a3-4ddc-8192-0e1b3697626b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7ltxr" Oct 31 04:10:22.052697 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:10:22.052519 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/df76adb1-33a3-4ddc-8192-0e1b3697626b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7ltxr\" (UID: \"df76adb1-33a3-4ddc-8192-0e1b3697626b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7ltxr" Oct 31 04:10:22.052697 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:10:22.052606 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/df76adb1-33a3-4ddc-8192-0e1b3697626b-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7ltxr\" (UID: \"df76adb1-33a3-4ddc-8192-0e1b3697626b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7ltxr" Oct 31 04:10:22.052697 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:10:22.052691 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/df76adb1-33a3-4ddc-8192-0e1b3697626b-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7ltxr\" (UID: \"df76adb1-33a3-4ddc-8192-0e1b3697626b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7ltxr" Oct 31 04:10:22.052887 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:10:22.052727 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dzft\" (UniqueName: \"kubernetes.io/projected/df76adb1-33a3-4ddc-8192-0e1b3697626b-kube-api-access-8dzft\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7ltxr\" (UID: \"df76adb1-33a3-4ddc-8192-0e1b3697626b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7ltxr" Oct 31 04:10:22.052887 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:10:22.052783 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df76adb1-33a3-4ddc-8192-0e1b3697626b-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7ltxr\" (UID: \"df76adb1-33a3-4ddc-8192-0e1b3697626b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7ltxr" Oct 31 04:10:22.052887 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:10:22.052860 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/df76adb1-33a3-4ddc-8192-0e1b3697626b-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7ltxr\" (UID: \"df76adb1-33a3-4ddc-8192-0e1b3697626b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7ltxr" Oct 31 04:10:22.153312 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:10:22.153271 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/df76adb1-33a3-4ddc-8192-0e1b3697626b-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7ltxr\" (UID: \"df76adb1-33a3-4ddc-8192-0e1b3697626b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7ltxr" Oct 31 04:10:22.153312 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:10:22.153312 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dzft\" (UniqueName: \"kubernetes.io/projected/df76adb1-33a3-4ddc-8192-0e1b3697626b-kube-api-access-8dzft\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7ltxr\" (UID: \"df76adb1-33a3-4ddc-8192-0e1b3697626b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7ltxr" Oct 31 04:10:22.153580 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:10:22.153387 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df76adb1-33a3-4ddc-8192-0e1b3697626b-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7ltxr\" (UID: \"df76adb1-33a3-4ddc-8192-0e1b3697626b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7ltxr" Oct 31 04:10:22.153580 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:10:22.153411 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/df76adb1-33a3-4ddc-8192-0e1b3697626b-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7ltxr\" (UID: \"df76adb1-33a3-4ddc-8192-0e1b3697626b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7ltxr" Oct 31 04:10:22.153580 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:10:22.153542 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/df76adb1-33a3-4ddc-8192-0e1b3697626b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7ltxr\" (UID: \"df76adb1-33a3-4ddc-8192-0e1b3697626b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7ltxr" Oct 31 04:10:22.153742 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:10:22.153592 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/df76adb1-33a3-4ddc-8192-0e1b3697626b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7ltxr\" (UID: \"df76adb1-33a3-4ddc-8192-0e1b3697626b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7ltxr" Oct 31 04:10:22.153742 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:10:22.153637 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/df76adb1-33a3-4ddc-8192-0e1b3697626b-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7ltxr\" (UID: \"df76adb1-33a3-4ddc-8192-0e1b3697626b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7ltxr" Oct 31 04:10:22.154405 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:10:22.154375 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/df76adb1-33a3-4ddc-8192-0e1b3697626b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7ltxr\" (UID: \"df76adb1-33a3-4ddc-8192-0e1b3697626b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7ltxr" Oct 31 04:10:22.155899 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:10:22.155873 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/df76adb1-33a3-4ddc-8192-0e1b3697626b-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7ltxr\" (UID: \"df76adb1-33a3-4ddc-8192-0e1b3697626b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7ltxr" Oct 31 04:10:22.155899 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:10:22.155873 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/df76adb1-33a3-4ddc-8192-0e1b3697626b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7ltxr\" (UID: \"df76adb1-33a3-4ddc-8192-0e1b3697626b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7ltxr" Oct 31 04:10:22.156051 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:10:22.155905 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/df76adb1-33a3-4ddc-8192-0e1b3697626b-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7ltxr\" (UID: \"df76adb1-33a3-4ddc-8192-0e1b3697626b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7ltxr" Oct 31 04:10:22.156051 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:10:22.155908 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/df76adb1-33a3-4ddc-8192-0e1b3697626b-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7ltxr\" (UID: \"df76adb1-33a3-4ddc-8192-0e1b3697626b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7ltxr" Oct 31 04:10:22.177365 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:10:22.177313 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df76adb1-33a3-4ddc-8192-0e1b3697626b-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7ltxr\" (UID: \"df76adb1-33a3-4ddc-8192-0e1b3697626b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7ltxr" Oct 31 04:10:22.177512 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:10:22.177496 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dzft\" (UniqueName: \"kubernetes.io/projected/df76adb1-33a3-4ddc-8192-0e1b3697626b-kube-api-access-8dzft\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7ltxr\" (UID: \"df76adb1-33a3-4ddc-8192-0e1b3697626b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7ltxr" Oct 31 04:10:22.278499 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:10:22.278414 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7ltxr" Oct 31 04:10:22.607285 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:10:22.607255 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7ltxr"] Oct 31 04:10:22.607770 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:10:22.607747 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf76adb1_33a3_4ddc_8192_0e1b3697626b.slice/crio-8544c05d9ed67c79211e5e99f5cfa04de4e95ee5597c4857ec56e4f2cb640cae WatchSource:0}: Error finding container 8544c05d9ed67c79211e5e99f5cfa04de4e95ee5597c4857ec56e4f2cb640cae: Status 404 returned error can't find the container with id 8544c05d9ed67c79211e5e99f5cfa04de4e95ee5597c4857ec56e4f2cb640cae Oct 31 04:10:22.609731 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:10:22.609701 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:10:22.609804 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:10:22.609762 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:10:22.798033 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:10:22.797988 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7ltxr" event={"ID":"df76adb1-33a3-4ddc-8192-0e1b3697626b","Type":"ContainerStarted","Data":"5b09abafe980877f0e3748e645cd270331754fe03118ec19f767ba27e682afdc"} Oct 31 04:10:22.798185 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:10:22.798048 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7ltxr" event={"ID":"df76adb1-33a3-4ddc-8192-0e1b3697626b","Type":"ContainerStarted","Data":"8544c05d9ed67c79211e5e99f5cfa04de4e95ee5597c4857ec56e4f2cb640cae"} Oct 31 04:10:22.798185 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:10:22.798083 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7ltxr" Oct 31 04:10:22.852202 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:10:22.852153 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7ltxr" podStartSLOduration=1.8521397560000001 podStartE2EDuration="1.852139756s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:10:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:10:22.851614283 +0000 UTC m=+370.565948899" watchObservedRunningTime="2026-10-31 04:10:22.852139756 +0000 UTC m=+370.566474373" Oct 31 04:10:23.803714 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:10:23.803674 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7ltxr" Oct 31 04:10:23.899538 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:10:23.899507 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-bvgmx"] Oct 31 04:10:23.899734 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:10:23.899710 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-bvgmx" podUID="375f24d5-f66f-4a0f-be70-59bc85fa0352" containerName="discovery" containerID="cri-o://5000d073035f05154ea361e364716cb81b60de25091e9c70df21f229de9b1625" gracePeriod=30 Oct 31 04:10:24.146522 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:10:24.146492 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-bvgmx" Oct 31 04:10:24.178229 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:10:24.178196 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/375f24d5-f66f-4a0f-be70-59bc85fa0352-istio-token\" (UniqueName: \"kubernetes.io/projected/375f24d5-f66f-4a0f-be70-59bc85fa0352-istio-token\") pod \"375f24d5-f66f-4a0f-be70-59bc85fa0352\" (UID: \"375f24d5-f66f-4a0f-be70-59bc85fa0352\") " Oct 31 04:10:24.178409 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:10:24.178237 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/375f24d5-f66f-4a0f-be70-59bc85fa0352-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/375f24d5-f66f-4a0f-be70-59bc85fa0352-istio-csr-ca-configmap\") pod \"375f24d5-f66f-4a0f-be70-59bc85fa0352\" (UID: \"375f24d5-f66f-4a0f-be70-59bc85fa0352\") " Oct 31 04:10:24.178409 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:10:24.178264 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/375f24d5-f66f-4a0f-be70-59bc85fa0352-kube-api-access-xwmnr\" (UniqueName: \"kubernetes.io/projected/375f24d5-f66f-4a0f-be70-59bc85fa0352-kube-api-access-xwmnr\") pod \"375f24d5-f66f-4a0f-be70-59bc85fa0352\" (UID: \"375f24d5-f66f-4a0f-be70-59bc85fa0352\") " Oct 31 04:10:24.178409 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:10:24.178300 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/375f24d5-f66f-4a0f-be70-59bc85fa0352-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/375f24d5-f66f-4a0f-be70-59bc85fa0352-istio-csr-dns-cert\") pod \"375f24d5-f66f-4a0f-be70-59bc85fa0352\" (UID: \"375f24d5-f66f-4a0f-be70-59bc85fa0352\") " Oct 31 04:10:24.178409 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:10:24.178322 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/375f24d5-f66f-4a0f-be70-59bc85fa0352-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/375f24d5-f66f-4a0f-be70-59bc85fa0352-istio-kubeconfig\") pod \"375f24d5-f66f-4a0f-be70-59bc85fa0352\" (UID: \"375f24d5-f66f-4a0f-be70-59bc85fa0352\") " Oct 31 04:10:24.178409 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:10:24.178374 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/375f24d5-f66f-4a0f-be70-59bc85fa0352-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/375f24d5-f66f-4a0f-be70-59bc85fa0352-local-certs\") pod \"375f24d5-f66f-4a0f-be70-59bc85fa0352\" (UID: \"375f24d5-f66f-4a0f-be70-59bc85fa0352\") " Oct 31 04:10:24.178409 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:10:24.178398 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/375f24d5-f66f-4a0f-be70-59bc85fa0352-cacerts\" (UniqueName: \"kubernetes.io/secret/375f24d5-f66f-4a0f-be70-59bc85fa0352-cacerts\") pod \"375f24d5-f66f-4a0f-be70-59bc85fa0352\" (UID: \"375f24d5-f66f-4a0f-be70-59bc85fa0352\") " Oct 31 04:10:24.178752 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:10:24.178726 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/375f24d5-f66f-4a0f-be70-59bc85fa0352-istio-csr-ca-configmap" pod "375f24d5-f66f-4a0f-be70-59bc85fa0352" (UID: "375f24d5-f66f-4a0f-be70-59bc85fa0352"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:10:24.180800 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:10:24.180766 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/375f24d5-f66f-4a0f-be70-59bc85fa0352-local-certs" pod "375f24d5-f66f-4a0f-be70-59bc85fa0352" (UID: "375f24d5-f66f-4a0f-be70-59bc85fa0352"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:10:24.180956 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:10:24.180844 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/375f24d5-f66f-4a0f-be70-59bc85fa0352-istio-kubeconfig" pod "375f24d5-f66f-4a0f-be70-59bc85fa0352" (UID: "375f24d5-f66f-4a0f-be70-59bc85fa0352"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:10:24.180956 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:10:24.180922 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/375f24d5-f66f-4a0f-be70-59bc85fa0352-istio-csr-dns-cert" pod "375f24d5-f66f-4a0f-be70-59bc85fa0352" (UID: "375f24d5-f66f-4a0f-be70-59bc85fa0352"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:10:24.181065 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:10:24.181044 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/375f24d5-f66f-4a0f-be70-59bc85fa0352-cacerts" pod "375f24d5-f66f-4a0f-be70-59bc85fa0352" (UID: "375f24d5-f66f-4a0f-be70-59bc85fa0352"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:10:24.181065 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:10:24.181053 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/375f24d5-f66f-4a0f-be70-59bc85fa0352-kube-api-access-xwmnr" pod "375f24d5-f66f-4a0f-be70-59bc85fa0352" (UID: "375f24d5-f66f-4a0f-be70-59bc85fa0352"). InnerVolumeSpecName "kube-api-access-xwmnr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:10:24.182423 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:10:24.182401 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/375f24d5-f66f-4a0f-be70-59bc85fa0352-istio-token" pod "375f24d5-f66f-4a0f-be70-59bc85fa0352" (UID: "375f24d5-f66f-4a0f-be70-59bc85fa0352"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:10:24.279159 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:10:24.279122 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwmnr\" (UniqueName: \"kubernetes.io/projected/375f24d5-f66f-4a0f-be70-59bc85fa0352-kube-api-access-xwmnr\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:10:24.279159 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:10:24.279152 4376 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/375f24d5-f66f-4a0f-be70-59bc85fa0352-istio-csr-dns-cert\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:10:24.279159 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:10:24.279161 4376 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/375f24d5-f66f-4a0f-be70-59bc85fa0352-istio-kubeconfig\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:10:24.279159 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:10:24.279170 4376 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/375f24d5-f66f-4a0f-be70-59bc85fa0352-local-certs\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:10:24.279466 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:10:24.279178 4376 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/375f24d5-f66f-4a0f-be70-59bc85fa0352-cacerts\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:10:24.279466 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:10:24.279186 4376 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/375f24d5-f66f-4a0f-be70-59bc85fa0352-istio-token\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:10:24.279466 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:10:24.279195 4376 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/375f24d5-f66f-4a0f-be70-59bc85fa0352-istio-csr-ca-configmap\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:10:24.807745 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:10:24.807663 4376 generic.go:356] "Generic (PLEG): container finished" podID="375f24d5-f66f-4a0f-be70-59bc85fa0352" containerID="5000d073035f05154ea361e364716cb81b60de25091e9c70df21f229de9b1625" exitCode=0 Oct 31 04:10:24.807745 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:10:24.807714 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-bvgmx" Oct 31 04:10:24.808212 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:10:24.807747 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-bvgmx" event={"ID":"375f24d5-f66f-4a0f-be70-59bc85fa0352","Type":"ContainerDied","Data":"5000d073035f05154ea361e364716cb81b60de25091e9c70df21f229de9b1625"} Oct 31 04:10:24.808212 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:10:24.807797 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-bvgmx" event={"ID":"375f24d5-f66f-4a0f-be70-59bc85fa0352","Type":"ContainerDied","Data":"b56b7ed148389508bde058592ebc838dac508e30e657afb18eb9876e916a8c92"} Oct 31 04:10:24.808212 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:10:24.807817 4376 scope.go:122] "RemoveContainer" containerID="5000d073035f05154ea361e364716cb81b60de25091e9c70df21f229de9b1625" Oct 31 04:10:24.815844 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:10:24.815824 4376 scope.go:122] "RemoveContainer" containerID="5000d073035f05154ea361e364716cb81b60de25091e9c70df21f229de9b1625" Oct 31 04:10:24.816140 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:10:24.816118 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5000d073035f05154ea361e364716cb81b60de25091e9c70df21f229de9b1625\": container with ID starting with 5000d073035f05154ea361e364716cb81b60de25091e9c70df21f229de9b1625 not found: ID does not exist" containerID="5000d073035f05154ea361e364716cb81b60de25091e9c70df21f229de9b1625" Oct 31 04:10:24.816193 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:10:24.816151 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5000d073035f05154ea361e364716cb81b60de25091e9c70df21f229de9b1625"} err="failed to get container status \"5000d073035f05154ea361e364716cb81b60de25091e9c70df21f229de9b1625\": rpc error: code = NotFound desc = could not find container \"5000d073035f05154ea361e364716cb81b60de25091e9c70df21f229de9b1625\": container with ID starting with 5000d073035f05154ea361e364716cb81b60de25091e9c70df21f229de9b1625 not found: ID does not exist" Oct 31 04:10:24.863736 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:10:24.863687 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-bvgmx"] Oct 31 04:10:24.876726 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:10:24.876682 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-bvgmx"] Oct 31 04:10:26.441120 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:10:26.441080 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="375f24d5-f66f-4a0f-be70-59bc85fa0352" path="/var/lib/kubelet/pods/375f24d5-f66f-4a0f-be70-59bc85fa0352/volumes" Oct 31 04:10:40.998505 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:10:40.998456 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 04:10:46.350534 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:10:46.350497 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5ff475d6d4-2b6sq"] Oct 31 04:10:46.350922 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:10:46.350788 4376 cpu_manager.go:414] "containerMap: removing container" podUID="375f24d5-f66f-4a0f-be70-59bc85fa0352" containerName="discovery" Oct 31 04:10:46.350922 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:10:46.350798 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="375f24d5-f66f-4a0f-be70-59bc85fa0352" containerName="discovery" Oct 31 04:10:46.350922 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:10:46.350844 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="375f24d5-f66f-4a0f-be70-59bc85fa0352" containerName="discovery" Oct 31 04:10:46.357552 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:10:46.357529 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5ff475d6d4-2b6sq" Oct 31 04:10:46.377522 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:10:46.377486 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5ff475d6d4-2b6sq"] Oct 31 04:10:46.388292 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:10:46.388261 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Oct 31 04:10:46.388489 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:10:46.388369 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Oct 31 04:10:46.388489 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:10:46.388388 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Oct 31 04:10:46.388489 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:10:46.388400 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-gw86m\"" Oct 31 04:10:46.451130 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:10:46.451098 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df20597b-855c-4c85-a6f3-35defe10dd5b-metrics-certs\") pod \"maas-controller-5ff475d6d4-2b6sq\" (UID: \"df20597b-855c-4c85-a6f3-35defe10dd5b\") " pod="opendatahub/maas-controller-5ff475d6d4-2b6sq" Oct 31 04:10:46.451286 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:10:46.451142 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/df20597b-855c-4c85-a6f3-35defe10dd5b-webhook-cert\") pod \"maas-controller-5ff475d6d4-2b6sq\" (UID: \"df20597b-855c-4c85-a6f3-35defe10dd5b\") " pod="opendatahub/maas-controller-5ff475d6d4-2b6sq" Oct 31 04:10:46.451286 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:10:46.451227 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pxj2\" (UniqueName: \"kubernetes.io/projected/df20597b-855c-4c85-a6f3-35defe10dd5b-kube-api-access-5pxj2\") pod \"maas-controller-5ff475d6d4-2b6sq\" (UID: \"df20597b-855c-4c85-a6f3-35defe10dd5b\") " pod="opendatahub/maas-controller-5ff475d6d4-2b6sq" Oct 31 04:10:46.551806 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:10:46.551767 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df20597b-855c-4c85-a6f3-35defe10dd5b-metrics-certs\") pod \"maas-controller-5ff475d6d4-2b6sq\" (UID: \"df20597b-855c-4c85-a6f3-35defe10dd5b\") " pod="opendatahub/maas-controller-5ff475d6d4-2b6sq" Oct 31 04:10:46.551996 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:10:46.551841 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/df20597b-855c-4c85-a6f3-35defe10dd5b-webhook-cert\") pod \"maas-controller-5ff475d6d4-2b6sq\" (UID: \"df20597b-855c-4c85-a6f3-35defe10dd5b\") " pod="opendatahub/maas-controller-5ff475d6d4-2b6sq" Oct 31 04:10:46.551996 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:10:46.551875 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pxj2\" (UniqueName: \"kubernetes.io/projected/df20597b-855c-4c85-a6f3-35defe10dd5b-kube-api-access-5pxj2\") pod \"maas-controller-5ff475d6d4-2b6sq\" (UID: \"df20597b-855c-4c85-a6f3-35defe10dd5b\") " pod="opendatahub/maas-controller-5ff475d6d4-2b6sq" Oct 31 04:10:46.554272 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:10:46.554244 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/df20597b-855c-4c85-a6f3-35defe10dd5b-webhook-cert\") pod \"maas-controller-5ff475d6d4-2b6sq\" (UID: \"df20597b-855c-4c85-a6f3-35defe10dd5b\") " pod="opendatahub/maas-controller-5ff475d6d4-2b6sq" Oct 31 04:10:46.554399 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:10:46.554282 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df20597b-855c-4c85-a6f3-35defe10dd5b-metrics-certs\") pod \"maas-controller-5ff475d6d4-2b6sq\" (UID: \"df20597b-855c-4c85-a6f3-35defe10dd5b\") " pod="opendatahub/maas-controller-5ff475d6d4-2b6sq" Oct 31 04:10:46.575264 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:10:46.575229 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pxj2\" (UniqueName: \"kubernetes.io/projected/df20597b-855c-4c85-a6f3-35defe10dd5b-kube-api-access-5pxj2\") pod \"maas-controller-5ff475d6d4-2b6sq\" (UID: \"df20597b-855c-4c85-a6f3-35defe10dd5b\") " pod="opendatahub/maas-controller-5ff475d6d4-2b6sq" Oct 31 04:10:46.669172 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:10:46.669138 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5ff475d6d4-2b6sq" Oct 31 04:10:46.921080 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:10:46.921004 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-85c4f7957c-mh7qn"] Oct 31 04:10:46.925466 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:10:46.925444 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-85c4f7957c-mh7qn" Oct 31 04:10:46.944184 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:10:46.944157 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-85c4f7957c-mh7qn"] Oct 31 04:10:46.967513 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:10:46.967485 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5ff475d6d4-2b6sq"] Oct 31 04:10:46.968155 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:10:46.968125 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf20597b_855c_4c85_a6f3_35defe10dd5b.slice/crio-156975ee05b339e0175878cdab8515fba0e064e0455962c0698205568d46b89d WatchSource:0}: Error finding container 156975ee05b339e0175878cdab8515fba0e064e0455962c0698205568d46b89d: Status 404 returned error can't find the container with id 156975ee05b339e0175878cdab8515fba0e064e0455962c0698205568d46b89d Oct 31 04:10:47.055730 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:10:47.055696 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e27a6c1-bb5d-44e4-80ab-5617e8046bbf-metrics-certs\") pod \"maas-controller-85c4f7957c-mh7qn\" (UID: \"6e27a6c1-bb5d-44e4-80ab-5617e8046bbf\") " pod="opendatahub/maas-controller-85c4f7957c-mh7qn" Oct 31 04:10:47.055918 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:10:47.055829 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gh8bz\" (UniqueName: \"kubernetes.io/projected/6e27a6c1-bb5d-44e4-80ab-5617e8046bbf-kube-api-access-gh8bz\") pod \"maas-controller-85c4f7957c-mh7qn\" (UID: \"6e27a6c1-bb5d-44e4-80ab-5617e8046bbf\") " pod="opendatahub/maas-controller-85c4f7957c-mh7qn" Oct 31 04:10:47.055918 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:10:47.055858 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6e27a6c1-bb5d-44e4-80ab-5617e8046bbf-webhook-cert\") pod \"maas-controller-85c4f7957c-mh7qn\" (UID: \"6e27a6c1-bb5d-44e4-80ab-5617e8046bbf\") " pod="opendatahub/maas-controller-85c4f7957c-mh7qn" Oct 31 04:10:47.157216 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:10:47.157177 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e27a6c1-bb5d-44e4-80ab-5617e8046bbf-metrics-certs\") pod \"maas-controller-85c4f7957c-mh7qn\" (UID: \"6e27a6c1-bb5d-44e4-80ab-5617e8046bbf\") " pod="opendatahub/maas-controller-85c4f7957c-mh7qn" Oct 31 04:10:47.157420 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:10:47.157285 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gh8bz\" (UniqueName: \"kubernetes.io/projected/6e27a6c1-bb5d-44e4-80ab-5617e8046bbf-kube-api-access-gh8bz\") pod \"maas-controller-85c4f7957c-mh7qn\" (UID: \"6e27a6c1-bb5d-44e4-80ab-5617e8046bbf\") " pod="opendatahub/maas-controller-85c4f7957c-mh7qn" Oct 31 04:10:47.157420 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:10:47.157316 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6e27a6c1-bb5d-44e4-80ab-5617e8046bbf-webhook-cert\") pod \"maas-controller-85c4f7957c-mh7qn\" (UID: \"6e27a6c1-bb5d-44e4-80ab-5617e8046bbf\") " pod="opendatahub/maas-controller-85c4f7957c-mh7qn" Oct 31 04:10:47.159903 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:10:47.159880 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e27a6c1-bb5d-44e4-80ab-5617e8046bbf-metrics-certs\") pod \"maas-controller-85c4f7957c-mh7qn\" (UID: \"6e27a6c1-bb5d-44e4-80ab-5617e8046bbf\") " pod="opendatahub/maas-controller-85c4f7957c-mh7qn" Oct 31 04:10:47.159954 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:10:47.159880 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6e27a6c1-bb5d-44e4-80ab-5617e8046bbf-webhook-cert\") pod \"maas-controller-85c4f7957c-mh7qn\" (UID: \"6e27a6c1-bb5d-44e4-80ab-5617e8046bbf\") " pod="opendatahub/maas-controller-85c4f7957c-mh7qn" Oct 31 04:10:47.181582 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:10:47.181531 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gh8bz\" (UniqueName: \"kubernetes.io/projected/6e27a6c1-bb5d-44e4-80ab-5617e8046bbf-kube-api-access-gh8bz\") pod \"maas-controller-85c4f7957c-mh7qn\" (UID: \"6e27a6c1-bb5d-44e4-80ab-5617e8046bbf\") " pod="opendatahub/maas-controller-85c4f7957c-mh7qn" Oct 31 04:10:47.237531 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:10:47.237488 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-85c4f7957c-mh7qn" Oct 31 04:10:47.534634 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:10:47.534602 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-85c4f7957c-mh7qn"] Oct 31 04:10:47.535291 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:10:47.535262 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e27a6c1_bb5d_44e4_80ab_5617e8046bbf.slice/crio-d9d86c3f840a5497963162d48fe6e163957d2ee9e3f6af3ba9e65bf717bcc439 WatchSource:0}: Error finding container d9d86c3f840a5497963162d48fe6e163957d2ee9e3f6af3ba9e65bf717bcc439: Status 404 returned error can't find the container with id d9d86c3f840a5497963162d48fe6e163957d2ee9e3f6af3ba9e65bf717bcc439 Oct 31 04:10:47.899112 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:10:47.899064 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-85c4f7957c-mh7qn" event={"ID":"6e27a6c1-bb5d-44e4-80ab-5617e8046bbf","Type":"ContainerStarted","Data":"d9d86c3f840a5497963162d48fe6e163957d2ee9e3f6af3ba9e65bf717bcc439"} Oct 31 04:10:47.900528 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:10:47.900496 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5ff475d6d4-2b6sq" event={"ID":"df20597b-855c-4c85-a6f3-35defe10dd5b","Type":"ContainerStarted","Data":"156975ee05b339e0175878cdab8515fba0e064e0455962c0698205568d46b89d"} Oct 31 04:10:49.913146 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:10:49.913107 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5ff475d6d4-2b6sq" event={"ID":"df20597b-855c-4c85-a6f3-35defe10dd5b","Type":"ContainerStarted","Data":"53df7d402cd344ee18a5c73e5dfb86cb3833e8e055a50c834a296b902eb23a7f"} Oct 31 04:10:49.913614 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:10:49.913284 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5ff475d6d4-2b6sq" Oct 31 04:10:49.961503 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:10:49.961447 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-5ff475d6d4-2b6sq" podStartSLOduration=1.182480108 podStartE2EDuration="3.961430463s" totalImagesPullingTime="2.778950355s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:10:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:10:49.959372585 +0000 UTC m=+397.673707204" watchObservedRunningTime="2026-10-31 04:10:49.961430463 +0000 UTC m=+397.675765079" Oct 31 04:10:50.918785 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:10:50.918749 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-85c4f7957c-mh7qn" event={"ID":"6e27a6c1-bb5d-44e4-80ab-5617e8046bbf","Type":"ContainerStarted","Data":"55a11a9eb2adedcb1f5c82fa468906d6931229faae3b1a459950a1898a152131"} Oct 31 04:10:50.964022 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:10:50.963963 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-85c4f7957c-mh7qn" podStartSLOduration=2.323706818 podStartE2EDuration="4.963948709s" totalImagesPullingTime="2.640241891s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:10:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:10:50.963021218 +0000 UTC m=+398.677355835" watchObservedRunningTime="2026-10-31 04:10:50.963948709 +0000 UTC m=+398.678283356" Oct 31 04:10:51.923370 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:10:51.923320 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-85c4f7957c-mh7qn" Oct 31 04:10:51.980075 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:10:51.980043 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-zzvpx"] Oct 31 04:10:51.983451 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:10:51.983428 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-zzvpx" Oct 31 04:10:52.011633 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:10:52.011606 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\"" Oct 31 04:10:52.011821 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:10:52.011785 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\"" Oct 31 04:10:52.011821 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:10:52.011805 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-9kf7p\"" Oct 31 04:10:52.011919 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:10:52.011830 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\"" Oct 31 04:10:52.013164 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:10:52.013144 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-zzvpx"] Oct 31 04:10:52.104918 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:10:52.104876 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/c8c856e4-7a53-4077-8f5e-4faf46ec9d2a-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-zzvpx\" (UID: \"c8c856e4-7a53-4077-8f5e-4faf46ec9d2a\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-zzvpx" Oct 31 04:10:52.104918 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:10:52.104914 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c8c856e4-7a53-4077-8f5e-4faf46ec9d2a-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-zzvpx\" (UID: \"c8c856e4-7a53-4077-8f5e-4faf46ec9d2a\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-zzvpx" Oct 31 04:10:52.105166 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:10:52.104938 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xqss\" (UniqueName: \"kubernetes.io/projected/c8c856e4-7a53-4077-8f5e-4faf46ec9d2a-kube-api-access-9xqss\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-zzvpx\" (UID: \"c8c856e4-7a53-4077-8f5e-4faf46ec9d2a\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-zzvpx" Oct 31 04:10:52.105166 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:10:52.105031 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c8c856e4-7a53-4077-8f5e-4faf46ec9d2a-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-zzvpx\" (UID: \"c8c856e4-7a53-4077-8f5e-4faf46ec9d2a\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-zzvpx" Oct 31 04:10:52.205995 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:10:52.205890 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/c8c856e4-7a53-4077-8f5e-4faf46ec9d2a-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-zzvpx\" (UID: \"c8c856e4-7a53-4077-8f5e-4faf46ec9d2a\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-zzvpx" Oct 31 04:10:52.205995 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:10:52.205932 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c8c856e4-7a53-4077-8f5e-4faf46ec9d2a-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-zzvpx\" (UID: \"c8c856e4-7a53-4077-8f5e-4faf46ec9d2a\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-zzvpx" Oct 31 04:10:52.206221 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:10:52.206178 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xqss\" (UniqueName: \"kubernetes.io/projected/c8c856e4-7a53-4077-8f5e-4faf46ec9d2a-kube-api-access-9xqss\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-zzvpx\" (UID: \"c8c856e4-7a53-4077-8f5e-4faf46ec9d2a\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-zzvpx" Oct 31 04:10:52.206284 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:10:52.206235 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c8c856e4-7a53-4077-8f5e-4faf46ec9d2a-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-zzvpx\" (UID: \"c8c856e4-7a53-4077-8f5e-4faf46ec9d2a\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-zzvpx" Oct 31 04:10:52.206724 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:10:52.206697 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c8c856e4-7a53-4077-8f5e-4faf46ec9d2a-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-zzvpx\" (UID: \"c8c856e4-7a53-4077-8f5e-4faf46ec9d2a\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-zzvpx" Oct 31 04:10:52.206921 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:10:52.206901 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/c8c856e4-7a53-4077-8f5e-4faf46ec9d2a-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-zzvpx\" (UID: \"c8c856e4-7a53-4077-8f5e-4faf46ec9d2a\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-zzvpx" Oct 31 04:10:52.208661 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:10:52.208641 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c8c856e4-7a53-4077-8f5e-4faf46ec9d2a-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-zzvpx\" (UID: \"c8c856e4-7a53-4077-8f5e-4faf46ec9d2a\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-zzvpx" Oct 31 04:10:52.231767 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:10:52.231735 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xqss\" (UniqueName: \"kubernetes.io/projected/c8c856e4-7a53-4077-8f5e-4faf46ec9d2a-kube-api-access-9xqss\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-zzvpx\" (UID: \"c8c856e4-7a53-4077-8f5e-4faf46ec9d2a\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-zzvpx" Oct 31 04:10:52.295789 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:10:52.295750 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-zzvpx" Oct 31 04:10:52.590708 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:10:52.590681 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-zzvpx"] Oct 31 04:10:52.591677 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:10:52.591644 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8c856e4_7a53_4077_8f5e_4faf46ec9d2a.slice/crio-451e9af0d70a1c585fc2fd235d5a5903533bbd9db2095ba30f49c6e566dcf330 WatchSource:0}: Error finding container 451e9af0d70a1c585fc2fd235d5a5903533bbd9db2095ba30f49c6e566dcf330: Status 404 returned error can't find the container with id 451e9af0d70a1c585fc2fd235d5a5903533bbd9db2095ba30f49c6e566dcf330 Oct 31 04:10:52.929300 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:10:52.929264 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-zzvpx" event={"ID":"c8c856e4-7a53-4077-8f5e-4faf46ec9d2a","Type":"ContainerStarted","Data":"451e9af0d70a1c585fc2fd235d5a5903533bbd9db2095ba30f49c6e566dcf330"} Oct 31 04:11:00.921002 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:11:00.920970 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5ff475d6d4-2b6sq" Oct 31 04:11:02.932754 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:11:02.932710 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-85c4f7957c-mh7qn" Oct 31 04:11:03.014059 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:11:03.014019 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5ff475d6d4-2b6sq"] Oct 31 04:11:03.016156 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:11:03.014375 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-5ff475d6d4-2b6sq" podUID="df20597b-855c-4c85-a6f3-35defe10dd5b" containerName="manager" containerID="cri-o://53df7d402cd344ee18a5c73e5dfb86cb3833e8e055a50c834a296b902eb23a7f" gracePeriod=30 Oct 31 04:11:03.269357 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:11:03.269311 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5ff475d6d4-2b6sq" Oct 31 04:11:03.416944 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:11:03.416910 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df20597b-855c-4c85-a6f3-35defe10dd5b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/df20597b-855c-4c85-a6f3-35defe10dd5b-metrics-certs\") pod \"df20597b-855c-4c85-a6f3-35defe10dd5b\" (UID: \"df20597b-855c-4c85-a6f3-35defe10dd5b\") " Oct 31 04:11:03.417152 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:11:03.416968 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df20597b-855c-4c85-a6f3-35defe10dd5b-kube-api-access-5pxj2\" (UniqueName: \"kubernetes.io/projected/df20597b-855c-4c85-a6f3-35defe10dd5b-kube-api-access-5pxj2\") pod \"df20597b-855c-4c85-a6f3-35defe10dd5b\" (UID: \"df20597b-855c-4c85-a6f3-35defe10dd5b\") " Oct 31 04:11:03.417152 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:11:03.417017 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df20597b-855c-4c85-a6f3-35defe10dd5b-webhook-cert\" (UniqueName: \"kubernetes.io/secret/df20597b-855c-4c85-a6f3-35defe10dd5b-webhook-cert\") pod \"df20597b-855c-4c85-a6f3-35defe10dd5b\" (UID: \"df20597b-855c-4c85-a6f3-35defe10dd5b\") " Oct 31 04:11:03.419198 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:11:03.419165 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df20597b-855c-4c85-a6f3-35defe10dd5b-metrics-certs" pod "df20597b-855c-4c85-a6f3-35defe10dd5b" (UID: "df20597b-855c-4c85-a6f3-35defe10dd5b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:11:03.419307 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:11:03.419164 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df20597b-855c-4c85-a6f3-35defe10dd5b-kube-api-access-5pxj2" pod "df20597b-855c-4c85-a6f3-35defe10dd5b" (UID: "df20597b-855c-4c85-a6f3-35defe10dd5b"). InnerVolumeSpecName "kube-api-access-5pxj2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:11:03.419307 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:11:03.419177 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df20597b-855c-4c85-a6f3-35defe10dd5b-webhook-cert" pod "df20597b-855c-4c85-a6f3-35defe10dd5b" (UID: "df20597b-855c-4c85-a6f3-35defe10dd5b"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:11:03.517541 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:11:03.517466 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5pxj2\" (UniqueName: \"kubernetes.io/projected/df20597b-855c-4c85-a6f3-35defe10dd5b-kube-api-access-5pxj2\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:11:03.517541 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:11:03.517494 4376 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/df20597b-855c-4c85-a6f3-35defe10dd5b-webhook-cert\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:11:03.517541 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:11:03.517504 4376 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df20597b-855c-4c85-a6f3-35defe10dd5b-metrics-certs\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:11:03.982041 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:11:03.982006 4376 generic.go:356] "Generic (PLEG): container finished" podID="df20597b-855c-4c85-a6f3-35defe10dd5b" containerID="53df7d402cd344ee18a5c73e5dfb86cb3833e8e055a50c834a296b902eb23a7f" exitCode=0 Oct 31 04:11:03.982550 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:11:03.982051 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5ff475d6d4-2b6sq" Oct 31 04:11:03.982550 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:11:03.982106 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5ff475d6d4-2b6sq" event={"ID":"df20597b-855c-4c85-a6f3-35defe10dd5b","Type":"ContainerDied","Data":"53df7d402cd344ee18a5c73e5dfb86cb3833e8e055a50c834a296b902eb23a7f"} Oct 31 04:11:03.982550 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:11:03.982170 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5ff475d6d4-2b6sq" event={"ID":"df20597b-855c-4c85-a6f3-35defe10dd5b","Type":"ContainerDied","Data":"156975ee05b339e0175878cdab8515fba0e064e0455962c0698205568d46b89d"} Oct 31 04:11:03.982550 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:11:03.982205 4376 scope.go:122] "RemoveContainer" containerID="53df7d402cd344ee18a5c73e5dfb86cb3833e8e055a50c834a296b902eb23a7f" Oct 31 04:11:03.983726 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:11:03.983701 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-zzvpx" event={"ID":"c8c856e4-7a53-4077-8f5e-4faf46ec9d2a","Type":"ContainerStarted","Data":"805726161b9788a01af08da200d1b14efc0c3cb4cf7e98459e911715f26d46a1"} Oct 31 04:11:03.983850 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:11:03.983814 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-zzvpx" Oct 31 04:11:03.990833 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:11:03.990819 4376 scope.go:122] "RemoveContainer" containerID="53df7d402cd344ee18a5c73e5dfb86cb3833e8e055a50c834a296b902eb23a7f" Oct 31 04:11:03.991095 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:11:03.991069 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53df7d402cd344ee18a5c73e5dfb86cb3833e8e055a50c834a296b902eb23a7f\": container with ID starting with 53df7d402cd344ee18a5c73e5dfb86cb3833e8e055a50c834a296b902eb23a7f not found: ID does not exist" containerID="53df7d402cd344ee18a5c73e5dfb86cb3833e8e055a50c834a296b902eb23a7f" Oct 31 04:11:03.991187 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:11:03.991101 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53df7d402cd344ee18a5c73e5dfb86cb3833e8e055a50c834a296b902eb23a7f"} err="failed to get container status \"53df7d402cd344ee18a5c73e5dfb86cb3833e8e055a50c834a296b902eb23a7f\": rpc error: code = NotFound desc = could not find container \"53df7d402cd344ee18a5c73e5dfb86cb3833e8e055a50c834a296b902eb23a7f\": container with ID starting with 53df7d402cd344ee18a5c73e5dfb86cb3833e8e055a50c834a296b902eb23a7f not found: ID does not exist" Oct 31 04:11:04.034830 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:11:04.034792 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5ff475d6d4-2b6sq"] Oct 31 04:11:04.047317 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:11:04.047278 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5ff475d6d4-2b6sq"] Oct 31 04:11:04.080379 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:11:04.080312 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-zzvpx" podStartSLOduration=2.381352011 podStartE2EDuration="13.080296639s" totalImagesPullingTime="10.698944628s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:10:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:11:04.079222916 +0000 UTC m=+411.793557533" watchObservedRunningTime="2026-10-31 04:11:04.080296639 +0000 UTC m=+411.794631255" Oct 31 04:11:04.440666 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:11:04.440635 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df20597b-855c-4c85-a6f3-35defe10dd5b" path="/var/lib/kubelet/pods/df20597b-855c-4c85-a6f3-35defe10dd5b/volumes" Oct 31 04:11:10.997564 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:11:10.997529 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 04:11:14.994787 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:11:14.994756 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-zzvpx" Oct 31 04:11:25.900653 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:11:25.900612 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 04:11:35.090033 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:11:35.089989 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-b94596b94-xgfr2"] Oct 31 04:11:35.090443 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:11:35.090285 4376 cpu_manager.go:414] "containerMap: removing container" podUID="df20597b-855c-4c85-a6f3-35defe10dd5b" containerName="manager" Oct 31 04:11:35.090443 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:11:35.090296 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df20597b-855c-4c85-a6f3-35defe10dd5b" containerName="manager" Oct 31 04:11:35.090443 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:11:35.090390 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="df20597b-855c-4c85-a6f3-35defe10dd5b" containerName="manager" Oct 31 04:11:35.094598 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:11:35.094579 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-b94596b94-xgfr2" Oct 31 04:11:35.124632 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:11:35.124605 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-b94596b94-xgfr2"] Oct 31 04:11:35.135674 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:11:35.135647 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 04:11:35.135674 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:11:35.135666 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 04:11:35.170545 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:11:35.170512 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/047e9b6c-64dc-4c2f-b1a9-fc21e50df6b5-metrics-certs\") pod \"maas-api-b94596b94-xgfr2\" (UID: \"047e9b6c-64dc-4c2f-b1a9-fc21e50df6b5\") " pod="odh-ai-gateway-infra/maas-api-b94596b94-xgfr2" Oct 31 04:11:35.170699 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:11:35.170559 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/047e9b6c-64dc-4c2f-b1a9-fc21e50df6b5-maas-api-tls\") pod \"maas-api-b94596b94-xgfr2\" (UID: \"047e9b6c-64dc-4c2f-b1a9-fc21e50df6b5\") " pod="odh-ai-gateway-infra/maas-api-b94596b94-xgfr2" Oct 31 04:11:35.170699 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:11:35.170638 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5r9l\" (UniqueName: \"kubernetes.io/projected/047e9b6c-64dc-4c2f-b1a9-fc21e50df6b5-kube-api-access-w5r9l\") pod \"maas-api-b94596b94-xgfr2\" (UID: \"047e9b6c-64dc-4c2f-b1a9-fc21e50df6b5\") " pod="odh-ai-gateway-infra/maas-api-b94596b94-xgfr2" Oct 31 04:11:35.170770 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:11:35.170714 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/047e9b6c-64dc-4c2f-b1a9-fc21e50df6b5-openshift-service-ca\") pod \"maas-api-b94596b94-xgfr2\" (UID: \"047e9b6c-64dc-4c2f-b1a9-fc21e50df6b5\") " pod="odh-ai-gateway-infra/maas-api-b94596b94-xgfr2" Oct 31 04:11:35.178770 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:11:35.178750 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-qt6xg\"" Oct 31 04:11:35.178883 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:11:35.178787 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 31 04:11:35.178883 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:11:35.178793 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Oct 31 04:11:35.271474 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:11:35.271436 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/047e9b6c-64dc-4c2f-b1a9-fc21e50df6b5-openshift-service-ca\") pod \"maas-api-b94596b94-xgfr2\" (UID: \"047e9b6c-64dc-4c2f-b1a9-fc21e50df6b5\") " pod="odh-ai-gateway-infra/maas-api-b94596b94-xgfr2" Oct 31 04:11:35.271655 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:11:35.271487 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/047e9b6c-64dc-4c2f-b1a9-fc21e50df6b5-metrics-certs\") pod \"maas-api-b94596b94-xgfr2\" (UID: \"047e9b6c-64dc-4c2f-b1a9-fc21e50df6b5\") " pod="odh-ai-gateway-infra/maas-api-b94596b94-xgfr2" Oct 31 04:11:35.271655 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:11:35.271519 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/047e9b6c-64dc-4c2f-b1a9-fc21e50df6b5-maas-api-tls\") pod \"maas-api-b94596b94-xgfr2\" (UID: \"047e9b6c-64dc-4c2f-b1a9-fc21e50df6b5\") " pod="odh-ai-gateway-infra/maas-api-b94596b94-xgfr2" Oct 31 04:11:35.271655 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:11:35.271538 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5r9l\" (UniqueName: \"kubernetes.io/projected/047e9b6c-64dc-4c2f-b1a9-fc21e50df6b5-kube-api-access-w5r9l\") pod \"maas-api-b94596b94-xgfr2\" (UID: \"047e9b6c-64dc-4c2f-b1a9-fc21e50df6b5\") " pod="odh-ai-gateway-infra/maas-api-b94596b94-xgfr2" Oct 31 04:11:35.272235 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:11:35.272206 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/047e9b6c-64dc-4c2f-b1a9-fc21e50df6b5-openshift-service-ca\") pod \"maas-api-b94596b94-xgfr2\" (UID: \"047e9b6c-64dc-4c2f-b1a9-fc21e50df6b5\") " pod="odh-ai-gateway-infra/maas-api-b94596b94-xgfr2" Oct 31 04:11:35.273974 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:11:35.273951 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/047e9b6c-64dc-4c2f-b1a9-fc21e50df6b5-maas-api-tls\") pod \"maas-api-b94596b94-xgfr2\" (UID: \"047e9b6c-64dc-4c2f-b1a9-fc21e50df6b5\") " pod="odh-ai-gateway-infra/maas-api-b94596b94-xgfr2" Oct 31 04:11:35.274070 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:11:35.274007 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/047e9b6c-64dc-4c2f-b1a9-fc21e50df6b5-metrics-certs\") pod \"maas-api-b94596b94-xgfr2\" (UID: \"047e9b6c-64dc-4c2f-b1a9-fc21e50df6b5\") " pod="odh-ai-gateway-infra/maas-api-b94596b94-xgfr2" Oct 31 04:11:35.301619 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:11:35.301583 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5r9l\" (UniqueName: \"kubernetes.io/projected/047e9b6c-64dc-4c2f-b1a9-fc21e50df6b5-kube-api-access-w5r9l\") pod \"maas-api-b94596b94-xgfr2\" (UID: \"047e9b6c-64dc-4c2f-b1a9-fc21e50df6b5\") " pod="odh-ai-gateway-infra/maas-api-b94596b94-xgfr2" Oct 31 04:11:35.406890 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:11:35.406837 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-b94596b94-xgfr2" Oct 31 04:11:35.701728 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:11:35.701699 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-b94596b94-xgfr2"] Oct 31 04:11:35.702607 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:11:35.702580 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod047e9b6c_64dc_4c2f_b1a9_fc21e50df6b5.slice/crio-041698f2bc9d485a6e0d42b1dab40809fa6017858809a4d0cf6bef8b113013af WatchSource:0}: Error finding container 041698f2bc9d485a6e0d42b1dab40809fa6017858809a4d0cf6bef8b113013af: Status 404 returned error can't find the container with id 041698f2bc9d485a6e0d42b1dab40809fa6017858809a4d0cf6bef8b113013af Oct 31 04:11:36.116565 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:11:36.116520 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-b94596b94-xgfr2" event={"ID":"047e9b6c-64dc-4c2f-b1a9-fc21e50df6b5","Type":"ContainerStarted","Data":"041698f2bc9d485a6e0d42b1dab40809fa6017858809a4d0cf6bef8b113013af"} Oct 31 04:11:38.127395 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:11:38.127349 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-b94596b94-xgfr2" event={"ID":"047e9b6c-64dc-4c2f-b1a9-fc21e50df6b5","Type":"ContainerStarted","Data":"cfa18122b28717dc9adec7c976e041e1bd14b72e444248031c5b4667381d6d91"} Oct 31 04:11:38.127793 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:11:38.127479 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-b94596b94-xgfr2" Oct 31 04:11:38.170917 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:11:38.170863 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-b94596b94-xgfr2" podStartSLOduration=1.769468727 podStartE2EDuration="3.170851241s" totalImagesPullingTime="1.401382514s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:11:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:11:38.170354425 +0000 UTC m=+445.884689034" watchObservedRunningTime="2026-10-31 04:11:38.170851241 +0000 UTC m=+445.885185858" Oct 31 04:11:40.998429 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:11:40.998392 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 04:11:44.136827 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:11:44.136792 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-b94596b94-xgfr2" Oct 31 04:12:10.998430 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:12:10.998395 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 04:12:13.834856 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:12:13.834819 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-mqjnw"] Oct 31 04:12:13.838737 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:12:13.838719 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-mqjnw" Oct 31 04:12:13.867774 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:12:13.867748 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\"" Oct 31 04:12:13.867950 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:12:13.867793 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-x8t52\"" Oct 31 04:12:13.867950 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:12:13.867816 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\"" Oct 31 04:12:13.869582 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:12:13.869497 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-mqjnw"] Oct 31 04:12:13.981533 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:12:13.981497 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da8f093b-cea4-45a5-8ec5-160dae0db636-metrics-certs\") pod \"ai-gateway-controller-7d85675f98-mqjnw\" (UID: \"da8f093b-cea4-45a5-8ec5-160dae0db636\") " pod="opendatahub/ai-gateway-controller-7d85675f98-mqjnw" Oct 31 04:12:13.981716 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:12:13.981560 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69sgw\" (UniqueName: \"kubernetes.io/projected/da8f093b-cea4-45a5-8ec5-160dae0db636-kube-api-access-69sgw\") pod \"ai-gateway-controller-7d85675f98-mqjnw\" (UID: \"da8f093b-cea4-45a5-8ec5-160dae0db636\") " pod="opendatahub/ai-gateway-controller-7d85675f98-mqjnw" Oct 31 04:12:14.082201 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:12:14.082165 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69sgw\" (UniqueName: \"kubernetes.io/projected/da8f093b-cea4-45a5-8ec5-160dae0db636-kube-api-access-69sgw\") pod \"ai-gateway-controller-7d85675f98-mqjnw\" (UID: \"da8f093b-cea4-45a5-8ec5-160dae0db636\") " pod="opendatahub/ai-gateway-controller-7d85675f98-mqjnw" Oct 31 04:12:14.082393 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:12:14.082231 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da8f093b-cea4-45a5-8ec5-160dae0db636-metrics-certs\") pod \"ai-gateway-controller-7d85675f98-mqjnw\" (UID: \"da8f093b-cea4-45a5-8ec5-160dae0db636\") " pod="opendatahub/ai-gateway-controller-7d85675f98-mqjnw" Oct 31 04:12:14.084639 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:12:14.084614 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da8f093b-cea4-45a5-8ec5-160dae0db636-metrics-certs\") pod \"ai-gateway-controller-7d85675f98-mqjnw\" (UID: \"da8f093b-cea4-45a5-8ec5-160dae0db636\") " pod="opendatahub/ai-gateway-controller-7d85675f98-mqjnw" Oct 31 04:12:14.106166 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:12:14.106104 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69sgw\" (UniqueName: \"kubernetes.io/projected/da8f093b-cea4-45a5-8ec5-160dae0db636-kube-api-access-69sgw\") pod \"ai-gateway-controller-7d85675f98-mqjnw\" (UID: \"da8f093b-cea4-45a5-8ec5-160dae0db636\") " pod="opendatahub/ai-gateway-controller-7d85675f98-mqjnw" Oct 31 04:12:14.150696 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:12:14.150667 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-mqjnw" Oct 31 04:12:14.448354 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:12:14.448309 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda8f093b_cea4_45a5_8ec5_160dae0db636.slice/crio-3e03a64d2720a071ef8c69ca46125501fdb2907d03fad3e8b553da60fb20b5d0 WatchSource:0}: Error finding container 3e03a64d2720a071ef8c69ca46125501fdb2907d03fad3e8b553da60fb20b5d0: Status 404 returned error can't find the container with id 3e03a64d2720a071ef8c69ca46125501fdb2907d03fad3e8b553da60fb20b5d0 Oct 31 04:12:14.449032 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:12:14.449005 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-mqjnw"] Oct 31 04:12:15.269112 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:12:15.269076 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-mqjnw" event={"ID":"da8f093b-cea4-45a5-8ec5-160dae0db636","Type":"ContainerStarted","Data":"3e03a64d2720a071ef8c69ca46125501fdb2907d03fad3e8b553da60fb20b5d0"} Oct 31 04:12:16.274899 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:12:16.274814 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-mqjnw" event={"ID":"da8f093b-cea4-45a5-8ec5-160dae0db636","Type":"ContainerStarted","Data":"673a09022c304323b90cb5bfeec39fcaf8153ea955b0e46264bb64176249fc84"} Oct 31 04:12:16.275257 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:12:16.274942 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-7d85675f98-mqjnw" Oct 31 04:12:16.317160 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:12:16.317111 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-7d85675f98-mqjnw" podStartSLOduration=1.79197622 podStartE2EDuration="3.317098821s" totalImagesPullingTime="1.525122601s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:12:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:12:16.316240175 +0000 UTC m=+484.030574789" watchObservedRunningTime="2026-10-31 04:12:16.317098821 +0000 UTC m=+484.031433448" Oct 31 04:12:18.290263 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:12:18.290230 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-n2mk7"] Oct 31 04:12:18.293539 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:12:18.293521 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-n2mk7" Oct 31 04:12:18.311611 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:12:18.311584 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9700b83d-909b-46b4-98d4-c01c16aed865-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-n2mk7\" (UID: \"9700b83d-909b-46b4-98d4-c01c16aed865\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-n2mk7" Oct 31 04:12:18.311754 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:12:18.311705 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtgwc\" (UniqueName: \"kubernetes.io/projected/9700b83d-909b-46b4-98d4-c01c16aed865-kube-api-access-wtgwc\") pod \"payload-pre-processing-7b98bb678c-n2mk7\" (UID: \"9700b83d-909b-46b4-98d4-c01c16aed865\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-n2mk7" Oct 31 04:12:18.323044 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:12:18.323014 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Oct 31 04:12:18.323170 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:12:18.323064 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-sw4pr\"" Oct 31 04:12:18.325109 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:12:18.325085 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-n2mk7"] Oct 31 04:12:18.387914 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:12:18.387884 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-cpb5g"] Oct 31 04:12:18.391148 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:12:18.391131 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-cpb5g" Oct 31 04:12:18.410611 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:12:18.410582 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-cpb5g"] Oct 31 04:12:18.412119 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:12:18.412091 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9700b83d-909b-46b4-98d4-c01c16aed865-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-n2mk7\" (UID: \"9700b83d-909b-46b4-98d4-c01c16aed865\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-n2mk7" Oct 31 04:12:18.412225 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:12:18.412156 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83cfbbfb-a15c-4967-b20c-e102351c6b7b-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-cpb5g\" (UID: \"83cfbbfb-a15c-4967-b20c-e102351c6b7b\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-cpb5g" Oct 31 04:12:18.412225 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:12:18.412199 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkngk\" (UniqueName: \"kubernetes.io/projected/83cfbbfb-a15c-4967-b20c-e102351c6b7b-kube-api-access-kkngk\") pod \"payload-processing-55fbbbf9dc-cpb5g\" (UID: \"83cfbbfb-a15c-4967-b20c-e102351c6b7b\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-cpb5g" Oct 31 04:12:18.412321 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:12:18.412230 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtgwc\" (UniqueName: \"kubernetes.io/projected/9700b83d-909b-46b4-98d4-c01c16aed865-kube-api-access-wtgwc\") pod \"payload-pre-processing-7b98bb678c-n2mk7\" (UID: \"9700b83d-909b-46b4-98d4-c01c16aed865\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-n2mk7" Oct 31 04:12:18.412890 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:12:18.412856 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9700b83d-909b-46b4-98d4-c01c16aed865-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-n2mk7\" (UID: \"9700b83d-909b-46b4-98d4-c01c16aed865\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-n2mk7" Oct 31 04:12:18.437029 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:12:18.437001 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtgwc\" (UniqueName: \"kubernetes.io/projected/9700b83d-909b-46b4-98d4-c01c16aed865-kube-api-access-wtgwc\") pod \"payload-pre-processing-7b98bb678c-n2mk7\" (UID: \"9700b83d-909b-46b4-98d4-c01c16aed865\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-n2mk7" Oct 31 04:12:18.513138 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:12:18.513095 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83cfbbfb-a15c-4967-b20c-e102351c6b7b-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-cpb5g\" (UID: \"83cfbbfb-a15c-4967-b20c-e102351c6b7b\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-cpb5g" Oct 31 04:12:18.513321 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:12:18.513230 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkngk\" (UniqueName: \"kubernetes.io/projected/83cfbbfb-a15c-4967-b20c-e102351c6b7b-kube-api-access-kkngk\") pod \"payload-processing-55fbbbf9dc-cpb5g\" (UID: \"83cfbbfb-a15c-4967-b20c-e102351c6b7b\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-cpb5g" Oct 31 04:12:18.513841 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:12:18.513816 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83cfbbfb-a15c-4967-b20c-e102351c6b7b-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-cpb5g\" (UID: \"83cfbbfb-a15c-4967-b20c-e102351c6b7b\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-cpb5g" Oct 31 04:12:18.539087 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:12:18.539057 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkngk\" (UniqueName: \"kubernetes.io/projected/83cfbbfb-a15c-4967-b20c-e102351c6b7b-kube-api-access-kkngk\") pod \"payload-processing-55fbbbf9dc-cpb5g\" (UID: \"83cfbbfb-a15c-4967-b20c-e102351c6b7b\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-cpb5g" Oct 31 04:12:18.605435 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:12:18.605352 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-n2mk7" Oct 31 04:12:18.703645 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:12:18.703614 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-cpb5g" Oct 31 04:12:18.897462 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:12:18.897435 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-n2mk7"] Oct 31 04:12:18.897963 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:12:18.897937 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9700b83d_909b_46b4_98d4_c01c16aed865.slice/crio-fc0d20a3fd68873f8697cb6fd431fd01ed647c035666155f4adfc6d9a6a3e4fc WatchSource:0}: Error finding container fc0d20a3fd68873f8697cb6fd431fd01ed647c035666155f4adfc6d9a6a3e4fc: Status 404 returned error can't find the container with id fc0d20a3fd68873f8697cb6fd431fd01ed647c035666155f4adfc6d9a6a3e4fc Oct 31 04:12:18.997535 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:12:18.997506 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-cpb5g"] Oct 31 04:12:18.997918 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:12:18.997896 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83cfbbfb_a15c_4967_b20c_e102351c6b7b.slice/crio-c961467db3ca252f01f1dfbe317bfa3f99bc5fec6ab5939c9729e1d0d0884e32 WatchSource:0}: Error finding container c961467db3ca252f01f1dfbe317bfa3f99bc5fec6ab5939c9729e1d0d0884e32: Status 404 returned error can't find the container with id c961467db3ca252f01f1dfbe317bfa3f99bc5fec6ab5939c9729e1d0d0884e32 Oct 31 04:12:19.288125 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:12:19.288044 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-n2mk7" event={"ID":"9700b83d-909b-46b4-98d4-c01c16aed865","Type":"ContainerStarted","Data":"fc0d20a3fd68873f8697cb6fd431fd01ed647c035666155f4adfc6d9a6a3e4fc"} Oct 31 04:12:19.289268 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:12:19.289237 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-cpb5g" event={"ID":"83cfbbfb-a15c-4967-b20c-e102351c6b7b","Type":"ContainerStarted","Data":"c961467db3ca252f01f1dfbe317bfa3f99bc5fec6ab5939c9729e1d0d0884e32"} Oct 31 04:12:21.299826 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:12:21.299796 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-n2mk7" event={"ID":"9700b83d-909b-46b4-98d4-c01c16aed865","Type":"ContainerStarted","Data":"80762b253d0de69f03f21af7886ecccde022c45bfcf55ea10854983986c8581c"} Oct 31 04:12:21.300265 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:12:21.299985 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-n2mk7" Oct 31 04:12:21.343943 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:12:21.343890 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-n2mk7" podStartSLOduration=1.049718856 podStartE2EDuration="3.343874624s" totalImagesPullingTime="2.294155768s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:12:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:12:21.342624815 +0000 UTC m=+489.056959452" watchObservedRunningTime="2026-10-31 04:12:21.343874624 +0000 UTC m=+489.058209241" Oct 31 04:12:22.304822 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:12:22.304786 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-cpb5g" event={"ID":"83cfbbfb-a15c-4967-b20c-e102351c6b7b","Type":"ContainerStarted","Data":"b577ebd21405364322a9ee69cce9b1e2fbe1ccd28f3dea8f3b5823010c860e76"} Oct 31 04:12:22.305227 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:12:22.304873 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-cpb5g" Oct 31 04:12:22.346634 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:12:22.346583 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-55fbbbf9dc-cpb5g" podStartSLOduration=2.110020731 podStartE2EDuration="4.346570857s" totalImagesPullingTime="2.236550126s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:12:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:12:22.345855184 +0000 UTC m=+490.060189802" watchObservedRunningTime="2026-10-31 04:12:22.346570857 +0000 UTC m=+490.060905473" Oct 31 04:12:27.281311 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:12:27.281230 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-7d85675f98-mqjnw" Oct 31 04:12:32.308485 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:12:32.308453 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-n2mk7" Oct 31 04:12:33.311512 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:12:33.311486 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-cpb5g" Oct 31 04:12:40.998491 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:12:40.998449 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 04:12:42.077514 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:12:42.077481 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-mqjnw"] Oct 31 04:12:42.077960 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:12:42.077673 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-7d85675f98-mqjnw" podUID="da8f093b-cea4-45a5-8ec5-160dae0db636" containerName="manager" containerID="cri-o://673a09022c304323b90cb5bfeec39fcaf8153ea955b0e46264bb64176249fc84" gracePeriod=30 Oct 31 04:12:42.325491 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:12:42.325462 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-mqjnw" Oct 31 04:12:42.388144 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:12:42.388110 4376 generic.go:356] "Generic (PLEG): container finished" podID="da8f093b-cea4-45a5-8ec5-160dae0db636" containerID="673a09022c304323b90cb5bfeec39fcaf8153ea955b0e46264bb64176249fc84" exitCode=0 Oct 31 04:12:42.388316 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:12:42.388169 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-mqjnw" Oct 31 04:12:42.388316 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:12:42.388184 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-mqjnw" event={"ID":"da8f093b-cea4-45a5-8ec5-160dae0db636","Type":"ContainerDied","Data":"673a09022c304323b90cb5bfeec39fcaf8153ea955b0e46264bb64176249fc84"} Oct 31 04:12:42.388316 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:12:42.388269 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-mqjnw" event={"ID":"da8f093b-cea4-45a5-8ec5-160dae0db636","Type":"ContainerDied","Data":"3e03a64d2720a071ef8c69ca46125501fdb2907d03fad3e8b553da60fb20b5d0"} Oct 31 04:12:42.388316 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:12:42.388292 4376 scope.go:122] "RemoveContainer" containerID="673a09022c304323b90cb5bfeec39fcaf8153ea955b0e46264bb64176249fc84" Oct 31 04:12:42.397011 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:12:42.396995 4376 scope.go:122] "RemoveContainer" containerID="673a09022c304323b90cb5bfeec39fcaf8153ea955b0e46264bb64176249fc84" Oct 31 04:12:42.397284 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:12:42.397267 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"673a09022c304323b90cb5bfeec39fcaf8153ea955b0e46264bb64176249fc84\": container with ID starting with 673a09022c304323b90cb5bfeec39fcaf8153ea955b0e46264bb64176249fc84 not found: ID does not exist" containerID="673a09022c304323b90cb5bfeec39fcaf8153ea955b0e46264bb64176249fc84" Oct 31 04:12:42.397349 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:12:42.397294 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"673a09022c304323b90cb5bfeec39fcaf8153ea955b0e46264bb64176249fc84"} err="failed to get container status \"673a09022c304323b90cb5bfeec39fcaf8153ea955b0e46264bb64176249fc84\": rpc error: code = NotFound desc = could not find container \"673a09022c304323b90cb5bfeec39fcaf8153ea955b0e46264bb64176249fc84\": container with ID starting with 673a09022c304323b90cb5bfeec39fcaf8153ea955b0e46264bb64176249fc84 not found: ID does not exist" Oct 31 04:12:42.418733 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:12:42.418692 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da8f093b-cea4-45a5-8ec5-160dae0db636-metrics-certs\" (UniqueName: \"kubernetes.io/secret/da8f093b-cea4-45a5-8ec5-160dae0db636-metrics-certs\") pod \"da8f093b-cea4-45a5-8ec5-160dae0db636\" (UID: \"da8f093b-cea4-45a5-8ec5-160dae0db636\") " Oct 31 04:12:42.418733 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:12:42.418738 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da8f093b-cea4-45a5-8ec5-160dae0db636-kube-api-access-69sgw\" (UniqueName: \"kubernetes.io/projected/da8f093b-cea4-45a5-8ec5-160dae0db636-kube-api-access-69sgw\") pod \"da8f093b-cea4-45a5-8ec5-160dae0db636\" (UID: \"da8f093b-cea4-45a5-8ec5-160dae0db636\") " Oct 31 04:12:42.420784 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:12:42.420753 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da8f093b-cea4-45a5-8ec5-160dae0db636-metrics-certs" pod "da8f093b-cea4-45a5-8ec5-160dae0db636" (UID: "da8f093b-cea4-45a5-8ec5-160dae0db636"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:12:42.420896 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:12:42.420873 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da8f093b-cea4-45a5-8ec5-160dae0db636-kube-api-access-69sgw" pod "da8f093b-cea4-45a5-8ec5-160dae0db636" (UID: "da8f093b-cea4-45a5-8ec5-160dae0db636"). InnerVolumeSpecName "kube-api-access-69sgw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:12:42.519962 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:12:42.519925 4376 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da8f093b-cea4-45a5-8ec5-160dae0db636-metrics-certs\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:12:42.519962 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:12:42.519953 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69sgw\" (UniqueName: \"kubernetes.io/projected/da8f093b-cea4-45a5-8ec5-160dae0db636-kube-api-access-69sgw\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:12:42.727716 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:12:42.727622 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-mqjnw"] Oct 31 04:12:42.731195 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:12:42.731171 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-mqjnw"] Oct 31 04:12:44.440930 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:12:44.440896 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da8f093b-cea4-45a5-8ec5-160dae0db636" path="/var/lib/kubelet/pods/da8f093b-cea4-45a5-8ec5-160dae0db636/volumes" Oct 31 04:12:50.095244 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:12:50.095206 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-phl42"] Oct 31 04:12:50.095770 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:12:50.095583 4376 cpu_manager.go:414] "containerMap: removing container" podUID="da8f093b-cea4-45a5-8ec5-160dae0db636" containerName="manager" Oct 31 04:12:50.095770 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:12:50.095596 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da8f093b-cea4-45a5-8ec5-160dae0db636" containerName="manager" Oct 31 04:12:50.095770 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:12:50.095657 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="da8f093b-cea4-45a5-8ec5-160dae0db636" containerName="manager" Oct 31 04:12:50.101309 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:12:50.101285 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-phl42" Oct 31 04:12:50.117908 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:12:50.117877 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-phl42"] Oct 31 04:12:50.119630 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:12:50.119599 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Oct 31 04:12:50.129706 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:12:50.129677 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-wcn6k\"" Oct 31 04:12:50.129864 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:12:50.129723 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Oct 31 04:12:50.129864 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:12:50.129809 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Oct 31 04:12:50.185159 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:12:50.185124 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrqcl\" (UniqueName: \"kubernetes.io/projected/3d152b49-e42e-4584-a7ee-7b540322fd96-kube-api-access-nrqcl\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-phl42\" (UID: \"3d152b49-e42e-4584-a7ee-7b540322fd96\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-phl42" Oct 31 04:12:50.185159 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:12:50.185166 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d152b49-e42e-4584-a7ee-7b540322fd96-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-phl42\" (UID: \"3d152b49-e42e-4584-a7ee-7b540322fd96\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-phl42" Oct 31 04:12:50.185390 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:12:50.185196 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d152b49-e42e-4584-a7ee-7b540322fd96-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-phl42\" (UID: \"3d152b49-e42e-4584-a7ee-7b540322fd96\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-phl42" Oct 31 04:12:50.185390 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:12:50.185314 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d152b49-e42e-4584-a7ee-7b540322fd96-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-phl42\" (UID: \"3d152b49-e42e-4584-a7ee-7b540322fd96\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-phl42" Oct 31 04:12:50.185466 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:12:50.185397 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d152b49-e42e-4584-a7ee-7b540322fd96-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-phl42\" (UID: \"3d152b49-e42e-4584-a7ee-7b540322fd96\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-phl42" Oct 31 04:12:50.185466 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:12:50.185414 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d152b49-e42e-4584-a7ee-7b540322fd96-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-phl42\" (UID: \"3d152b49-e42e-4584-a7ee-7b540322fd96\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-phl42" Oct 31 04:12:50.286740 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:12:50.286698 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d152b49-e42e-4584-a7ee-7b540322fd96-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-phl42\" (UID: \"3d152b49-e42e-4584-a7ee-7b540322fd96\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-phl42" Oct 31 04:12:50.286740 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:12:50.286749 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d152b49-e42e-4584-a7ee-7b540322fd96-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-phl42\" (UID: \"3d152b49-e42e-4584-a7ee-7b540322fd96\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-phl42" Oct 31 04:12:50.287007 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:12:50.286781 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d152b49-e42e-4584-a7ee-7b540322fd96-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-phl42\" (UID: \"3d152b49-e42e-4584-a7ee-7b540322fd96\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-phl42" Oct 31 04:12:50.287007 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:12:50.286843 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d152b49-e42e-4584-a7ee-7b540322fd96-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-phl42\" (UID: \"3d152b49-e42e-4584-a7ee-7b540322fd96\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-phl42" Oct 31 04:12:50.287007 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:12:50.286867 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d152b49-e42e-4584-a7ee-7b540322fd96-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-phl42\" (UID: \"3d152b49-e42e-4584-a7ee-7b540322fd96\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-phl42" Oct 31 04:12:50.287007 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:12:50.286899 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrqcl\" (UniqueName: \"kubernetes.io/projected/3d152b49-e42e-4584-a7ee-7b540322fd96-kube-api-access-nrqcl\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-phl42\" (UID: \"3d152b49-e42e-4584-a7ee-7b540322fd96\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-phl42" Oct 31 04:12:50.287271 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:12:50.287229 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d152b49-e42e-4584-a7ee-7b540322fd96-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-phl42\" (UID: \"3d152b49-e42e-4584-a7ee-7b540322fd96\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-phl42" Oct 31 04:12:50.287419 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:12:50.287278 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d152b49-e42e-4584-a7ee-7b540322fd96-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-phl42\" (UID: \"3d152b49-e42e-4584-a7ee-7b540322fd96\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-phl42" Oct 31 04:12:50.287419 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:12:50.287352 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d152b49-e42e-4584-a7ee-7b540322fd96-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-phl42\" (UID: \"3d152b49-e42e-4584-a7ee-7b540322fd96\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-phl42" Oct 31 04:12:50.289096 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:12:50.289074 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d152b49-e42e-4584-a7ee-7b540322fd96-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-phl42\" (UID: \"3d152b49-e42e-4584-a7ee-7b540322fd96\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-phl42" Oct 31 04:12:50.289615 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:12:50.289598 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d152b49-e42e-4584-a7ee-7b540322fd96-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-phl42\" (UID: \"3d152b49-e42e-4584-a7ee-7b540322fd96\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-phl42" Oct 31 04:12:50.311973 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:12:50.311945 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrqcl\" (UniqueName: \"kubernetes.io/projected/3d152b49-e42e-4584-a7ee-7b540322fd96-kube-api-access-nrqcl\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-phl42\" (UID: \"3d152b49-e42e-4584-a7ee-7b540322fd96\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-phl42" Oct 31 04:12:50.414948 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:12:50.414911 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-phl42" Oct 31 04:12:50.719342 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:12:50.719300 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-phl42"] Oct 31 04:12:50.720231 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:12:50.720201 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d152b49_e42e_4584_a7ee_7b540322fd96.slice/crio-bfd366b4c5f2935b00e350c404bd763612686cf422edd6fcb20ca01aeeca5dff WatchSource:0}: Error finding container bfd366b4c5f2935b00e350c404bd763612686cf422edd6fcb20ca01aeeca5dff: Status 404 returned error can't find the container with id bfd366b4c5f2935b00e350c404bd763612686cf422edd6fcb20ca01aeeca5dff Oct 31 04:12:51.355294 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:12:51.355253 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 04:12:51.431838 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:12:51.431797 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-phl42" event={"ID":"3d152b49-e42e-4584-a7ee-7b540322fd96","Type":"ContainerStarted","Data":"bfd366b4c5f2935b00e350c404bd763612686cf422edd6fcb20ca01aeeca5dff"} Oct 31 04:12:52.441663 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:12:52.441627 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-phl42" event={"ID":"3d152b49-e42e-4584-a7ee-7b540322fd96","Type":"ContainerStarted","Data":"488cce00fb4e87a43c8dfd3baa578408f7edffba790efb05bd6105b61b8f6e0d"} Oct 31 04:12:52.485034 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:12:52.484985 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-phl42" podStartSLOduration=1.281278317 podStartE2EDuration="2.484968339s" totalImagesPullingTime="1.203690022s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:12:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:12:52.484815036 +0000 UTC m=+520.199149655" watchObservedRunningTime="2026-10-31 04:12:52.484968339 +0000 UTC m=+520.199302959" Oct 31 04:12:53.415984 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:12:53.415944 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-phl42" Oct 31 04:12:54.841905 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:12:54.841869 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 04:12:57.018889 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:12:57.018855 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-gpzs9"] Oct 31 04:12:57.022128 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:12:57.022101 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-gpzs9" Oct 31 04:12:57.051734 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:12:57.051697 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Oct 31 04:12:57.053855 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:12:57.053829 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-gpzs9"] Oct 31 04:12:57.145149 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:12:57.145110 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91164053-d830-4b60-a4ef-ed2877b7c27c-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-gpzs9\" (UID: \"91164053-d830-4b60-a4ef-ed2877b7c27c\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-gpzs9" Oct 31 04:12:57.145319 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:12:57.145162 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91164053-d830-4b60-a4ef-ed2877b7c27c-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-gpzs9\" (UID: \"91164053-d830-4b60-a4ef-ed2877b7c27c\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-gpzs9" Oct 31 04:12:57.145319 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:12:57.145243 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91164053-d830-4b60-a4ef-ed2877b7c27c-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-gpzs9\" (UID: \"91164053-d830-4b60-a4ef-ed2877b7c27c\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-gpzs9" Oct 31 04:12:57.145319 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:12:57.145279 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dj9mc\" (UniqueName: \"kubernetes.io/projected/91164053-d830-4b60-a4ef-ed2877b7c27c-kube-api-access-dj9mc\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-gpzs9\" (UID: \"91164053-d830-4b60-a4ef-ed2877b7c27c\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-gpzs9" Oct 31 04:12:57.145450 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:12:57.145354 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91164053-d830-4b60-a4ef-ed2877b7c27c-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-gpzs9\" (UID: \"91164053-d830-4b60-a4ef-ed2877b7c27c\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-gpzs9" Oct 31 04:12:57.145450 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:12:57.145386 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91164053-d830-4b60-a4ef-ed2877b7c27c-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-gpzs9\" (UID: \"91164053-d830-4b60-a4ef-ed2877b7c27c\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-gpzs9" Oct 31 04:12:57.246520 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:12:57.246477 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91164053-d830-4b60-a4ef-ed2877b7c27c-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-gpzs9\" (UID: \"91164053-d830-4b60-a4ef-ed2877b7c27c\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-gpzs9" Oct 31 04:12:57.246707 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:12:57.246563 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91164053-d830-4b60-a4ef-ed2877b7c27c-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-gpzs9\" (UID: \"91164053-d830-4b60-a4ef-ed2877b7c27c\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-gpzs9" Oct 31 04:12:57.246707 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:12:57.246588 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91164053-d830-4b60-a4ef-ed2877b7c27c-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-gpzs9\" (UID: \"91164053-d830-4b60-a4ef-ed2877b7c27c\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-gpzs9" Oct 31 04:12:57.246707 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:12:57.246603 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91164053-d830-4b60-a4ef-ed2877b7c27c-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-gpzs9\" (UID: \"91164053-d830-4b60-a4ef-ed2877b7c27c\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-gpzs9" Oct 31 04:12:57.246707 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:12:57.246624 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dj9mc\" (UniqueName: \"kubernetes.io/projected/91164053-d830-4b60-a4ef-ed2877b7c27c-kube-api-access-dj9mc\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-gpzs9\" (UID: \"91164053-d830-4b60-a4ef-ed2877b7c27c\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-gpzs9" Oct 31 04:12:57.246707 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:12:57.246665 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91164053-d830-4b60-a4ef-ed2877b7c27c-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-gpzs9\" (UID: \"91164053-d830-4b60-a4ef-ed2877b7c27c\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-gpzs9" Oct 31 04:12:57.246985 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:12:57.246962 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91164053-d830-4b60-a4ef-ed2877b7c27c-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-gpzs9\" (UID: \"91164053-d830-4b60-a4ef-ed2877b7c27c\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-gpzs9" Oct 31 04:12:57.247053 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:12:57.246988 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91164053-d830-4b60-a4ef-ed2877b7c27c-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-gpzs9\" (UID: \"91164053-d830-4b60-a4ef-ed2877b7c27c\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-gpzs9" Oct 31 04:12:57.247122 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:12:57.247054 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91164053-d830-4b60-a4ef-ed2877b7c27c-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-gpzs9\" (UID: \"91164053-d830-4b60-a4ef-ed2877b7c27c\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-gpzs9" Oct 31 04:12:57.248886 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:12:57.248866 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91164053-d830-4b60-a4ef-ed2877b7c27c-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-gpzs9\" (UID: \"91164053-d830-4b60-a4ef-ed2877b7c27c\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-gpzs9" Oct 31 04:12:57.249063 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:12:57.249043 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91164053-d830-4b60-a4ef-ed2877b7c27c-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-gpzs9\" (UID: \"91164053-d830-4b60-a4ef-ed2877b7c27c\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-gpzs9" Oct 31 04:12:57.270608 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:12:57.270548 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dj9mc\" (UniqueName: \"kubernetes.io/projected/91164053-d830-4b60-a4ef-ed2877b7c27c-kube-api-access-dj9mc\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-gpzs9\" (UID: \"91164053-d830-4b60-a4ef-ed2877b7c27c\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-gpzs9" Oct 31 04:12:57.335496 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:12:57.335457 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-gpzs9" Oct 31 04:12:57.644647 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:12:57.644615 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-gpzs9"] Oct 31 04:12:57.645029 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:12:57.645005 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91164053_d830_4b60_a4ef_ed2877b7c27c.slice/crio-47f76be40ee4f34d3ad21d44cbfb25349c46e2d0bc71f758019ad784ea1bcf9b WatchSource:0}: Error finding container 47f76be40ee4f34d3ad21d44cbfb25349c46e2d0bc71f758019ad784ea1bcf9b: Status 404 returned error can't find the container with id 47f76be40ee4f34d3ad21d44cbfb25349c46e2d0bc71f758019ad784ea1bcf9b Oct 31 04:12:58.466612 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:12:58.466579 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-gpzs9" event={"ID":"91164053-d830-4b60-a4ef-ed2877b7c27c","Type":"ContainerStarted","Data":"6ad2a6fd5d7de76a3aea2b2b625b15c75b2ec164f5f3ed6ed4f4a6a75de21396"} Oct 31 04:12:58.467003 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:12:58.466628 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-gpzs9" event={"ID":"91164053-d830-4b60-a4ef-ed2877b7c27c","Type":"ContainerStarted","Data":"47f76be40ee4f34d3ad21d44cbfb25349c46e2d0bc71f758019ad784ea1bcf9b"} Oct 31 04:12:58.510211 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:12:58.510156 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-gpzs9" podStartSLOduration=2.30662295 podStartE2EDuration="2.510142525s" totalImagesPullingTime="203.519575ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:12:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:12:58.509809588 +0000 UTC m=+526.224144205" watchObservedRunningTime="2026-10-31 04:12:58.510142525 +0000 UTC m=+526.224477142" Oct 31 04:12:58.912404 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:12:58.909193 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 04:12:59.336281 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:12:59.336203 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-gpzs9" Oct 31 04:13:00.415505 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:00.415473 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-phl42" Oct 31 04:13:00.427927 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:00.427896 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-phl42" Oct 31 04:13:00.486787 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:00.486757 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-phl42" Oct 31 04:13:00.678208 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:00.678126 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-v7km4"] Oct 31 04:13:00.682069 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:00.682044 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-v7km4" Oct 31 04:13:00.703029 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:00.702999 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\"" Oct 31 04:13:00.703554 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:00.703534 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-v7km4"] Oct 31 04:13:00.775854 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:00.775819 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/839587e6-d175-4bfd-b531-0f1bc94b30b6-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-v7km4\" (UID: \"839587e6-d175-4bfd-b531-0f1bc94b30b6\") " pod="llm/echo-simulator-kserve-64b6f8f96c-v7km4" Oct 31 04:13:00.776042 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:00.775872 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fn6x6\" (UniqueName: \"kubernetes.io/projected/839587e6-d175-4bfd-b531-0f1bc94b30b6-kube-api-access-fn6x6\") pod \"echo-simulator-kserve-64b6f8f96c-v7km4\" (UID: \"839587e6-d175-4bfd-b531-0f1bc94b30b6\") " pod="llm/echo-simulator-kserve-64b6f8f96c-v7km4" Oct 31 04:13:00.776042 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:00.775960 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/839587e6-d175-4bfd-b531-0f1bc94b30b6-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-v7km4\" (UID: \"839587e6-d175-4bfd-b531-0f1bc94b30b6\") " pod="llm/echo-simulator-kserve-64b6f8f96c-v7km4" Oct 31 04:13:00.776042 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:00.776000 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/839587e6-d175-4bfd-b531-0f1bc94b30b6-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-v7km4\" (UID: \"839587e6-d175-4bfd-b531-0f1bc94b30b6\") " pod="llm/echo-simulator-kserve-64b6f8f96c-v7km4" Oct 31 04:13:00.776157 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:00.776059 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/839587e6-d175-4bfd-b531-0f1bc94b30b6-home\") pod \"echo-simulator-kserve-64b6f8f96c-v7km4\" (UID: \"839587e6-d175-4bfd-b531-0f1bc94b30b6\") " pod="llm/echo-simulator-kserve-64b6f8f96c-v7km4" Oct 31 04:13:00.776157 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:00.776089 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/839587e6-d175-4bfd-b531-0f1bc94b30b6-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-v7km4\" (UID: \"839587e6-d175-4bfd-b531-0f1bc94b30b6\") " pod="llm/echo-simulator-kserve-64b6f8f96c-v7km4" Oct 31 04:13:00.877509 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:00.877473 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/839587e6-d175-4bfd-b531-0f1bc94b30b6-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-v7km4\" (UID: \"839587e6-d175-4bfd-b531-0f1bc94b30b6\") " pod="llm/echo-simulator-kserve-64b6f8f96c-v7km4" Oct 31 04:13:00.877711 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:00.877528 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fn6x6\" (UniqueName: \"kubernetes.io/projected/839587e6-d175-4bfd-b531-0f1bc94b30b6-kube-api-access-fn6x6\") pod \"echo-simulator-kserve-64b6f8f96c-v7km4\" (UID: \"839587e6-d175-4bfd-b531-0f1bc94b30b6\") " pod="llm/echo-simulator-kserve-64b6f8f96c-v7km4" Oct 31 04:13:00.877711 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:00.877572 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/839587e6-d175-4bfd-b531-0f1bc94b30b6-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-v7km4\" (UID: \"839587e6-d175-4bfd-b531-0f1bc94b30b6\") " pod="llm/echo-simulator-kserve-64b6f8f96c-v7km4" Oct 31 04:13:00.877711 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:00.877603 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/839587e6-d175-4bfd-b531-0f1bc94b30b6-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-v7km4\" (UID: \"839587e6-d175-4bfd-b531-0f1bc94b30b6\") " pod="llm/echo-simulator-kserve-64b6f8f96c-v7km4" Oct 31 04:13:00.877711 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:00.877636 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/839587e6-d175-4bfd-b531-0f1bc94b30b6-home\") pod \"echo-simulator-kserve-64b6f8f96c-v7km4\" (UID: \"839587e6-d175-4bfd-b531-0f1bc94b30b6\") " pod="llm/echo-simulator-kserve-64b6f8f96c-v7km4" Oct 31 04:13:00.877711 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:00.877659 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/839587e6-d175-4bfd-b531-0f1bc94b30b6-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-v7km4\" (UID: \"839587e6-d175-4bfd-b531-0f1bc94b30b6\") " pod="llm/echo-simulator-kserve-64b6f8f96c-v7km4" Oct 31 04:13:00.878019 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:00.877994 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/839587e6-d175-4bfd-b531-0f1bc94b30b6-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-v7km4\" (UID: \"839587e6-d175-4bfd-b531-0f1bc94b30b6\") " pod="llm/echo-simulator-kserve-64b6f8f96c-v7km4" Oct 31 04:13:00.878099 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:00.878058 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/839587e6-d175-4bfd-b531-0f1bc94b30b6-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-v7km4\" (UID: \"839587e6-d175-4bfd-b531-0f1bc94b30b6\") " pod="llm/echo-simulator-kserve-64b6f8f96c-v7km4" Oct 31 04:13:00.878179 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:00.878157 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/839587e6-d175-4bfd-b531-0f1bc94b30b6-home\") pod \"echo-simulator-kserve-64b6f8f96c-v7km4\" (UID: \"839587e6-d175-4bfd-b531-0f1bc94b30b6\") " pod="llm/echo-simulator-kserve-64b6f8f96c-v7km4" Oct 31 04:13:00.879855 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:00.879834 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/839587e6-d175-4bfd-b531-0f1bc94b30b6-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-v7km4\" (UID: \"839587e6-d175-4bfd-b531-0f1bc94b30b6\") " pod="llm/echo-simulator-kserve-64b6f8f96c-v7km4" Oct 31 04:13:00.880015 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:00.879998 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/839587e6-d175-4bfd-b531-0f1bc94b30b6-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-v7km4\" (UID: \"839587e6-d175-4bfd-b531-0f1bc94b30b6\") " pod="llm/echo-simulator-kserve-64b6f8f96c-v7km4" Oct 31 04:13:00.901601 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:00.901567 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fn6x6\" (UniqueName: \"kubernetes.io/projected/839587e6-d175-4bfd-b531-0f1bc94b30b6-kube-api-access-fn6x6\") pod \"echo-simulator-kserve-64b6f8f96c-v7km4\" (UID: \"839587e6-d175-4bfd-b531-0f1bc94b30b6\") " pod="llm/echo-simulator-kserve-64b6f8f96c-v7km4" Oct 31 04:13:00.995949 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:00.995849 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-v7km4" Oct 31 04:13:01.293799 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:13:01.293765 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod839587e6_d175_4bfd_b531_0f1bc94b30b6.slice/crio-7bbfe393ffe51dddf7f39ff1cc6230770f5e60efbaab3f190aed5791ebb11426 WatchSource:0}: Error finding container 7bbfe393ffe51dddf7f39ff1cc6230770f5e60efbaab3f190aed5791ebb11426: Status 404 returned error can't find the container with id 7bbfe393ffe51dddf7f39ff1cc6230770f5e60efbaab3f190aed5791ebb11426 Oct 31 04:13:01.293926 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:01.293843 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-v7km4"] Oct 31 04:13:01.483182 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:01.483135 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-v7km4" event={"ID":"839587e6-d175-4bfd-b531-0f1bc94b30b6","Type":"ContainerStarted","Data":"7bbfe393ffe51dddf7f39ff1cc6230770f5e60efbaab3f190aed5791ebb11426"} Oct 31 04:13:03.414380 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:03.414346 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-d7npm"] Oct 31 04:13:03.417703 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:03.417685 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-d7npm" Oct 31 04:13:03.439713 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:03.439685 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-d7npm"] Oct 31 04:13:03.450587 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:03.450556 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Oct 31 04:13:03.498052 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:03.498018 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-v7km4" event={"ID":"839587e6-d175-4bfd-b531-0f1bc94b30b6","Type":"ContainerStarted","Data":"6fc9fe1565a41d0ffb6aa73a0ddb4e9ebc979d419bbfc225c28f2392c024064b"} Oct 31 04:13:03.501638 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:03.501611 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/174171e7-100c-4b84-becd-0176672f1805-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-d7npm\" (UID: \"174171e7-100c-4b84-becd-0176672f1805\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-d7npm" Oct 31 04:13:03.501791 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:03.501650 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/174171e7-100c-4b84-becd-0176672f1805-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-d7npm\" (UID: \"174171e7-100c-4b84-becd-0176672f1805\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-d7npm" Oct 31 04:13:03.501791 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:03.501682 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdwzv\" (UniqueName: \"kubernetes.io/projected/174171e7-100c-4b84-becd-0176672f1805-kube-api-access-cdwzv\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-d7npm\" (UID: \"174171e7-100c-4b84-becd-0176672f1805\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-d7npm" Oct 31 04:13:03.501905 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:03.501813 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/174171e7-100c-4b84-becd-0176672f1805-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-d7npm\" (UID: \"174171e7-100c-4b84-becd-0176672f1805\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-d7npm" Oct 31 04:13:03.501905 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:03.501869 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/174171e7-100c-4b84-becd-0176672f1805-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-d7npm\" (UID: \"174171e7-100c-4b84-becd-0176672f1805\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-d7npm" Oct 31 04:13:03.502004 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:03.501947 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/174171e7-100c-4b84-becd-0176672f1805-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-d7npm\" (UID: \"174171e7-100c-4b84-becd-0176672f1805\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-d7npm" Oct 31 04:13:03.544864 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:03.544808 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/echo-simulator-kserve-64b6f8f96c-v7km4" podStartSLOduration=2.022303864 podStartE2EDuration="3.54479313s" totalImagesPullingTime="1.522489266s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:13:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:13:03.544386436 +0000 UTC m=+531.258721057" watchObservedRunningTime="2026-10-31 04:13:03.54479313 +0000 UTC m=+531.259127748" Oct 31 04:13:03.602528 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:03.602492 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/174171e7-100c-4b84-becd-0176672f1805-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-d7npm\" (UID: \"174171e7-100c-4b84-becd-0176672f1805\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-d7npm" Oct 31 04:13:03.602698 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:03.602629 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/174171e7-100c-4b84-becd-0176672f1805-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-d7npm\" (UID: \"174171e7-100c-4b84-becd-0176672f1805\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-d7npm" Oct 31 04:13:03.602698 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:03.602664 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/174171e7-100c-4b84-becd-0176672f1805-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-d7npm\" (UID: \"174171e7-100c-4b84-becd-0176672f1805\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-d7npm" Oct 31 04:13:03.602827 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:03.602806 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/174171e7-100c-4b84-becd-0176672f1805-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-d7npm\" (UID: \"174171e7-100c-4b84-becd-0176672f1805\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-d7npm" Oct 31 04:13:03.602900 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:03.602883 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/174171e7-100c-4b84-becd-0176672f1805-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-d7npm\" (UID: \"174171e7-100c-4b84-becd-0176672f1805\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-d7npm" Oct 31 04:13:03.602950 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:03.602932 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdwzv\" (UniqueName: \"kubernetes.io/projected/174171e7-100c-4b84-becd-0176672f1805-kube-api-access-cdwzv\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-d7npm\" (UID: \"174171e7-100c-4b84-becd-0176672f1805\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-d7npm" Oct 31 04:13:03.603057 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:03.603040 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/174171e7-100c-4b84-becd-0176672f1805-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-d7npm\" (UID: \"174171e7-100c-4b84-becd-0176672f1805\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-d7npm" Oct 31 04:13:03.603114 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:03.603092 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/174171e7-100c-4b84-becd-0176672f1805-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-d7npm\" (UID: \"174171e7-100c-4b84-becd-0176672f1805\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-d7npm" Oct 31 04:13:03.603215 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:03.603194 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/174171e7-100c-4b84-becd-0176672f1805-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-d7npm\" (UID: \"174171e7-100c-4b84-becd-0176672f1805\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-d7npm" Oct 31 04:13:03.604925 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:03.604892 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/174171e7-100c-4b84-becd-0176672f1805-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-d7npm\" (UID: \"174171e7-100c-4b84-becd-0176672f1805\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-d7npm" Oct 31 04:13:03.605208 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:03.605187 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/174171e7-100c-4b84-becd-0176672f1805-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-d7npm\" (UID: \"174171e7-100c-4b84-becd-0176672f1805\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-d7npm" Oct 31 04:13:03.629987 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:03.629951 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdwzv\" (UniqueName: \"kubernetes.io/projected/174171e7-100c-4b84-becd-0176672f1805-kube-api-access-cdwzv\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-d7npm\" (UID: \"174171e7-100c-4b84-becd-0176672f1805\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-d7npm" Oct 31 04:13:03.740184 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:03.740107 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-d7npm" Oct 31 04:13:03.996856 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:03.996783 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-v7km4" Oct 31 04:13:04.034753 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:13:04.034727 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod174171e7_100c_4b84_becd_0176672f1805.slice/crio-34ea3f04cc340c5a6d8e9df9e39b0ea0d64cc1a1d6682de6f448544504684ee8 WatchSource:0}: Error finding container 34ea3f04cc340c5a6d8e9df9e39b0ea0d64cc1a1d6682de6f448544504684ee8: Status 404 returned error can't find the container with id 34ea3f04cc340c5a6d8e9df9e39b0ea0d64cc1a1d6682de6f448544504684ee8 Oct 31 04:13:04.034854 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:04.034745 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-d7npm"] Oct 31 04:13:04.503827 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:04.503738 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-d7npm" event={"ID":"174171e7-100c-4b84-becd-0176672f1805","Type":"ContainerStarted","Data":"56188ac75cff0a0bbf6a1d4c8533becc01189c5608e85b3c81afef74e997a044"} Oct 31 04:13:04.503827 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:04.503776 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-d7npm" event={"ID":"174171e7-100c-4b84-becd-0176672f1805","Type":"ContainerStarted","Data":"34ea3f04cc340c5a6d8e9df9e39b0ea0d64cc1a1d6682de6f448544504684ee8"} Oct 31 04:13:04.546960 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:04.546906 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-d7npm" podStartSLOduration=1.343175527 podStartE2EDuration="1.546893222s" totalImagesPullingTime="203.717695ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:13:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:13:04.5461681 +0000 UTC m=+532.260502718" watchObservedRunningTime="2026-10-31 04:13:04.546893222 +0000 UTC m=+532.261227838" Oct 31 04:13:04.740895 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:04.740846 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-d7npm" Oct 31 04:13:05.635622 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:05.635587 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 04:13:07.336145 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:07.336110 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-gpzs9" Oct 31 04:13:07.348786 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:07.348757 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-gpzs9" Oct 31 04:13:07.528490 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:07.528457 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-gpzs9" Oct 31 04:13:10.997117 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:10.997083 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/echo-simulator-kserve-64b6f8f96c-v7km4" Oct 31 04:13:10.999873 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:13:10.999844 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 04:13:11.009924 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:11.009891 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/echo-simulator-kserve-64b6f8f96c-v7km4" Oct 31 04:13:11.545455 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:11.545425 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-v7km4" Oct 31 04:13:13.740951 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:13.740916 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-d7npm" Oct 31 04:13:13.753221 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:13.753191 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-d7npm" Oct 31 04:13:14.558059 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:14.558025 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-d7npm" Oct 31 04:13:23.694460 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:23.694427 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-crjjl"] Oct 31 04:13:23.701962 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:23.701937 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-crjjl" Oct 31 04:13:23.728900 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:23.728872 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-crjjl"] Oct 31 04:13:23.733946 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:23.733926 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-sqqs6\"" Oct 31 04:13:23.782256 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:23.782221 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63486e50-4059-425f-b04a-4adea538d427-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-crjjl\" (UID: \"63486e50-4059-425f-b04a-4adea538d427\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-crjjl" Oct 31 04:13:23.782490 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:23.782275 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63486e50-4059-425f-b04a-4adea538d427-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-crjjl\" (UID: \"63486e50-4059-425f-b04a-4adea538d427\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-crjjl" Oct 31 04:13:23.782490 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:23.782324 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63486e50-4059-425f-b04a-4adea538d427-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-crjjl\" (UID: \"63486e50-4059-425f-b04a-4adea538d427\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-crjjl" Oct 31 04:13:23.782490 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:23.782384 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvgkj\" (UniqueName: \"kubernetes.io/projected/63486e50-4059-425f-b04a-4adea538d427-kube-api-access-nvgkj\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-crjjl\" (UID: \"63486e50-4059-425f-b04a-4adea538d427\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-crjjl" Oct 31 04:13:23.782490 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:23.782453 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63486e50-4059-425f-b04a-4adea538d427-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-crjjl\" (UID: \"63486e50-4059-425f-b04a-4adea538d427\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-crjjl" Oct 31 04:13:23.782729 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:23.782491 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63486e50-4059-425f-b04a-4adea538d427-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-crjjl\" (UID: \"63486e50-4059-425f-b04a-4adea538d427\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-crjjl" Oct 31 04:13:23.782729 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:23.782584 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63486e50-4059-425f-b04a-4adea538d427-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-crjjl\" (UID: \"63486e50-4059-425f-b04a-4adea538d427\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-crjjl" Oct 31 04:13:23.782729 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:23.782646 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63486e50-4059-425f-b04a-4adea538d427-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-crjjl\" (UID: \"63486e50-4059-425f-b04a-4adea538d427\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-crjjl" Oct 31 04:13:23.782729 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:23.782671 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63486e50-4059-425f-b04a-4adea538d427-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-crjjl\" (UID: \"63486e50-4059-425f-b04a-4adea538d427\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-crjjl" Oct 31 04:13:23.883264 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:23.883227 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63486e50-4059-425f-b04a-4adea538d427-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-crjjl\" (UID: \"63486e50-4059-425f-b04a-4adea538d427\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-crjjl" Oct 31 04:13:23.883264 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:23.883266 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63486e50-4059-425f-b04a-4adea538d427-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-crjjl\" (UID: \"63486e50-4059-425f-b04a-4adea538d427\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-crjjl" Oct 31 04:13:23.883550 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:23.883288 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63486e50-4059-425f-b04a-4adea538d427-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-crjjl\" (UID: \"63486e50-4059-425f-b04a-4adea538d427\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-crjjl" Oct 31 04:13:23.883550 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:23.883305 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63486e50-4059-425f-b04a-4adea538d427-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-crjjl\" (UID: \"63486e50-4059-425f-b04a-4adea538d427\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-crjjl" Oct 31 04:13:23.883550 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:23.883355 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63486e50-4059-425f-b04a-4adea538d427-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-crjjl\" (UID: \"63486e50-4059-425f-b04a-4adea538d427\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-crjjl" Oct 31 04:13:23.883550 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:23.883387 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63486e50-4059-425f-b04a-4adea538d427-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-crjjl\" (UID: \"63486e50-4059-425f-b04a-4adea538d427\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-crjjl" Oct 31 04:13:23.883550 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:23.883412 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63486e50-4059-425f-b04a-4adea538d427-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-crjjl\" (UID: \"63486e50-4059-425f-b04a-4adea538d427\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-crjjl" Oct 31 04:13:23.883550 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:23.883459 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvgkj\" (UniqueName: \"kubernetes.io/projected/63486e50-4059-425f-b04a-4adea538d427-kube-api-access-nvgkj\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-crjjl\" (UID: \"63486e50-4059-425f-b04a-4adea538d427\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-crjjl" Oct 31 04:13:23.883550 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:23.883519 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63486e50-4059-425f-b04a-4adea538d427-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-crjjl\" (UID: \"63486e50-4059-425f-b04a-4adea538d427\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-crjjl" Oct 31 04:13:23.883966 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:23.883758 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63486e50-4059-425f-b04a-4adea538d427-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-crjjl\" (UID: \"63486e50-4059-425f-b04a-4adea538d427\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-crjjl" Oct 31 04:13:23.883966 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:23.883789 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63486e50-4059-425f-b04a-4adea538d427-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-crjjl\" (UID: \"63486e50-4059-425f-b04a-4adea538d427\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-crjjl" Oct 31 04:13:23.883966 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:23.883896 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63486e50-4059-425f-b04a-4adea538d427-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-crjjl\" (UID: \"63486e50-4059-425f-b04a-4adea538d427\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-crjjl" Oct 31 04:13:23.883966 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:23.883963 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63486e50-4059-425f-b04a-4adea538d427-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-crjjl\" (UID: \"63486e50-4059-425f-b04a-4adea538d427\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-crjjl" Oct 31 04:13:23.884190 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:23.884171 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63486e50-4059-425f-b04a-4adea538d427-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-crjjl\" (UID: \"63486e50-4059-425f-b04a-4adea538d427\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-crjjl" Oct 31 04:13:23.885816 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:23.885796 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63486e50-4059-425f-b04a-4adea538d427-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-crjjl\" (UID: \"63486e50-4059-425f-b04a-4adea538d427\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-crjjl" Oct 31 04:13:23.885904 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:23.885888 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63486e50-4059-425f-b04a-4adea538d427-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-crjjl\" (UID: \"63486e50-4059-425f-b04a-4adea538d427\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-crjjl" Oct 31 04:13:23.907263 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:23.907234 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63486e50-4059-425f-b04a-4adea538d427-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-crjjl\" (UID: \"63486e50-4059-425f-b04a-4adea538d427\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-crjjl" Oct 31 04:13:23.907579 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:23.907556 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvgkj\" (UniqueName: \"kubernetes.io/projected/63486e50-4059-425f-b04a-4adea538d427-kube-api-access-nvgkj\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-crjjl\" (UID: \"63486e50-4059-425f-b04a-4adea538d427\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-crjjl" Oct 31 04:13:24.016561 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:24.016463 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-crjjl" Oct 31 04:13:24.336247 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:24.336220 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-crjjl"] Oct 31 04:13:24.336998 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:13:24.336971 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63486e50_4059_425f_b04a_4adea538d427.slice/crio-b5499f9070ec575618ec370fad01787adbbb585085137ae1ff2d5b0951e65cee WatchSource:0}: Error finding container b5499f9070ec575618ec370fad01787adbbb585085137ae1ff2d5b0951e65cee: Status 404 returned error can't find the container with id b5499f9070ec575618ec370fad01787adbbb585085137ae1ff2d5b0951e65cee Oct 31 04:13:24.595684 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:24.595589 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-crjjl" event={"ID":"63486e50-4059-425f-b04a-4adea538d427","Type":"ContainerStarted","Data":"b5499f9070ec575618ec370fad01787adbbb585085137ae1ff2d5b0951e65cee"} Oct 31 04:13:27.045624 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:27.045587 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:13:27.045927 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:27.045654 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:13:27.045927 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:27.045689 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:13:27.610037 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:27.610000 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-crjjl" event={"ID":"63486e50-4059-425f-b04a-4adea538d427","Type":"ContainerStarted","Data":"ed822ccdccebd79101c8df29b99ac5823650b406625a22bb6e931795f33bada2"} Oct 31 04:13:27.689166 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:27.689117 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-crjjl" podStartSLOduration=1.983248509 podStartE2EDuration="4.689100406s" totalImagesPullingTime="2.705851897s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:13:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:13:27.675453829 +0000 UTC m=+555.389788470" watchObservedRunningTime="2026-10-31 04:13:27.689100406 +0000 UTC m=+555.403435022" Oct 31 04:13:28.017419 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:28.017387 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-crjjl" Oct 31 04:13:28.018826 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:28.018802 4376 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-crjjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Oct 31 04:13:28.018941 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:28.018861 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-crjjl" podUID="63486e50-4059-425f-b04a-4adea538d427" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Oct 31 04:13:29.016831 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:29.016795 4376 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-crjjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Oct 31 04:13:29.017367 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:29.016856 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-crjjl" podUID="63486e50-4059-425f-b04a-4adea538d427" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Oct 31 04:13:30.017807 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:30.017770 4376 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-crjjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Oct 31 04:13:30.018200 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:30.017825 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-crjjl" podUID="63486e50-4059-425f-b04a-4adea538d427" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Oct 31 04:13:31.017686 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:31.017647 4376 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-crjjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Oct 31 04:13:31.017865 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:31.017707 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-crjjl" podUID="63486e50-4059-425f-b04a-4adea538d427" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Oct 31 04:13:32.017550 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:32.017510 4376 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-crjjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Oct 31 04:13:32.017710 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:32.017568 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-crjjl" podUID="63486e50-4059-425f-b04a-4adea538d427" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Oct 31 04:13:33.017179 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:33.017140 4376 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-crjjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Oct 31 04:13:33.019419 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:33.017209 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-crjjl" podUID="63486e50-4059-425f-b04a-4adea538d427" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Oct 31 04:13:34.017410 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:34.017359 4376 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-crjjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Oct 31 04:13:34.017805 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:34.017429 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-crjjl" podUID="63486e50-4059-425f-b04a-4adea538d427" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Oct 31 04:13:35.292786 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:35.292743 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-crjjl" Oct 31 04:13:35.293321 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:35.293076 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-crjjl" Oct 31 04:13:35.293873 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:35.293850 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-crjjl" Oct 31 04:13:38.741153 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:38.741114 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mrx9s"] Oct 31 04:13:38.744927 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:38.744906 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mrx9s" Oct 31 04:13:38.769824 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:38.769795 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mrx9s"] Oct 31 04:13:38.844392 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:38.844353 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70120ad8-48aa-44b4-b396-64eab56ce59b-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-mrx9s\" (UID: \"70120ad8-48aa-44b4-b396-64eab56ce59b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mrx9s" Oct 31 04:13:38.844392 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:38.844391 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70120ad8-48aa-44b4-b396-64eab56ce59b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-mrx9s\" (UID: \"70120ad8-48aa-44b4-b396-64eab56ce59b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mrx9s" Oct 31 04:13:38.844593 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:38.844410 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-546rh\" (UniqueName: \"kubernetes.io/projected/70120ad8-48aa-44b4-b396-64eab56ce59b-kube-api-access-546rh\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-mrx9s\" (UID: \"70120ad8-48aa-44b4-b396-64eab56ce59b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mrx9s" Oct 31 04:13:38.844593 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:38.844510 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70120ad8-48aa-44b4-b396-64eab56ce59b-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-mrx9s\" (UID: \"70120ad8-48aa-44b4-b396-64eab56ce59b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mrx9s" Oct 31 04:13:38.844593 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:38.844550 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70120ad8-48aa-44b4-b396-64eab56ce59b-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-mrx9s\" (UID: \"70120ad8-48aa-44b4-b396-64eab56ce59b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mrx9s" Oct 31 04:13:38.844720 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:38.844695 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70120ad8-48aa-44b4-b396-64eab56ce59b-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-mrx9s\" (UID: \"70120ad8-48aa-44b4-b396-64eab56ce59b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mrx9s" Oct 31 04:13:38.844758 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:38.844730 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70120ad8-48aa-44b4-b396-64eab56ce59b-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-mrx9s\" (UID: \"70120ad8-48aa-44b4-b396-64eab56ce59b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mrx9s" Oct 31 04:13:38.844792 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:38.844758 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70120ad8-48aa-44b4-b396-64eab56ce59b-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-mrx9s\" (UID: \"70120ad8-48aa-44b4-b396-64eab56ce59b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mrx9s" Oct 31 04:13:38.844876 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:38.844857 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70120ad8-48aa-44b4-b396-64eab56ce59b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-mrx9s\" (UID: \"70120ad8-48aa-44b4-b396-64eab56ce59b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mrx9s" Oct 31 04:13:38.945614 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:38.945581 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70120ad8-48aa-44b4-b396-64eab56ce59b-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-mrx9s\" (UID: \"70120ad8-48aa-44b4-b396-64eab56ce59b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mrx9s" Oct 31 04:13:38.945815 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:38.945651 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70120ad8-48aa-44b4-b396-64eab56ce59b-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-mrx9s\" (UID: \"70120ad8-48aa-44b4-b396-64eab56ce59b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mrx9s" Oct 31 04:13:38.945815 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:38.945679 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70120ad8-48aa-44b4-b396-64eab56ce59b-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-mrx9s\" (UID: \"70120ad8-48aa-44b4-b396-64eab56ce59b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mrx9s" Oct 31 04:13:38.945815 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:38.945699 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70120ad8-48aa-44b4-b396-64eab56ce59b-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-mrx9s\" (UID: \"70120ad8-48aa-44b4-b396-64eab56ce59b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mrx9s" Oct 31 04:13:38.945815 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:38.945727 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70120ad8-48aa-44b4-b396-64eab56ce59b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-mrx9s\" (UID: \"70120ad8-48aa-44b4-b396-64eab56ce59b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mrx9s" Oct 31 04:13:38.945815 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:38.945751 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70120ad8-48aa-44b4-b396-64eab56ce59b-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-mrx9s\" (UID: \"70120ad8-48aa-44b4-b396-64eab56ce59b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mrx9s" Oct 31 04:13:38.945815 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:38.945774 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70120ad8-48aa-44b4-b396-64eab56ce59b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-mrx9s\" (UID: \"70120ad8-48aa-44b4-b396-64eab56ce59b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mrx9s" Oct 31 04:13:38.945815 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:38.945798 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-546rh\" (UniqueName: \"kubernetes.io/projected/70120ad8-48aa-44b4-b396-64eab56ce59b-kube-api-access-546rh\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-mrx9s\" (UID: \"70120ad8-48aa-44b4-b396-64eab56ce59b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mrx9s" Oct 31 04:13:38.946241 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:38.945825 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70120ad8-48aa-44b4-b396-64eab56ce59b-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-mrx9s\" (UID: \"70120ad8-48aa-44b4-b396-64eab56ce59b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mrx9s" Oct 31 04:13:38.946241 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:38.946061 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70120ad8-48aa-44b4-b396-64eab56ce59b-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-mrx9s\" (UID: \"70120ad8-48aa-44b4-b396-64eab56ce59b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mrx9s" Oct 31 04:13:38.946241 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:38.946080 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70120ad8-48aa-44b4-b396-64eab56ce59b-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-mrx9s\" (UID: \"70120ad8-48aa-44b4-b396-64eab56ce59b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mrx9s" Oct 31 04:13:38.946241 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:38.946162 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70120ad8-48aa-44b4-b396-64eab56ce59b-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-mrx9s\" (UID: \"70120ad8-48aa-44b4-b396-64eab56ce59b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mrx9s" Oct 31 04:13:38.946241 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:38.946207 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70120ad8-48aa-44b4-b396-64eab56ce59b-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-mrx9s\" (UID: \"70120ad8-48aa-44b4-b396-64eab56ce59b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mrx9s" Oct 31 04:13:38.946549 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:38.946525 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70120ad8-48aa-44b4-b396-64eab56ce59b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-mrx9s\" (UID: \"70120ad8-48aa-44b4-b396-64eab56ce59b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mrx9s" Oct 31 04:13:38.948252 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:38.948223 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70120ad8-48aa-44b4-b396-64eab56ce59b-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-mrx9s\" (UID: \"70120ad8-48aa-44b4-b396-64eab56ce59b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mrx9s" Oct 31 04:13:38.948468 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:38.948452 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70120ad8-48aa-44b4-b396-64eab56ce59b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-mrx9s\" (UID: \"70120ad8-48aa-44b4-b396-64eab56ce59b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mrx9s" Oct 31 04:13:38.969929 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:38.969895 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70120ad8-48aa-44b4-b396-64eab56ce59b-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-mrx9s\" (UID: \"70120ad8-48aa-44b4-b396-64eab56ce59b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mrx9s" Oct 31 04:13:38.970052 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:38.969959 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-546rh\" (UniqueName: \"kubernetes.io/projected/70120ad8-48aa-44b4-b396-64eab56ce59b-kube-api-access-546rh\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-mrx9s\" (UID: \"70120ad8-48aa-44b4-b396-64eab56ce59b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mrx9s" Oct 31 04:13:39.058222 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:39.058133 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mrx9s" Oct 31 04:13:39.368320 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:13:39.368286 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70120ad8_48aa_44b4_b396_64eab56ce59b.slice/crio-52baa228b292f8319077488cc313e975602e3df8af32592d3e27433971eeab2d WatchSource:0}: Error finding container 52baa228b292f8319077488cc313e975602e3df8af32592d3e27433971eeab2d: Status 404 returned error can't find the container with id 52baa228b292f8319077488cc313e975602e3df8af32592d3e27433971eeab2d Oct 31 04:13:39.368472 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:39.368420 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mrx9s"] Oct 31 04:13:39.371426 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:39.371391 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:13:39.371524 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:39.371453 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:13:39.371524 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:39.371484 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:13:39.665287 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:39.665199 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mrx9s" event={"ID":"70120ad8-48aa-44b4-b396-64eab56ce59b","Type":"ContainerStarted","Data":"1dea307cb0f07118a743766d464d20933d0bf91533edea2b93501e78a8f198ca"} Oct 31 04:13:39.665448 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:39.665288 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mrx9s" event={"ID":"70120ad8-48aa-44b4-b396-64eab56ce59b","Type":"ContainerStarted","Data":"52baa228b292f8319077488cc313e975602e3df8af32592d3e27433971eeab2d"} Oct 31 04:13:39.711640 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:39.711582 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mrx9s" podStartSLOduration=1.711570109 podStartE2EDuration="1.711570109s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:13:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:13:39.711437262 +0000 UTC m=+567.425771900" watchObservedRunningTime="2026-10-31 04:13:39.711570109 +0000 UTC m=+567.425904726" Oct 31 04:13:40.061361 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:40.061247 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mrx9s" Oct 31 04:13:40.062055 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:40.061964 4376 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-mrx9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Oct 31 04:13:40.062237 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:40.062209 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mrx9s" podUID="70120ad8-48aa-44b4-b396-64eab56ce59b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Oct 31 04:13:41.001851 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:13:41.001821 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 04:13:41.059799 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:41.059753 4376 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-mrx9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Oct 31 04:13:41.060006 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:41.059818 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mrx9s" podUID="70120ad8-48aa-44b4-b396-64eab56ce59b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Oct 31 04:13:42.059352 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:42.058715 4376 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-mrx9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Oct 31 04:13:42.059352 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:42.058776 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mrx9s" podUID="70120ad8-48aa-44b4-b396-64eab56ce59b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Oct 31 04:13:43.058708 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:43.058676 4376 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-mrx9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Oct 31 04:13:43.058960 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:43.058934 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mrx9s" podUID="70120ad8-48aa-44b4-b396-64eab56ce59b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Oct 31 04:13:44.062747 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:44.062703 4376 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-mrx9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Oct 31 04:13:44.063247 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:44.062758 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mrx9s" podUID="70120ad8-48aa-44b4-b396-64eab56ce59b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Oct 31 04:13:45.062704 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:45.062661 4376 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-mrx9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Oct 31 04:13:45.062910 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:45.062730 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mrx9s" podUID="70120ad8-48aa-44b4-b396-64eab56ce59b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Oct 31 04:13:46.059606 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:46.059572 4376 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-mrx9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Oct 31 04:13:46.059826 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:46.059625 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mrx9s" podUID="70120ad8-48aa-44b4-b396-64eab56ce59b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Oct 31 04:13:47.464200 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:47.464164 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mrx9s" Oct 31 04:13:47.464742 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:47.464547 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mrx9s" Oct 31 04:13:47.465321 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:47.465296 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mrx9s" Oct 31 04:13:54.771999 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:54.771966 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-16cda44a-openshift-default-778c87d957-665xc"] Oct 31 04:13:54.775601 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:54.775578 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-16cda44a-openshift-default-778c87d957-665xc" Oct 31 04:13:54.787045 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:54.787024 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/297621de-e8a2-4854-b28a-d3f21dac11b2-istiod-ca-cert\") pod \"e2e-ait-16cda44a-openshift-default-778c87d957-665xc\" (UID: \"297621de-e8a2-4854-b28a-d3f21dac11b2\") " pod="openshift-ingress/e2e-ait-16cda44a-openshift-default-778c87d957-665xc" Oct 31 04:13:54.787146 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:54.787059 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/297621de-e8a2-4854-b28a-d3f21dac11b2-credential-socket\") pod \"e2e-ait-16cda44a-openshift-default-778c87d957-665xc\" (UID: \"297621de-e8a2-4854-b28a-d3f21dac11b2\") " pod="openshift-ingress/e2e-ait-16cda44a-openshift-default-778c87d957-665xc" Oct 31 04:13:54.787241 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:54.787221 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/297621de-e8a2-4854-b28a-d3f21dac11b2-istio-data\") pod \"e2e-ait-16cda44a-openshift-default-778c87d957-665xc\" (UID: \"297621de-e8a2-4854-b28a-d3f21dac11b2\") " pod="openshift-ingress/e2e-ait-16cda44a-openshift-default-778c87d957-665xc" Oct 31 04:13:54.787283 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:54.787248 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/297621de-e8a2-4854-b28a-d3f21dac11b2-istio-podinfo\") pod \"e2e-ait-16cda44a-openshift-default-778c87d957-665xc\" (UID: \"297621de-e8a2-4854-b28a-d3f21dac11b2\") " pod="openshift-ingress/e2e-ait-16cda44a-openshift-default-778c87d957-665xc" Oct 31 04:13:54.787316 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:54.787288 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/297621de-e8a2-4854-b28a-d3f21dac11b2-workload-certs\") pod \"e2e-ait-16cda44a-openshift-default-778c87d957-665xc\" (UID: \"297621de-e8a2-4854-b28a-d3f21dac11b2\") " pod="openshift-ingress/e2e-ait-16cda44a-openshift-default-778c87d957-665xc" Oct 31 04:13:54.787316 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:54.787303 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/297621de-e8a2-4854-b28a-d3f21dac11b2-istio-envoy\") pod \"e2e-ait-16cda44a-openshift-default-778c87d957-665xc\" (UID: \"297621de-e8a2-4854-b28a-d3f21dac11b2\") " pod="openshift-ingress/e2e-ait-16cda44a-openshift-default-778c87d957-665xc" Oct 31 04:13:54.787451 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:54.787433 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skq7z\" (UniqueName: \"kubernetes.io/projected/297621de-e8a2-4854-b28a-d3f21dac11b2-kube-api-access-skq7z\") pod \"e2e-ait-16cda44a-openshift-default-778c87d957-665xc\" (UID: \"297621de-e8a2-4854-b28a-d3f21dac11b2\") " pod="openshift-ingress/e2e-ait-16cda44a-openshift-default-778c87d957-665xc" Oct 31 04:13:54.787488 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:54.787464 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/297621de-e8a2-4854-b28a-d3f21dac11b2-workload-socket\") pod \"e2e-ait-16cda44a-openshift-default-778c87d957-665xc\" (UID: \"297621de-e8a2-4854-b28a-d3f21dac11b2\") " pod="openshift-ingress/e2e-ait-16cda44a-openshift-default-778c87d957-665xc" Oct 31 04:13:54.787530 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:54.787519 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/297621de-e8a2-4854-b28a-d3f21dac11b2-istio-token\") pod \"e2e-ait-16cda44a-openshift-default-778c87d957-665xc\" (UID: \"297621de-e8a2-4854-b28a-d3f21dac11b2\") " pod="openshift-ingress/e2e-ait-16cda44a-openshift-default-778c87d957-665xc" Oct 31 04:13:54.842731 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:54.842674 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-16cda44a-openshift-default-dockercfg-w2vjb\"" Oct 31 04:13:54.844738 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:54.844716 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-16cda44a-openshift-default-778c87d957-665xc"] Oct 31 04:13:54.888940 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:54.888910 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/297621de-e8a2-4854-b28a-d3f21dac11b2-istiod-ca-cert\") pod \"e2e-ait-16cda44a-openshift-default-778c87d957-665xc\" (UID: \"297621de-e8a2-4854-b28a-d3f21dac11b2\") " pod="openshift-ingress/e2e-ait-16cda44a-openshift-default-778c87d957-665xc" Oct 31 04:13:54.888940 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:54.888950 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/297621de-e8a2-4854-b28a-d3f21dac11b2-credential-socket\") pod \"e2e-ait-16cda44a-openshift-default-778c87d957-665xc\" (UID: \"297621de-e8a2-4854-b28a-d3f21dac11b2\") " pod="openshift-ingress/e2e-ait-16cda44a-openshift-default-778c87d957-665xc" Oct 31 04:13:54.889204 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:54.889014 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/297621de-e8a2-4854-b28a-d3f21dac11b2-istio-data\") pod \"e2e-ait-16cda44a-openshift-default-778c87d957-665xc\" (UID: \"297621de-e8a2-4854-b28a-d3f21dac11b2\") " pod="openshift-ingress/e2e-ait-16cda44a-openshift-default-778c87d957-665xc" Oct 31 04:13:54.889204 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:54.889035 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/297621de-e8a2-4854-b28a-d3f21dac11b2-istio-podinfo\") pod \"e2e-ait-16cda44a-openshift-default-778c87d957-665xc\" (UID: \"297621de-e8a2-4854-b28a-d3f21dac11b2\") " pod="openshift-ingress/e2e-ait-16cda44a-openshift-default-778c87d957-665xc" Oct 31 04:13:54.889204 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:54.889098 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/297621de-e8a2-4854-b28a-d3f21dac11b2-workload-certs\") pod \"e2e-ait-16cda44a-openshift-default-778c87d957-665xc\" (UID: \"297621de-e8a2-4854-b28a-d3f21dac11b2\") " pod="openshift-ingress/e2e-ait-16cda44a-openshift-default-778c87d957-665xc" Oct 31 04:13:54.889204 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:54.889123 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/297621de-e8a2-4854-b28a-d3f21dac11b2-istio-envoy\") pod \"e2e-ait-16cda44a-openshift-default-778c87d957-665xc\" (UID: \"297621de-e8a2-4854-b28a-d3f21dac11b2\") " pod="openshift-ingress/e2e-ait-16cda44a-openshift-default-778c87d957-665xc" Oct 31 04:13:54.889456 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:54.889240 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skq7z\" (UniqueName: \"kubernetes.io/projected/297621de-e8a2-4854-b28a-d3f21dac11b2-kube-api-access-skq7z\") pod \"e2e-ait-16cda44a-openshift-default-778c87d957-665xc\" (UID: \"297621de-e8a2-4854-b28a-d3f21dac11b2\") " pod="openshift-ingress/e2e-ait-16cda44a-openshift-default-778c87d957-665xc" Oct 31 04:13:54.889456 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:54.889280 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/297621de-e8a2-4854-b28a-d3f21dac11b2-workload-socket\") pod \"e2e-ait-16cda44a-openshift-default-778c87d957-665xc\" (UID: \"297621de-e8a2-4854-b28a-d3f21dac11b2\") " pod="openshift-ingress/e2e-ait-16cda44a-openshift-default-778c87d957-665xc" Oct 31 04:13:54.889456 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:54.889337 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/297621de-e8a2-4854-b28a-d3f21dac11b2-istio-token\") pod \"e2e-ait-16cda44a-openshift-default-778c87d957-665xc\" (UID: \"297621de-e8a2-4854-b28a-d3f21dac11b2\") " pod="openshift-ingress/e2e-ait-16cda44a-openshift-default-778c87d957-665xc" Oct 31 04:13:54.890142 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:54.890115 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/297621de-e8a2-4854-b28a-d3f21dac11b2-workload-certs\") pod \"e2e-ait-16cda44a-openshift-default-778c87d957-665xc\" (UID: \"297621de-e8a2-4854-b28a-d3f21dac11b2\") " pod="openshift-ingress/e2e-ait-16cda44a-openshift-default-778c87d957-665xc" Oct 31 04:13:54.890303 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:54.890125 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/297621de-e8a2-4854-b28a-d3f21dac11b2-workload-socket\") pod \"e2e-ait-16cda44a-openshift-default-778c87d957-665xc\" (UID: \"297621de-e8a2-4854-b28a-d3f21dac11b2\") " pod="openshift-ingress/e2e-ait-16cda44a-openshift-default-778c87d957-665xc" Oct 31 04:13:54.890303 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:54.890248 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/297621de-e8a2-4854-b28a-d3f21dac11b2-credential-socket\") pod \"e2e-ait-16cda44a-openshift-default-778c87d957-665xc\" (UID: \"297621de-e8a2-4854-b28a-d3f21dac11b2\") " pod="openshift-ingress/e2e-ait-16cda44a-openshift-default-778c87d957-665xc" Oct 31 04:13:54.890753 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:54.890720 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/297621de-e8a2-4854-b28a-d3f21dac11b2-istio-data\") pod \"e2e-ait-16cda44a-openshift-default-778c87d957-665xc\" (UID: \"297621de-e8a2-4854-b28a-d3f21dac11b2\") " pod="openshift-ingress/e2e-ait-16cda44a-openshift-default-778c87d957-665xc" Oct 31 04:13:54.890859 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:54.890755 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/297621de-e8a2-4854-b28a-d3f21dac11b2-istiod-ca-cert\") pod \"e2e-ait-16cda44a-openshift-default-778c87d957-665xc\" (UID: \"297621de-e8a2-4854-b28a-d3f21dac11b2\") " pod="openshift-ingress/e2e-ait-16cda44a-openshift-default-778c87d957-665xc" Oct 31 04:13:54.892672 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:54.892531 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/297621de-e8a2-4854-b28a-d3f21dac11b2-istio-envoy\") pod \"e2e-ait-16cda44a-openshift-default-778c87d957-665xc\" (UID: \"297621de-e8a2-4854-b28a-d3f21dac11b2\") " pod="openshift-ingress/e2e-ait-16cda44a-openshift-default-778c87d957-665xc" Oct 31 04:13:54.893863 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:54.893837 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/297621de-e8a2-4854-b28a-d3f21dac11b2-istio-podinfo\") pod \"e2e-ait-16cda44a-openshift-default-778c87d957-665xc\" (UID: \"297621de-e8a2-4854-b28a-d3f21dac11b2\") " pod="openshift-ingress/e2e-ait-16cda44a-openshift-default-778c87d957-665xc" Oct 31 04:13:54.922357 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:54.922307 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-skq7z\" (UniqueName: \"kubernetes.io/projected/297621de-e8a2-4854-b28a-d3f21dac11b2-kube-api-access-skq7z\") pod \"e2e-ait-16cda44a-openshift-default-778c87d957-665xc\" (UID: \"297621de-e8a2-4854-b28a-d3f21dac11b2\") " pod="openshift-ingress/e2e-ait-16cda44a-openshift-default-778c87d957-665xc" Oct 31 04:13:54.922887 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:54.922869 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/297621de-e8a2-4854-b28a-d3f21dac11b2-istio-token\") pod \"e2e-ait-16cda44a-openshift-default-778c87d957-665xc\" (UID: \"297621de-e8a2-4854-b28a-d3f21dac11b2\") " pod="openshift-ingress/e2e-ait-16cda44a-openshift-default-778c87d957-665xc" Oct 31 04:13:55.088615 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:55.088548 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-16cda44a-openshift-default-778c87d957-665xc" Oct 31 04:13:55.405069 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:13:55.405022 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod297621de_e8a2_4854_b28a_d3f21dac11b2.slice/crio-da38413380a46a9b38c23f38d53ffc78a397415ce6f2ba007a0e7ce82d812735 WatchSource:0}: Error finding container da38413380a46a9b38c23f38d53ffc78a397415ce6f2ba007a0e7ce82d812735: Status 404 returned error can't find the container with id da38413380a46a9b38c23f38d53ffc78a397415ce6f2ba007a0e7ce82d812735 Oct 31 04:13:55.405234 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:55.405215 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-16cda44a-openshift-default-778c87d957-665xc"] Oct 31 04:13:55.407127 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:55.407086 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:13:55.407215 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:55.407173 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:13:55.407264 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:55.407220 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:13:55.732488 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:55.732412 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-16cda44a-openshift-default-778c87d957-665xc" event={"ID":"297621de-e8a2-4854-b28a-d3f21dac11b2","Type":"ContainerStarted","Data":"5e9e7eaaf07d50910d76080d0d0d6e2d64f1429637ab887542ddf4215059be4f"} Oct 31 04:13:55.732488 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:55.732462 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-16cda44a-openshift-default-778c87d957-665xc" event={"ID":"297621de-e8a2-4854-b28a-d3f21dac11b2","Type":"ContainerStarted","Data":"da38413380a46a9b38c23f38d53ffc78a397415ce6f2ba007a0e7ce82d812735"} Oct 31 04:13:55.779274 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:55.779226 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-16cda44a-openshift-default-778c87d957-665xc" podStartSLOduration=1.779212243 podStartE2EDuration="1.779212243s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:13:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:13:55.777627968 +0000 UTC m=+583.491962585" watchObservedRunningTime="2026-10-31 04:13:55.779212243 +0000 UTC m=+583.493546912" Oct 31 04:13:56.089561 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:56.089476 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-16cda44a-openshift-default-778c87d957-665xc" Oct 31 04:13:56.094440 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:56.094413 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-16cda44a-openshift-default-778c87d957-665xc" Oct 31 04:13:56.737249 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:56.737219 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-16cda44a-openshift-default-778c87d957-665xc" Oct 31 04:13:56.738467 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:56.738442 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-16cda44a-openshift-default-778c87d957-665xc" Oct 31 04:13:56.755553 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:56.755528 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-4805aacc-openshift-default-78cc674747-nljxj"] Oct 31 04:13:56.759167 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:56.759141 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-4805aacc-openshift-default-78cc674747-nljxj" Oct 31 04:13:56.776476 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:56.776443 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-4805aacc-openshift-default-dockercfg-gkssh\"" Oct 31 04:13:56.781835 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:56.781812 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-4805aacc-openshift-default-78cc674747-nljxj"] Oct 31 04:13:56.807115 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:56.807077 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f870408-c032-4347-af32-6a0a573aa5fe-istio-envoy\") pod \"e2e-ait-4805aacc-openshift-default-78cc674747-nljxj\" (UID: \"3f870408-c032-4347-af32-6a0a573aa5fe\") " pod="openshift-ingress/e2e-ait-4805aacc-openshift-default-78cc674747-nljxj" Oct 31 04:13:56.807285 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:56.807246 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9j466\" (UniqueName: \"kubernetes.io/projected/3f870408-c032-4347-af32-6a0a573aa5fe-kube-api-access-9j466\") pod \"e2e-ait-4805aacc-openshift-default-78cc674747-nljxj\" (UID: \"3f870408-c032-4347-af32-6a0a573aa5fe\") " pod="openshift-ingress/e2e-ait-4805aacc-openshift-default-78cc674747-nljxj" Oct 31 04:13:56.807395 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:56.807313 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f870408-c032-4347-af32-6a0a573aa5fe-istiod-ca-cert\") pod \"e2e-ait-4805aacc-openshift-default-78cc674747-nljxj\" (UID: \"3f870408-c032-4347-af32-6a0a573aa5fe\") " pod="openshift-ingress/e2e-ait-4805aacc-openshift-default-78cc674747-nljxj" Oct 31 04:13:56.807395 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:56.807364 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f870408-c032-4347-af32-6a0a573aa5fe-credential-socket\") pod \"e2e-ait-4805aacc-openshift-default-78cc674747-nljxj\" (UID: \"3f870408-c032-4347-af32-6a0a573aa5fe\") " pod="openshift-ingress/e2e-ait-4805aacc-openshift-default-78cc674747-nljxj" Oct 31 04:13:56.807823 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:56.807409 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f870408-c032-4347-af32-6a0a573aa5fe-istio-podinfo\") pod \"e2e-ait-4805aacc-openshift-default-78cc674747-nljxj\" (UID: \"3f870408-c032-4347-af32-6a0a573aa5fe\") " pod="openshift-ingress/e2e-ait-4805aacc-openshift-default-78cc674747-nljxj" Oct 31 04:13:56.807823 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:56.807440 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f870408-c032-4347-af32-6a0a573aa5fe-istio-token\") pod \"e2e-ait-4805aacc-openshift-default-78cc674747-nljxj\" (UID: \"3f870408-c032-4347-af32-6a0a573aa5fe\") " pod="openshift-ingress/e2e-ait-4805aacc-openshift-default-78cc674747-nljxj" Oct 31 04:13:56.807823 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:56.807500 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f870408-c032-4347-af32-6a0a573aa5fe-workload-socket\") pod \"e2e-ait-4805aacc-openshift-default-78cc674747-nljxj\" (UID: \"3f870408-c032-4347-af32-6a0a573aa5fe\") " pod="openshift-ingress/e2e-ait-4805aacc-openshift-default-78cc674747-nljxj" Oct 31 04:13:56.807823 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:56.807523 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f870408-c032-4347-af32-6a0a573aa5fe-workload-certs\") pod \"e2e-ait-4805aacc-openshift-default-78cc674747-nljxj\" (UID: \"3f870408-c032-4347-af32-6a0a573aa5fe\") " pod="openshift-ingress/e2e-ait-4805aacc-openshift-default-78cc674747-nljxj" Oct 31 04:13:56.807823 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:56.807545 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f870408-c032-4347-af32-6a0a573aa5fe-istio-data\") pod \"e2e-ait-4805aacc-openshift-default-78cc674747-nljxj\" (UID: \"3f870408-c032-4347-af32-6a0a573aa5fe\") " pod="openshift-ingress/e2e-ait-4805aacc-openshift-default-78cc674747-nljxj" Oct 31 04:13:56.908805 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:56.908773 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f870408-c032-4347-af32-6a0a573aa5fe-workload-socket\") pod \"e2e-ait-4805aacc-openshift-default-78cc674747-nljxj\" (UID: \"3f870408-c032-4347-af32-6a0a573aa5fe\") " pod="openshift-ingress/e2e-ait-4805aacc-openshift-default-78cc674747-nljxj" Oct 31 04:13:56.908805 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:56.908805 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f870408-c032-4347-af32-6a0a573aa5fe-workload-certs\") pod \"e2e-ait-4805aacc-openshift-default-78cc674747-nljxj\" (UID: \"3f870408-c032-4347-af32-6a0a573aa5fe\") " pod="openshift-ingress/e2e-ait-4805aacc-openshift-default-78cc674747-nljxj" Oct 31 04:13:56.909020 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:56.908828 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f870408-c032-4347-af32-6a0a573aa5fe-istio-data\") pod \"e2e-ait-4805aacc-openshift-default-78cc674747-nljxj\" (UID: \"3f870408-c032-4347-af32-6a0a573aa5fe\") " pod="openshift-ingress/e2e-ait-4805aacc-openshift-default-78cc674747-nljxj" Oct 31 04:13:56.909020 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:56.908871 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f870408-c032-4347-af32-6a0a573aa5fe-istio-envoy\") pod \"e2e-ait-4805aacc-openshift-default-78cc674747-nljxj\" (UID: \"3f870408-c032-4347-af32-6a0a573aa5fe\") " pod="openshift-ingress/e2e-ait-4805aacc-openshift-default-78cc674747-nljxj" Oct 31 04:13:56.909020 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:56.908915 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9j466\" (UniqueName: \"kubernetes.io/projected/3f870408-c032-4347-af32-6a0a573aa5fe-kube-api-access-9j466\") pod \"e2e-ait-4805aacc-openshift-default-78cc674747-nljxj\" (UID: \"3f870408-c032-4347-af32-6a0a573aa5fe\") " pod="openshift-ingress/e2e-ait-4805aacc-openshift-default-78cc674747-nljxj" Oct 31 04:13:56.909145 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:56.909044 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f870408-c032-4347-af32-6a0a573aa5fe-istiod-ca-cert\") pod \"e2e-ait-4805aacc-openshift-default-78cc674747-nljxj\" (UID: \"3f870408-c032-4347-af32-6a0a573aa5fe\") " pod="openshift-ingress/e2e-ait-4805aacc-openshift-default-78cc674747-nljxj" Oct 31 04:13:56.909145 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:56.909081 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f870408-c032-4347-af32-6a0a573aa5fe-credential-socket\") pod \"e2e-ait-4805aacc-openshift-default-78cc674747-nljxj\" (UID: \"3f870408-c032-4347-af32-6a0a573aa5fe\") " pod="openshift-ingress/e2e-ait-4805aacc-openshift-default-78cc674747-nljxj" Oct 31 04:13:56.909145 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:56.909110 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f870408-c032-4347-af32-6a0a573aa5fe-istio-podinfo\") pod \"e2e-ait-4805aacc-openshift-default-78cc674747-nljxj\" (UID: \"3f870408-c032-4347-af32-6a0a573aa5fe\") " pod="openshift-ingress/e2e-ait-4805aacc-openshift-default-78cc674747-nljxj" Oct 31 04:13:56.909145 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:56.909139 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f870408-c032-4347-af32-6a0a573aa5fe-istio-token\") pod \"e2e-ait-4805aacc-openshift-default-78cc674747-nljxj\" (UID: \"3f870408-c032-4347-af32-6a0a573aa5fe\") " pod="openshift-ingress/e2e-ait-4805aacc-openshift-default-78cc674747-nljxj" Oct 31 04:13:56.909385 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:56.909262 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f870408-c032-4347-af32-6a0a573aa5fe-workload-certs\") pod \"e2e-ait-4805aacc-openshift-default-78cc674747-nljxj\" (UID: \"3f870408-c032-4347-af32-6a0a573aa5fe\") " pod="openshift-ingress/e2e-ait-4805aacc-openshift-default-78cc674747-nljxj" Oct 31 04:13:56.909385 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:56.909289 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f870408-c032-4347-af32-6a0a573aa5fe-istio-data\") pod \"e2e-ait-4805aacc-openshift-default-78cc674747-nljxj\" (UID: \"3f870408-c032-4347-af32-6a0a573aa5fe\") " pod="openshift-ingress/e2e-ait-4805aacc-openshift-default-78cc674747-nljxj" Oct 31 04:13:56.909514 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:56.909405 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f870408-c032-4347-af32-6a0a573aa5fe-workload-socket\") pod \"e2e-ait-4805aacc-openshift-default-78cc674747-nljxj\" (UID: \"3f870408-c032-4347-af32-6a0a573aa5fe\") " pod="openshift-ingress/e2e-ait-4805aacc-openshift-default-78cc674747-nljxj" Oct 31 04:13:56.909568 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:56.909517 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f870408-c032-4347-af32-6a0a573aa5fe-credential-socket\") pod \"e2e-ait-4805aacc-openshift-default-78cc674747-nljxj\" (UID: \"3f870408-c032-4347-af32-6a0a573aa5fe\") " pod="openshift-ingress/e2e-ait-4805aacc-openshift-default-78cc674747-nljxj" Oct 31 04:13:56.909790 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:56.909770 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f870408-c032-4347-af32-6a0a573aa5fe-istiod-ca-cert\") pod \"e2e-ait-4805aacc-openshift-default-78cc674747-nljxj\" (UID: \"3f870408-c032-4347-af32-6a0a573aa5fe\") " pod="openshift-ingress/e2e-ait-4805aacc-openshift-default-78cc674747-nljxj" Oct 31 04:13:56.911187 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:56.911163 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f870408-c032-4347-af32-6a0a573aa5fe-istio-envoy\") pod \"e2e-ait-4805aacc-openshift-default-78cc674747-nljxj\" (UID: \"3f870408-c032-4347-af32-6a0a573aa5fe\") " pod="openshift-ingress/e2e-ait-4805aacc-openshift-default-78cc674747-nljxj" Oct 31 04:13:56.911448 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:56.911433 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f870408-c032-4347-af32-6a0a573aa5fe-istio-podinfo\") pod \"e2e-ait-4805aacc-openshift-default-78cc674747-nljxj\" (UID: \"3f870408-c032-4347-af32-6a0a573aa5fe\") " pod="openshift-ingress/e2e-ait-4805aacc-openshift-default-78cc674747-nljxj" Oct 31 04:13:56.934180 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:56.934155 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f870408-c032-4347-af32-6a0a573aa5fe-istio-token\") pod \"e2e-ait-4805aacc-openshift-default-78cc674747-nljxj\" (UID: \"3f870408-c032-4347-af32-6a0a573aa5fe\") " pod="openshift-ingress/e2e-ait-4805aacc-openshift-default-78cc674747-nljxj" Oct 31 04:13:56.934406 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:56.934383 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9j466\" (UniqueName: \"kubernetes.io/projected/3f870408-c032-4347-af32-6a0a573aa5fe-kube-api-access-9j466\") pod \"e2e-ait-4805aacc-openshift-default-78cc674747-nljxj\" (UID: \"3f870408-c032-4347-af32-6a0a573aa5fe\") " pod="openshift-ingress/e2e-ait-4805aacc-openshift-default-78cc674747-nljxj" Oct 31 04:13:57.076139 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:57.076060 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-4805aacc-openshift-default-78cc674747-nljxj" Oct 31 04:13:57.385032 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:57.385005 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-4805aacc-openshift-default-78cc674747-nljxj"] Oct 31 04:13:57.385382 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:13:57.385358 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f870408_c032_4347_af32_6a0a573aa5fe.slice/crio-fe8c0d2d9a71a597baa03c8d6299ea47db0528f9a276fa68bfa05be49db7e97d WatchSource:0}: Error finding container fe8c0d2d9a71a597baa03c8d6299ea47db0528f9a276fa68bfa05be49db7e97d: Status 404 returned error can't find the container with id fe8c0d2d9a71a597baa03c8d6299ea47db0528f9a276fa68bfa05be49db7e97d Oct 31 04:13:57.387561 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:57.387527 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:13:57.387674 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:57.387602 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:13:57.387674 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:57.387651 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:13:57.742776 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:57.742686 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4805aacc-openshift-default-78cc674747-nljxj" event={"ID":"3f870408-c032-4347-af32-6a0a573aa5fe","Type":"ContainerStarted","Data":"8e1655be4ab5787d2a2b1f7d8e19850d2afb27ac91d330e34c268a7cdb30b3f1"} Oct 31 04:13:57.742776 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:57.742738 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4805aacc-openshift-default-78cc674747-nljxj" event={"ID":"3f870408-c032-4347-af32-6a0a573aa5fe","Type":"ContainerStarted","Data":"fe8c0d2d9a71a597baa03c8d6299ea47db0528f9a276fa68bfa05be49db7e97d"} Oct 31 04:13:57.790208 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:57.790149 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-4805aacc-openshift-default-78cc674747-nljxj" podStartSLOduration=1.7901241799999998 podStartE2EDuration="1.79012418s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:13:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:13:57.789082461 +0000 UTC m=+585.503417078" watchObservedRunningTime="2026-10-31 04:13:57.79012418 +0000 UTC m=+585.504458797" Oct 31 04:13:58.076723 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:58.076642 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-4805aacc-openshift-default-78cc674747-nljxj" Oct 31 04:13:58.082173 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:58.082145 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-4805aacc-openshift-default-78cc674747-nljxj" Oct 31 04:13:58.747354 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:58.747304 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-4805aacc-openshift-default-78cc674747-nljxj" Oct 31 04:13:58.748149 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:58.748127 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-4805aacc-openshift-default-78cc674747-nljxj" Oct 31 04:13:59.381956 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:59.380158 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-16cda44a-59f4f898bc-lhxq5"] Oct 31 04:13:59.391430 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:59.391398 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-16cda44a-59f4f898bc-lhxq5" Oct 31 04:13:59.401177 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:59.401151 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-16cda44a-59f4f898bc-lhxq5"] Oct 31 04:13:59.406968 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:59.406943 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-16cda44a\"" Oct 31 04:13:59.407111 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:59.407049 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-16cda44a-dockercfg-5q7r8\"" Oct 31 04:13:59.433320 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:59.433273 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd0487ed-0a23-45ad-828d-114d26ec873e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-16cda44a-59f4f898bc-lhxq5\" (UID: \"dd0487ed-0a23-45ad-828d-114d26ec873e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-16cda44a-59f4f898bc-lhxq5" Oct 31 04:13:59.433609 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:59.433433 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phrw4\" (UniqueName: \"kubernetes.io/projected/dd0487ed-0a23-45ad-828d-114d26ec873e-kube-api-access-phrw4\") pod \"payload-pre-processing-e2e-ait-16cda44a-59f4f898bc-lhxq5\" (UID: \"dd0487ed-0a23-45ad-828d-114d26ec873e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-16cda44a-59f4f898bc-lhxq5" Oct 31 04:13:59.478594 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:59.478562 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-16cda44a-589975f687-78vsg"] Oct 31 04:13:59.483590 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:59.483530 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-16cda44a-589975f687-78vsg" Oct 31 04:13:59.500431 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:59.500401 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-16cda44a-589975f687-78vsg"] Oct 31 04:13:59.534550 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:59.534511 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4vsc\" (UniqueName: \"kubernetes.io/projected/ab382e64-a6ee-4a64-8e7d-b99274571ab6-kube-api-access-w4vsc\") pod \"payload-processing-e2e-ait-16cda44a-589975f687-78vsg\" (UID: \"ab382e64-a6ee-4a64-8e7d-b99274571ab6\") " pod="openshift-ingress/payload-processing-e2e-ait-16cda44a-589975f687-78vsg" Oct 31 04:13:59.534702 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:59.534646 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd0487ed-0a23-45ad-828d-114d26ec873e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-16cda44a-59f4f898bc-lhxq5\" (UID: \"dd0487ed-0a23-45ad-828d-114d26ec873e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-16cda44a-59f4f898bc-lhxq5" Oct 31 04:13:59.534753 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:59.534695 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phrw4\" (UniqueName: \"kubernetes.io/projected/dd0487ed-0a23-45ad-828d-114d26ec873e-kube-api-access-phrw4\") pod \"payload-pre-processing-e2e-ait-16cda44a-59f4f898bc-lhxq5\" (UID: \"dd0487ed-0a23-45ad-828d-114d26ec873e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-16cda44a-59f4f898bc-lhxq5" Oct 31 04:13:59.534804 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:59.534753 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab382e64-a6ee-4a64-8e7d-b99274571ab6-plugins-config-volume\") pod \"payload-processing-e2e-ait-16cda44a-589975f687-78vsg\" (UID: \"ab382e64-a6ee-4a64-8e7d-b99274571ab6\") " pod="openshift-ingress/payload-processing-e2e-ait-16cda44a-589975f687-78vsg" Oct 31 04:13:59.535342 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:59.535310 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd0487ed-0a23-45ad-828d-114d26ec873e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-16cda44a-59f4f898bc-lhxq5\" (UID: \"dd0487ed-0a23-45ad-828d-114d26ec873e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-16cda44a-59f4f898bc-lhxq5" Oct 31 04:13:59.560918 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:59.560890 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phrw4\" (UniqueName: \"kubernetes.io/projected/dd0487ed-0a23-45ad-828d-114d26ec873e-kube-api-access-phrw4\") pod \"payload-pre-processing-e2e-ait-16cda44a-59f4f898bc-lhxq5\" (UID: \"dd0487ed-0a23-45ad-828d-114d26ec873e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-16cda44a-59f4f898bc-lhxq5" Oct 31 04:13:59.635955 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:59.635864 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab382e64-a6ee-4a64-8e7d-b99274571ab6-plugins-config-volume\") pod \"payload-processing-e2e-ait-16cda44a-589975f687-78vsg\" (UID: \"ab382e64-a6ee-4a64-8e7d-b99274571ab6\") " pod="openshift-ingress/payload-processing-e2e-ait-16cda44a-589975f687-78vsg" Oct 31 04:13:59.635955 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:59.635912 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4vsc\" (UniqueName: \"kubernetes.io/projected/ab382e64-a6ee-4a64-8e7d-b99274571ab6-kube-api-access-w4vsc\") pod \"payload-processing-e2e-ait-16cda44a-589975f687-78vsg\" (UID: \"ab382e64-a6ee-4a64-8e7d-b99274571ab6\") " pod="openshift-ingress/payload-processing-e2e-ait-16cda44a-589975f687-78vsg" Oct 31 04:13:59.636691 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:59.636656 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab382e64-a6ee-4a64-8e7d-b99274571ab6-plugins-config-volume\") pod \"payload-processing-e2e-ait-16cda44a-589975f687-78vsg\" (UID: \"ab382e64-a6ee-4a64-8e7d-b99274571ab6\") " pod="openshift-ingress/payload-processing-e2e-ait-16cda44a-589975f687-78vsg" Oct 31 04:13:59.660168 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:59.660131 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4vsc\" (UniqueName: \"kubernetes.io/projected/ab382e64-a6ee-4a64-8e7d-b99274571ab6-kube-api-access-w4vsc\") pod \"payload-processing-e2e-ait-16cda44a-589975f687-78vsg\" (UID: \"ab382e64-a6ee-4a64-8e7d-b99274571ab6\") " pod="openshift-ingress/payload-processing-e2e-ait-16cda44a-589975f687-78vsg" Oct 31 04:13:59.707983 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:59.707950 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-16cda44a-59f4f898bc-lhxq5" Oct 31 04:13:59.815511 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:13:59.815470 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-16cda44a-589975f687-78vsg" Oct 31 04:14:00.010697 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:14:00.010667 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd0487ed_0a23_45ad_828d_114d26ec873e.slice/crio-2bc6b1fe7da563bf1dccd103335f709666fe7f29f5db99622edfc17d0549fb10 WatchSource:0}: Error finding container 2bc6b1fe7da563bf1dccd103335f709666fe7f29f5db99622edfc17d0549fb10: Status 404 returned error can't find the container with id 2bc6b1fe7da563bf1dccd103335f709666fe7f29f5db99622edfc17d0549fb10 Oct 31 04:14:00.010878 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:00.010858 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-16cda44a-59f4f898bc-lhxq5"] Oct 31 04:14:00.114555 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:00.114529 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-16cda44a-589975f687-78vsg"] Oct 31 04:14:00.115275 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:14:00.115250 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab382e64_a6ee_4a64_8e7d_b99274571ab6.slice/crio-77bb5f8e11a4eeae60888d32bcf66987f4c53dba90232293bc682702ec725260 WatchSource:0}: Error finding container 77bb5f8e11a4eeae60888d32bcf66987f4c53dba90232293bc682702ec725260: Status 404 returned error can't find the container with id 77bb5f8e11a4eeae60888d32bcf66987f4c53dba90232293bc682702ec725260 Oct 31 04:14:00.757664 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:00.757625 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-16cda44a-59f4f898bc-lhxq5" event={"ID":"dd0487ed-0a23-45ad-828d-114d26ec873e","Type":"ContainerStarted","Data":"0a9f2038e707a9da5d67e2458c35ab7d279e87cf86f3cdcba96ab2c40fe29357"} Oct 31 04:14:00.758123 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:00.757696 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-16cda44a-59f4f898bc-lhxq5" Oct 31 04:14:00.758123 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:00.757719 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-16cda44a-59f4f898bc-lhxq5" event={"ID":"dd0487ed-0a23-45ad-828d-114d26ec873e","Type":"ContainerStarted","Data":"2bc6b1fe7da563bf1dccd103335f709666fe7f29f5db99622edfc17d0549fb10"} Oct 31 04:14:00.759225 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:00.759199 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-16cda44a-589975f687-78vsg" event={"ID":"ab382e64-a6ee-4a64-8e7d-b99274571ab6","Type":"ContainerStarted","Data":"c33f21f7efb4c1753bf498896d541350214367b38100640551aeb12e24fd5036"} Oct 31 04:14:00.759365 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:00.759248 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-16cda44a-589975f687-78vsg" event={"ID":"ab382e64-a6ee-4a64-8e7d-b99274571ab6","Type":"ContainerStarted","Data":"77bb5f8e11a4eeae60888d32bcf66987f4c53dba90232293bc682702ec725260"} Oct 31 04:14:00.799532 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:00.799478 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-16cda44a-59f4f898bc-lhxq5" podStartSLOduration=1.799464152 podStartE2EDuration="1.799464152s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:13:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:14:00.799215253 +0000 UTC m=+588.513549882" watchObservedRunningTime="2026-10-31 04:14:00.799464152 +0000 UTC m=+588.513798769" Oct 31 04:14:00.845133 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:00.845081 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-16cda44a-589975f687-78vsg" podStartSLOduration=1.84506654 podStartE2EDuration="1.84506654s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:13:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:14:00.843757265 +0000 UTC m=+588.558091882" watchObservedRunningTime="2026-10-31 04:14:00.84506654 +0000 UTC m=+588.559401156" Oct 31 04:14:01.054742 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:01.054650 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-a0d1a61d-58fd568b6c-kch6z"] Oct 31 04:14:01.059058 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:01.059037 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-a0d1a61d-58fd568b6c-kch6z" Oct 31 04:14:01.074394 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:01.074363 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-a0d1a61d-dockercfg-2rwjr\"" Oct 31 04:14:01.074394 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:01.074364 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-a0d1a61d\"" Oct 31 04:14:01.075270 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:01.075248 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-a0d1a61d-58fd568b6c-kch6z"] Oct 31 04:14:01.156658 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:01.156621 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/850a4333-5224-4732-967d-0b19bbca006f-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-a0d1a61d-58fd568b6c-kch6z\" (UID: \"850a4333-5224-4732-967d-0b19bbca006f\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-a0d1a61d-58fd568b6c-kch6z" Oct 31 04:14:01.158668 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:01.158642 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plz24\" (UniqueName: \"kubernetes.io/projected/850a4333-5224-4732-967d-0b19bbca006f-kube-api-access-plz24\") pod \"payload-pre-processing-e2e-migrate-a0d1a61d-58fd568b6c-kch6z\" (UID: \"850a4333-5224-4732-967d-0b19bbca006f\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-a0d1a61d-58fd568b6c-kch6z" Oct 31 04:14:01.259821 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:01.259789 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plz24\" (UniqueName: \"kubernetes.io/projected/850a4333-5224-4732-967d-0b19bbca006f-kube-api-access-plz24\") pod \"payload-pre-processing-e2e-migrate-a0d1a61d-58fd568b6c-kch6z\" (UID: \"850a4333-5224-4732-967d-0b19bbca006f\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-a0d1a61d-58fd568b6c-kch6z" Oct 31 04:14:01.259987 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:01.259866 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/850a4333-5224-4732-967d-0b19bbca006f-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-a0d1a61d-58fd568b6c-kch6z\" (UID: \"850a4333-5224-4732-967d-0b19bbca006f\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-a0d1a61d-58fd568b6c-kch6z" Oct 31 04:14:01.260523 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:01.260505 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/850a4333-5224-4732-967d-0b19bbca006f-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-a0d1a61d-58fd568b6c-kch6z\" (UID: \"850a4333-5224-4732-967d-0b19bbca006f\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-a0d1a61d-58fd568b6c-kch6z" Oct 31 04:14:01.286285 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:01.286253 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-plz24\" (UniqueName: \"kubernetes.io/projected/850a4333-5224-4732-967d-0b19bbca006f-kube-api-access-plz24\") pod \"payload-pre-processing-e2e-migrate-a0d1a61d-58fd568b6c-kch6z\" (UID: \"850a4333-5224-4732-967d-0b19bbca006f\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-a0d1a61d-58fd568b6c-kch6z" Oct 31 04:14:01.372509 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:01.372482 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-a0d1a61d-58fd568b6c-kch6z" Oct 31 04:14:01.672748 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:01.672721 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-a0d1a61d-58fd568b6c-kch6z"] Oct 31 04:14:01.673088 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:14:01.673066 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod850a4333_5224_4732_967d_0b19bbca006f.slice/crio-9850bdcd3eec7602a2db2349f7a2d23e2461af9287f4914aea927c7ae3a6fbc2 WatchSource:0}: Error finding container 9850bdcd3eec7602a2db2349f7a2d23e2461af9287f4914aea927c7ae3a6fbc2: Status 404 returned error can't find the container with id 9850bdcd3eec7602a2db2349f7a2d23e2461af9287f4914aea927c7ae3a6fbc2 Oct 31 04:14:01.765832 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:01.765792 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-a0d1a61d-58fd568b6c-kch6z" event={"ID":"850a4333-5224-4732-967d-0b19bbca006f","Type":"ContainerStarted","Data":"2f645c60a21d92fa291d3e499e3f4dfdcad047b04869db9aaafe92545dc89152"} Oct 31 04:14:01.766259 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:01.765857 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-a0d1a61d-58fd568b6c-kch6z" event={"ID":"850a4333-5224-4732-967d-0b19bbca006f","Type":"ContainerStarted","Data":"9850bdcd3eec7602a2db2349f7a2d23e2461af9287f4914aea927c7ae3a6fbc2"} Oct 31 04:14:01.766259 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:01.766167 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-a0d1a61d-58fd568b6c-kch6z" Oct 31 04:14:01.766259 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:01.766206 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-16cda44a-589975f687-78vsg" Oct 31 04:14:01.808570 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:01.808517 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-a0d1a61d-58fd568b6c-kch6z" podStartSLOduration=0.808501831 podStartE2EDuration="808.501831ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:14:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:14:01.80655024 +0000 UTC m=+589.520884859" watchObservedRunningTime="2026-10-31 04:14:01.808501831 +0000 UTC m=+589.522836461" Oct 31 04:14:02.397100 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:02.397069 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 04:14:03.493027 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:03.492993 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-44f032ad-7b89c4449f-gjfv8"] Oct 31 04:14:03.496820 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:03.496799 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-44f032ad-7b89c4449f-gjfv8" Oct 31 04:14:03.515531 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:03.515505 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-44f032ad-7b89c4449f-gjfv8"] Oct 31 04:14:03.526025 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:03.526003 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-44f032ad\"" Oct 31 04:14:03.526156 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:03.526091 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-44f032ad-dockercfg-bz68f\"" Oct 31 04:14:03.608451 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:03.608409 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-44f032ad-868c96f9c9-fvrrj"] Oct 31 04:14:03.616290 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:03.616269 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-44f032ad-868c96f9c9-fvrrj"] Oct 31 04:14:03.616559 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:03.616513 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-44f032ad-868c96f9c9-fvrrj" Oct 31 04:14:03.700792 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:03.700762 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvtzx\" (UniqueName: \"kubernetes.io/projected/9b428b05-ad88-44cc-b719-85d7cd8ae9b2-kube-api-access-vvtzx\") pod \"payload-pre-processing-e2e-ait-44f032ad-7b89c4449f-gjfv8\" (UID: \"9b428b05-ad88-44cc-b719-85d7cd8ae9b2\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-44f032ad-7b89c4449f-gjfv8" Oct 31 04:14:03.700792 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:03.700803 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b428b05-ad88-44cc-b719-85d7cd8ae9b2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-44f032ad-7b89c4449f-gjfv8\" (UID: \"9b428b05-ad88-44cc-b719-85d7cd8ae9b2\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-44f032ad-7b89c4449f-gjfv8" Oct 31 04:14:03.801409 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:03.801303 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd9af128-7cf1-4c41-96f1-c159eaa13692-plugins-config-volume\") pod \"payload-processing-e2e-ait-44f032ad-868c96f9c9-fvrrj\" (UID: \"cd9af128-7cf1-4c41-96f1-c159eaa13692\") " pod="openshift-ingress/payload-processing-e2e-ait-44f032ad-868c96f9c9-fvrrj" Oct 31 04:14:03.801409 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:03.801371 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvtzx\" (UniqueName: \"kubernetes.io/projected/9b428b05-ad88-44cc-b719-85d7cd8ae9b2-kube-api-access-vvtzx\") pod \"payload-pre-processing-e2e-ait-44f032ad-7b89c4449f-gjfv8\" (UID: \"9b428b05-ad88-44cc-b719-85d7cd8ae9b2\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-44f032ad-7b89c4449f-gjfv8" Oct 31 04:14:03.801623 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:03.801479 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b428b05-ad88-44cc-b719-85d7cd8ae9b2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-44f032ad-7b89c4449f-gjfv8\" (UID: \"9b428b05-ad88-44cc-b719-85d7cd8ae9b2\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-44f032ad-7b89c4449f-gjfv8" Oct 31 04:14:03.801665 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:03.801631 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmbd2\" (UniqueName: \"kubernetes.io/projected/cd9af128-7cf1-4c41-96f1-c159eaa13692-kube-api-access-nmbd2\") pod \"payload-processing-e2e-ait-44f032ad-868c96f9c9-fvrrj\" (UID: \"cd9af128-7cf1-4c41-96f1-c159eaa13692\") " pod="openshift-ingress/payload-processing-e2e-ait-44f032ad-868c96f9c9-fvrrj" Oct 31 04:14:03.802196 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:03.802178 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b428b05-ad88-44cc-b719-85d7cd8ae9b2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-44f032ad-7b89c4449f-gjfv8\" (UID: \"9b428b05-ad88-44cc-b719-85d7cd8ae9b2\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-44f032ad-7b89c4449f-gjfv8" Oct 31 04:14:03.825063 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:03.825036 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvtzx\" (UniqueName: \"kubernetes.io/projected/9b428b05-ad88-44cc-b719-85d7cd8ae9b2-kube-api-access-vvtzx\") pod \"payload-pre-processing-e2e-ait-44f032ad-7b89c4449f-gjfv8\" (UID: \"9b428b05-ad88-44cc-b719-85d7cd8ae9b2\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-44f032ad-7b89c4449f-gjfv8" Oct 31 04:14:03.902196 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:03.902160 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmbd2\" (UniqueName: \"kubernetes.io/projected/cd9af128-7cf1-4c41-96f1-c159eaa13692-kube-api-access-nmbd2\") pod \"payload-processing-e2e-ait-44f032ad-868c96f9c9-fvrrj\" (UID: \"cd9af128-7cf1-4c41-96f1-c159eaa13692\") " pod="openshift-ingress/payload-processing-e2e-ait-44f032ad-868c96f9c9-fvrrj" Oct 31 04:14:03.902388 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:03.902217 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd9af128-7cf1-4c41-96f1-c159eaa13692-plugins-config-volume\") pod \"payload-processing-e2e-ait-44f032ad-868c96f9c9-fvrrj\" (UID: \"cd9af128-7cf1-4c41-96f1-c159eaa13692\") " pod="openshift-ingress/payload-processing-e2e-ait-44f032ad-868c96f9c9-fvrrj" Oct 31 04:14:03.902920 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:03.902901 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd9af128-7cf1-4c41-96f1-c159eaa13692-plugins-config-volume\") pod \"payload-processing-e2e-ait-44f032ad-868c96f9c9-fvrrj\" (UID: \"cd9af128-7cf1-4c41-96f1-c159eaa13692\") " pod="openshift-ingress/payload-processing-e2e-ait-44f032ad-868c96f9c9-fvrrj" Oct 31 04:14:03.925753 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:03.925724 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmbd2\" (UniqueName: \"kubernetes.io/projected/cd9af128-7cf1-4c41-96f1-c159eaa13692-kube-api-access-nmbd2\") pod \"payload-processing-e2e-ait-44f032ad-868c96f9c9-fvrrj\" (UID: \"cd9af128-7cf1-4c41-96f1-c159eaa13692\") " pod="openshift-ingress/payload-processing-e2e-ait-44f032ad-868c96f9c9-fvrrj" Oct 31 04:14:03.930559 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:03.930532 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-44f032ad-868c96f9c9-fvrrj" Oct 31 04:14:04.108867 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:04.108791 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-44f032ad-7b89c4449f-gjfv8" Oct 31 04:14:04.267732 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:04.267704 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-44f032ad-868c96f9c9-fvrrj"] Oct 31 04:14:04.270178 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:14:04.270155 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd9af128_7cf1_4c41_96f1_c159eaa13692.slice/crio-92ebf5bbf0af4abb8d61a6159cee6795848aee89e441a6c7f4554aea51fd63b9 WatchSource:0}: Error finding container 92ebf5bbf0af4abb8d61a6159cee6795848aee89e441a6c7f4554aea51fd63b9: Status 404 returned error can't find the container with id 92ebf5bbf0af4abb8d61a6159cee6795848aee89e441a6c7f4554aea51fd63b9 Oct 31 04:14:04.368755 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:04.368712 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4805aacc-77dfbbd755-hd54k"] Oct 31 04:14:04.373187 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:04.373166 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-4805aacc-77dfbbd755-hd54k" Oct 31 04:14:04.387318 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:04.387293 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4805aacc-77dfbbd755-hd54k"] Oct 31 04:14:04.410018 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:04.409981 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79fd2619-b05f-4da8-862d-a0e3e224bea3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-4805aacc-77dfbbd755-hd54k\" (UID: \"79fd2619-b05f-4da8-862d-a0e3e224bea3\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4805aacc-77dfbbd755-hd54k" Oct 31 04:14:04.410120 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:04.410092 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnjm9\" (UniqueName: \"kubernetes.io/projected/79fd2619-b05f-4da8-862d-a0e3e224bea3-kube-api-access-gnjm9\") pod \"payload-pre-processing-e2e-ait-4805aacc-77dfbbd755-hd54k\" (UID: \"79fd2619-b05f-4da8-862d-a0e3e224bea3\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4805aacc-77dfbbd755-hd54k" Oct 31 04:14:04.436348 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:04.436300 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-4805aacc\"" Oct 31 04:14:04.436497 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:04.436473 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-4805aacc-dockercfg-h4gwn\"" Oct 31 04:14:04.499572 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:04.499549 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-44f032ad-7b89c4449f-gjfv8"] Oct 31 04:14:04.510713 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:04.510691 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnjm9\" (UniqueName: \"kubernetes.io/projected/79fd2619-b05f-4da8-862d-a0e3e224bea3-kube-api-access-gnjm9\") pod \"payload-pre-processing-e2e-ait-4805aacc-77dfbbd755-hd54k\" (UID: \"79fd2619-b05f-4da8-862d-a0e3e224bea3\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4805aacc-77dfbbd755-hd54k" Oct 31 04:14:04.510813 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:04.510770 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79fd2619-b05f-4da8-862d-a0e3e224bea3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-4805aacc-77dfbbd755-hd54k\" (UID: \"79fd2619-b05f-4da8-862d-a0e3e224bea3\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4805aacc-77dfbbd755-hd54k" Oct 31 04:14:04.511428 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:04.511406 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79fd2619-b05f-4da8-862d-a0e3e224bea3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-4805aacc-77dfbbd755-hd54k\" (UID: \"79fd2619-b05f-4da8-862d-a0e3e224bea3\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4805aacc-77dfbbd755-hd54k" Oct 31 04:14:04.535493 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:04.535474 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnjm9\" (UniqueName: \"kubernetes.io/projected/79fd2619-b05f-4da8-862d-a0e3e224bea3-kube-api-access-gnjm9\") pod \"payload-pre-processing-e2e-ait-4805aacc-77dfbbd755-hd54k\" (UID: \"79fd2619-b05f-4da8-862d-a0e3e224bea3\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4805aacc-77dfbbd755-hd54k" Oct 31 04:14:04.687702 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:04.687622 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-4805aacc-77dfbbd755-hd54k" Oct 31 04:14:04.805310 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:04.804860 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-44f032ad-7b89c4449f-gjfv8" event={"ID":"9b428b05-ad88-44cc-b719-85d7cd8ae9b2","Type":"ContainerStarted","Data":"6dea152092f5b56b46ba44229b2cbb8615777cb24ece1601cbe10ee6b3c7871b"} Oct 31 04:14:04.805310 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:04.804965 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-44f032ad-7b89c4449f-gjfv8" event={"ID":"9b428b05-ad88-44cc-b719-85d7cd8ae9b2","Type":"ContainerStarted","Data":"562b10620e58c9c580410786b6675d3ab0ef0bf927a1a03dbab5c5b0588b647b"} Oct 31 04:14:04.805310 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:04.805022 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-44f032ad-7b89c4449f-gjfv8" Oct 31 04:14:04.806974 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:04.806502 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-44f032ad-868c96f9c9-fvrrj" event={"ID":"cd9af128-7cf1-4c41-96f1-c159eaa13692","Type":"ContainerStarted","Data":"88d1c735e4c49ecbb2717bed6b8c4d719b62ff35eeece0965899c73929d0b809"} Oct 31 04:14:04.806974 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:04.806544 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-44f032ad-868c96f9c9-fvrrj" event={"ID":"cd9af128-7cf1-4c41-96f1-c159eaa13692","Type":"ContainerStarted","Data":"92ebf5bbf0af4abb8d61a6159cee6795848aee89e441a6c7f4554aea51fd63b9"} Oct 31 04:14:04.806974 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:04.806941 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-44f032ad-868c96f9c9-fvrrj" Oct 31 04:14:04.906452 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:04.906399 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-44f032ad-7b89c4449f-gjfv8" podStartSLOduration=1.906386676 podStartE2EDuration="1.906386676s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:14:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:14:04.864620841 +0000 UTC m=+592.578955474" watchObservedRunningTime="2026-10-31 04:14:04.906386676 +0000 UTC m=+592.620721293" Oct 31 04:14:04.906684 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:04.906664 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-44f032ad-868c96f9c9-fvrrj" podStartSLOduration=1.906659823 podStartE2EDuration="1.906659823s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:14:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:14:04.90574953 +0000 UTC m=+592.620084167" watchObservedRunningTime="2026-10-31 04:14:04.906659823 +0000 UTC m=+592.620994439" Oct 31 04:14:04.997920 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:14:04.997887 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79fd2619_b05f_4da8_862d_a0e3e224bea3.slice/crio-dd2f9817345a85c87fbe50a0c8985df12c51c7da48a13107a1dd9f1fd4c27d69 WatchSource:0}: Error finding container dd2f9817345a85c87fbe50a0c8985df12c51c7da48a13107a1dd9f1fd4c27d69: Status 404 returned error can't find the container with id dd2f9817345a85c87fbe50a0c8985df12c51c7da48a13107a1dd9f1fd4c27d69 Oct 31 04:14:04.998099 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:04.998080 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4805aacc-77dfbbd755-hd54k"] Oct 31 04:14:05.812788 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:05.812753 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-4805aacc-77dfbbd755-hd54k" event={"ID":"79fd2619-b05f-4da8-862d-a0e3e224bea3","Type":"ContainerStarted","Data":"4b50dd51ef27bc5278a060c961a185a93ebbf3f5745bda546804443cf786076a"} Oct 31 04:14:05.813231 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:05.812812 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-4805aacc-77dfbbd755-hd54k" event={"ID":"79fd2619-b05f-4da8-862d-a0e3e224bea3","Type":"ContainerStarted","Data":"dd2f9817345a85c87fbe50a0c8985df12c51c7da48a13107a1dd9f1fd4c27d69"} Oct 31 04:14:05.813231 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:05.813011 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-4805aacc-77dfbbd755-hd54k" Oct 31 04:14:05.860221 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:05.859554 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-4805aacc-77dfbbd755-hd54k" podStartSLOduration=1.8595341520000002 podStartE2EDuration="1.859534152s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:14:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:14:05.85683767 +0000 UTC m=+593.571172287" watchObservedRunningTime="2026-10-31 04:14:05.859534152 +0000 UTC m=+593.573868770" Oct 31 04:14:09.574422 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:09.574384 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-44f032ad-666df5c575-kxg5z"] Oct 31 04:14:09.578652 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:09.578626 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-44f032ad-666df5c575-kxg5z" Oct 31 04:14:09.597250 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:09.597215 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8e1248b-7093-4b13-997d-4c3005c02110-maas-api-tls\") pod \"maas-api-e2e-ait-44f032ad-666df5c575-kxg5z\" (UID: \"f8e1248b-7093-4b13-997d-4c3005c02110\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-44f032ad-666df5c575-kxg5z" Oct 31 04:14:09.597471 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:09.597297 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8e1248b-7093-4b13-997d-4c3005c02110-openshift-service-ca\") pod \"maas-api-e2e-ait-44f032ad-666df5c575-kxg5z\" (UID: \"f8e1248b-7093-4b13-997d-4c3005c02110\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-44f032ad-666df5c575-kxg5z" Oct 31 04:14:09.597471 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:09.597362 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkknc\" (UniqueName: \"kubernetes.io/projected/f8e1248b-7093-4b13-997d-4c3005c02110-kube-api-access-zkknc\") pod \"maas-api-e2e-ait-44f032ad-666df5c575-kxg5z\" (UID: \"f8e1248b-7093-4b13-997d-4c3005c02110\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-44f032ad-666df5c575-kxg5z" Oct 31 04:14:09.597471 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:09.597412 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8e1248b-7093-4b13-997d-4c3005c02110-metrics-certs\") pod \"maas-api-e2e-ait-44f032ad-666df5c575-kxg5z\" (UID: \"f8e1248b-7093-4b13-997d-4c3005c02110\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-44f032ad-666df5c575-kxg5z" Oct 31 04:14:09.597675 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:09.597657 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-44f032ad-serving-cert\"" Oct 31 04:14:09.610729 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:09.610694 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-44f032ad-666df5c575-kxg5z"] Oct 31 04:14:09.698429 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:09.698394 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8e1248b-7093-4b13-997d-4c3005c02110-metrics-certs\") pod \"maas-api-e2e-ait-44f032ad-666df5c575-kxg5z\" (UID: \"f8e1248b-7093-4b13-997d-4c3005c02110\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-44f032ad-666df5c575-kxg5z" Oct 31 04:14:09.698636 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:09.698473 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8e1248b-7093-4b13-997d-4c3005c02110-maas-api-tls\") pod \"maas-api-e2e-ait-44f032ad-666df5c575-kxg5z\" (UID: \"f8e1248b-7093-4b13-997d-4c3005c02110\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-44f032ad-666df5c575-kxg5z" Oct 31 04:14:09.698636 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:09.698533 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8e1248b-7093-4b13-997d-4c3005c02110-openshift-service-ca\") pod \"maas-api-e2e-ait-44f032ad-666df5c575-kxg5z\" (UID: \"f8e1248b-7093-4b13-997d-4c3005c02110\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-44f032ad-666df5c575-kxg5z" Oct 31 04:14:09.698636 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:09.698576 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkknc\" (UniqueName: \"kubernetes.io/projected/f8e1248b-7093-4b13-997d-4c3005c02110-kube-api-access-zkknc\") pod \"maas-api-e2e-ait-44f032ad-666df5c575-kxg5z\" (UID: \"f8e1248b-7093-4b13-997d-4c3005c02110\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-44f032ad-666df5c575-kxg5z" Oct 31 04:14:09.698816 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:14:09.698634 4376 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-44f032ad-serving-cert: secret "maas-api-e2e-ait-44f032ad-serving-cert" not found Oct 31 04:14:09.698816 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:14:09.698709 4376 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8e1248b-7093-4b13-997d-4c3005c02110-maas-api-tls podName:f8e1248b-7093-4b13-997d-4c3005c02110 nodeName:}" failed. No retries permitted until 2026-10-31 04:14:10.198687659 +0000 UTC m=+597.913022274 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f8e1248b-7093-4b13-997d-4c3005c02110-maas-api-tls") pod "maas-api-e2e-ait-44f032ad-666df5c575-kxg5z" (UID: "f8e1248b-7093-4b13-997d-4c3005c02110") : secret "maas-api-e2e-ait-44f032ad-serving-cert" not found Oct 31 04:14:09.699288 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:09.699267 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8e1248b-7093-4b13-997d-4c3005c02110-openshift-service-ca\") pod \"maas-api-e2e-ait-44f032ad-666df5c575-kxg5z\" (UID: \"f8e1248b-7093-4b13-997d-4c3005c02110\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-44f032ad-666df5c575-kxg5z" Oct 31 04:14:09.701228 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:09.701204 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8e1248b-7093-4b13-997d-4c3005c02110-metrics-certs\") pod \"maas-api-e2e-ait-44f032ad-666df5c575-kxg5z\" (UID: \"f8e1248b-7093-4b13-997d-4c3005c02110\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-44f032ad-666df5c575-kxg5z" Oct 31 04:14:09.725129 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:09.725097 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkknc\" (UniqueName: \"kubernetes.io/projected/f8e1248b-7093-4b13-997d-4c3005c02110-kube-api-access-zkknc\") pod \"maas-api-e2e-ait-44f032ad-666df5c575-kxg5z\" (UID: \"f8e1248b-7093-4b13-997d-4c3005c02110\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-44f032ad-666df5c575-kxg5z" Oct 31 04:14:10.204864 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:10.204825 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8e1248b-7093-4b13-997d-4c3005c02110-maas-api-tls\") pod \"maas-api-e2e-ait-44f032ad-666df5c575-kxg5z\" (UID: \"f8e1248b-7093-4b13-997d-4c3005c02110\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-44f032ad-666df5c575-kxg5z" Oct 31 04:14:10.207259 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:10.207230 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8e1248b-7093-4b13-997d-4c3005c02110-maas-api-tls\") pod \"maas-api-e2e-ait-44f032ad-666df5c575-kxg5z\" (UID: \"f8e1248b-7093-4b13-997d-4c3005c02110\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-44f032ad-666df5c575-kxg5z" Oct 31 04:14:10.492617 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:10.492529 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-44f032ad-666df5c575-kxg5z" Oct 31 04:14:10.824625 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:10.824003 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-44f032ad-666df5c575-kxg5z"] Oct 31 04:14:10.852252 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:10.852212 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-44f032ad-666df5c575-kxg5z" event={"ID":"f8e1248b-7093-4b13-997d-4c3005c02110","Type":"ContainerStarted","Data":"57540015739a89767677e907991d4a3c8a14177432a0bb5bfb86037ad6641ce7"} Oct 31 04:14:11.008945 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:14:11.008908 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 04:14:11.769626 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:11.769595 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-16cda44a-59f4f898bc-lhxq5" Oct 31 04:14:11.859914 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:11.859879 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-44f032ad-666df5c575-kxg5z" event={"ID":"f8e1248b-7093-4b13-997d-4c3005c02110","Type":"ContainerStarted","Data":"f5f07db5caf75271c375e5a5bdc31a8d79f5368cb9937d70ea5f0dbb73f7a95a"} Oct 31 04:14:11.861266 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:11.859997 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-44f032ad-666df5c575-kxg5z" Oct 31 04:14:11.898342 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:11.898246 4376 patch_prober.go:29] interesting pod/e2e-ait-16cda44a-openshift-default-778c87d957-665xc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": read tcp 10.133.0.2:59464->10.133.0.37:15021: read: connection reset by peer" start-of-body= Oct 31 04:14:11.898481 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:11.898315 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-16cda44a-openshift-default-778c87d957-665xc" podUID="297621de-e8a2-4854-b28a-d3f21dac11b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": read tcp 10.133.0.2:59464->10.133.0.37:15021: read: connection reset by peer" Oct 31 04:14:11.907307 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:11.907250 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-44f032ad-666df5c575-kxg5z" podStartSLOduration=2.610654866 podStartE2EDuration="2.907235124s" totalImagesPullingTime="296.580258ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:14:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:14:11.905519395 +0000 UTC m=+599.619854045" watchObservedRunningTime="2026-10-31 04:14:11.907235124 +0000 UTC m=+599.621569740" Oct 31 04:14:12.773079 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:12.773046 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-16cda44a-589975f687-78vsg" Oct 31 04:14:12.773274 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:12.773231 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-a0d1a61d-58fd568b6c-kch6z" Oct 31 04:14:12.867010 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:12.866977 4376 generic.go:356] "Generic (PLEG): container finished" podID="297621de-e8a2-4854-b28a-d3f21dac11b2" containerID="5e9e7eaaf07d50910d76080d0d0d6e2d64f1429637ab887542ddf4215059be4f" exitCode=0 Oct 31 04:14:12.867526 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:12.867044 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-16cda44a-openshift-default-778c87d957-665xc" event={"ID":"297621de-e8a2-4854-b28a-d3f21dac11b2","Type":"ContainerDied","Data":"5e9e7eaaf07d50910d76080d0d0d6e2d64f1429637ab887542ddf4215059be4f"} Oct 31 04:14:12.867709 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:12.867691 4376 scope.go:122] "RemoveContainer" containerID="5e9e7eaaf07d50910d76080d0d0d6e2d64f1429637ab887542ddf4215059be4f" Oct 31 04:14:12.869024 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:12.868989 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:14:12.869141 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:12.869064 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:14:12.869141 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:12.869099 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:14:13.185684 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:13.185652 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 04:14:13.874506 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:13.874471 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-16cda44a-openshift-default-778c87d957-665xc" event={"ID":"297621de-e8a2-4854-b28a-d3f21dac11b2","Type":"ContainerStarted","Data":"d0d0a7222098fc9246bf2c005ed204566d0ab5d489cb3d61f8f4bfd8c1461f87"} Oct 31 04:14:14.089124 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:14.089087 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-16cda44a-openshift-default-778c87d957-665xc" Oct 31 04:14:14.089303 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:14.089282 4376 patch_prober.go:29] interesting pod/e2e-ait-16cda44a-openshift-default-778c87d957-665xc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Oct 31 04:14:14.089407 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:14.089356 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-16cda44a-openshift-default-778c87d957-665xc" podUID="297621de-e8a2-4854-b28a-d3f21dac11b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Oct 31 04:14:15.089958 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:15.089925 4376 patch_prober.go:29] interesting pod/e2e-ait-16cda44a-openshift-default-778c87d957-665xc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Oct 31 04:14:15.091067 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:15.091031 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-16cda44a-openshift-default-778c87d957-665xc" podUID="297621de-e8a2-4854-b28a-d3f21dac11b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Oct 31 04:14:15.582266 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:15.582215 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-f8a8088d-7d6557f575-7vg47"] Oct 31 04:14:15.594179 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:15.593517 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f8a8088d-7d6557f575-7vg47" Oct 31 04:14:15.600284 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:15.599388 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-f8a8088d-7d6557f575-7vg47"] Oct 31 04:14:15.610940 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:15.609611 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-f8a8088d-serving-cert\"" Oct 31 04:14:15.696169 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:15.696138 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/19b4fe89-322d-460f-8ccd-d5b950cbda65-metrics-certs\") pod \"maas-api-e2e-derive-f8a8088d-7d6557f575-7vg47\" (UID: \"19b4fe89-322d-460f-8ccd-d5b950cbda65\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f8a8088d-7d6557f575-7vg47" Oct 31 04:14:15.696789 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:15.696767 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/19b4fe89-322d-460f-8ccd-d5b950cbda65-maas-api-tls\") pod \"maas-api-e2e-derive-f8a8088d-7d6557f575-7vg47\" (UID: \"19b4fe89-322d-460f-8ccd-d5b950cbda65\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f8a8088d-7d6557f575-7vg47" Oct 31 04:14:15.696922 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:15.696909 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwxm7\" (UniqueName: \"kubernetes.io/projected/19b4fe89-322d-460f-8ccd-d5b950cbda65-kube-api-access-bwxm7\") pod \"maas-api-e2e-derive-f8a8088d-7d6557f575-7vg47\" (UID: \"19b4fe89-322d-460f-8ccd-d5b950cbda65\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f8a8088d-7d6557f575-7vg47" Oct 31 04:14:15.697004 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:15.696995 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/19b4fe89-322d-460f-8ccd-d5b950cbda65-openshift-service-ca\") pod \"maas-api-e2e-derive-f8a8088d-7d6557f575-7vg47\" (UID: \"19b4fe89-322d-460f-8ccd-d5b950cbda65\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f8a8088d-7d6557f575-7vg47" Oct 31 04:14:15.798780 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:15.798183 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/19b4fe89-322d-460f-8ccd-d5b950cbda65-maas-api-tls\") pod \"maas-api-e2e-derive-f8a8088d-7d6557f575-7vg47\" (UID: \"19b4fe89-322d-460f-8ccd-d5b950cbda65\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f8a8088d-7d6557f575-7vg47" Oct 31 04:14:15.798780 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:15.798250 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwxm7\" (UniqueName: \"kubernetes.io/projected/19b4fe89-322d-460f-8ccd-d5b950cbda65-kube-api-access-bwxm7\") pod \"maas-api-e2e-derive-f8a8088d-7d6557f575-7vg47\" (UID: \"19b4fe89-322d-460f-8ccd-d5b950cbda65\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f8a8088d-7d6557f575-7vg47" Oct 31 04:14:15.798780 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:15.798286 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/19b4fe89-322d-460f-8ccd-d5b950cbda65-openshift-service-ca\") pod \"maas-api-e2e-derive-f8a8088d-7d6557f575-7vg47\" (UID: \"19b4fe89-322d-460f-8ccd-d5b950cbda65\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f8a8088d-7d6557f575-7vg47" Oct 31 04:14:15.798780 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:15.798368 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/19b4fe89-322d-460f-8ccd-d5b950cbda65-metrics-certs\") pod \"maas-api-e2e-derive-f8a8088d-7d6557f575-7vg47\" (UID: \"19b4fe89-322d-460f-8ccd-d5b950cbda65\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f8a8088d-7d6557f575-7vg47" Oct 31 04:14:15.804112 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:15.801256 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/19b4fe89-322d-460f-8ccd-d5b950cbda65-openshift-service-ca\") pod \"maas-api-e2e-derive-f8a8088d-7d6557f575-7vg47\" (UID: \"19b4fe89-322d-460f-8ccd-d5b950cbda65\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f8a8088d-7d6557f575-7vg47" Oct 31 04:14:15.804112 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:15.803274 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/19b4fe89-322d-460f-8ccd-d5b950cbda65-metrics-certs\") pod \"maas-api-e2e-derive-f8a8088d-7d6557f575-7vg47\" (UID: \"19b4fe89-322d-460f-8ccd-d5b950cbda65\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f8a8088d-7d6557f575-7vg47" Oct 31 04:14:15.805343 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:15.805276 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/19b4fe89-322d-460f-8ccd-d5b950cbda65-maas-api-tls\") pod \"maas-api-e2e-derive-f8a8088d-7d6557f575-7vg47\" (UID: \"19b4fe89-322d-460f-8ccd-d5b950cbda65\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f8a8088d-7d6557f575-7vg47" Oct 31 04:14:15.817307 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:15.817277 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-44f032ad-7b89c4449f-gjfv8" Oct 31 04:14:15.823448 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:15.823419 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwxm7\" (UniqueName: \"kubernetes.io/projected/19b4fe89-322d-460f-8ccd-d5b950cbda65-kube-api-access-bwxm7\") pod \"maas-api-e2e-derive-f8a8088d-7d6557f575-7vg47\" (UID: \"19b4fe89-322d-460f-8ccd-d5b950cbda65\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f8a8088d-7d6557f575-7vg47" Oct 31 04:14:15.911027 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:15.910999 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f8a8088d-7d6557f575-7vg47" Oct 31 04:14:16.094287 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:16.093760 4376 patch_prober.go:29] interesting pod/e2e-ait-16cda44a-openshift-default-778c87d957-665xc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Oct 31 04:14:16.094287 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:16.093826 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-16cda44a-openshift-default-778c87d957-665xc" podUID="297621de-e8a2-4854-b28a-d3f21dac11b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Oct 31 04:14:16.290370 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:16.290282 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-f8a8088d-7d6557f575-7vg47"] Oct 31 04:14:16.823348 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:16.823312 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-44f032ad-868c96f9c9-fvrrj" Oct 31 04:14:16.823535 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:16.823495 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-4805aacc-77dfbbd755-hd54k" Oct 31 04:14:16.909446 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:16.909404 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f8a8088d-7d6557f575-7vg47" event={"ID":"19b4fe89-322d-460f-8ccd-d5b950cbda65","Type":"ContainerStarted","Data":"c9c74826ced90edf3433ecc3762b3c961070917aabd532268b189e51fecb9abb"} Oct 31 04:14:16.910460 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:16.910419 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f8a8088d-7d6557f575-7vg47" event={"ID":"19b4fe89-322d-460f-8ccd-d5b950cbda65","Type":"ContainerStarted","Data":"5ee7759cb33ec04e740854b468314a62d448fd316520e50fa645e6a4a88796a4"} Oct 31 04:14:16.910613 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:16.910472 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f8a8088d-7d6557f575-7vg47" Oct 31 04:14:17.012512 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:17.012463 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f8a8088d-7d6557f575-7vg47" podStartSLOduration=1.689009906 podStartE2EDuration="2.012449807s" totalImagesPullingTime="323.439901ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:14:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:14:17.011862583 +0000 UTC m=+604.726197217" watchObservedRunningTime="2026-10-31 04:14:17.012449807 +0000 UTC m=+604.726784424" Oct 31 04:14:17.094563 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:17.093664 4376 patch_prober.go:29] interesting pod/e2e-ait-16cda44a-openshift-default-778c87d957-665xc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Oct 31 04:14:17.094563 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:17.093715 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-16cda44a-openshift-default-778c87d957-665xc" podUID="297621de-e8a2-4854-b28a-d3f21dac11b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Oct 31 04:14:17.892355 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:17.891507 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-44f032ad-666df5c575-kxg5z" Oct 31 04:14:18.089789 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:18.089748 4376 patch_prober.go:29] interesting pod/e2e-ait-16cda44a-openshift-default-778c87d957-665xc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Oct 31 04:14:18.090060 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:18.090032 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-16cda44a-openshift-default-778c87d957-665xc" podUID="297621de-e8a2-4854-b28a-d3f21dac11b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Oct 31 04:14:19.170009 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:19.169981 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-16cda44a-openshift-default-778c87d957-665xc" Oct 31 04:14:19.171230 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:19.171196 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-16cda44a-openshift-default-778c87d957-665xc" Oct 31 04:14:19.172906 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:19.172878 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-16cda44a-openshift-default-778c87d957-665xc" Oct 31 04:14:20.282741 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:20.282704 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 04:14:22.920315 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:22.920281 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f8a8088d-7d6557f575-7vg47" Oct 31 04:14:23.945467 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:23.945436 4376 generic.go:356] "Generic (PLEG): container finished" podID="3f870408-c032-4347-af32-6a0a573aa5fe" containerID="8e1655be4ab5787d2a2b1f7d8e19850d2afb27ac91d330e34c268a7cdb30b3f1" exitCode=0 Oct 31 04:14:23.945944 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:23.945507 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4805aacc-openshift-default-78cc674747-nljxj" event={"ID":"3f870408-c032-4347-af32-6a0a573aa5fe","Type":"ContainerDied","Data":"8e1655be4ab5787d2a2b1f7d8e19850d2afb27ac91d330e34c268a7cdb30b3f1"} Oct 31 04:14:23.945944 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:23.945929 4376 scope.go:122] "RemoveContainer" containerID="8e1655be4ab5787d2a2b1f7d8e19850d2afb27ac91d330e34c268a7cdb30b3f1" Oct 31 04:14:23.947244 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:23.947219 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:14:23.947306 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:23.947286 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:14:23.947367 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:23.947317 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:14:23.991190 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:23.991162 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-16cda44a-589975f687-78vsg"] Oct 31 04:14:24.041633 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:24.041321 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-16cda44a-59f4f898bc-lhxq5"] Oct 31 04:14:24.967943 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:24.967152 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4805aacc-openshift-default-78cc674747-nljxj" event={"ID":"3f870408-c032-4347-af32-6a0a573aa5fe","Type":"ContainerStarted","Data":"84dffa99310042b0addbd2d2d69f194d7fcb0644bc66c3f892e3ff1485c82763"} Oct 31 04:14:25.076429 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:25.076394 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-4805aacc-openshift-default-78cc674747-nljxj" Oct 31 04:14:25.077623 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:25.077596 4376 patch_prober.go:29] interesting pod/e2e-ait-4805aacc-openshift-default-78cc674747-nljxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Oct 31 04:14:25.077773 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:25.077640 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-4805aacc-openshift-default-78cc674747-nljxj" podUID="3f870408-c032-4347-af32-6a0a573aa5fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Oct 31 04:14:25.109291 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:25.109262 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-5wwbn"] Oct 31 04:14:25.112730 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:25.112708 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-5wwbn" Oct 31 04:14:25.136155 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:25.136135 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-44f032ad-openshift-default-dockercfg-qx5tv\"" Oct 31 04:14:25.149905 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:25.149875 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-5wwbn"] Oct 31 04:14:25.229428 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:25.229352 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36a9f9d8-95e7-4a96-8cb0-0e02e47a4d9a-istio-data\") pod \"e2e-ait-44f032ad-openshift-default-5f845cc957-5wwbn\" (UID: \"36a9f9d8-95e7-4a96-8cb0-0e02e47a4d9a\") " pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-5wwbn" Oct 31 04:14:25.229428 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:25.229396 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36a9f9d8-95e7-4a96-8cb0-0e02e47a4d9a-istiod-ca-cert\") pod \"e2e-ait-44f032ad-openshift-default-5f845cc957-5wwbn\" (UID: \"36a9f9d8-95e7-4a96-8cb0-0e02e47a4d9a\") " pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-5wwbn" Oct 31 04:14:25.229614 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:25.229481 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36a9f9d8-95e7-4a96-8cb0-0e02e47a4d9a-istio-token\") pod \"e2e-ait-44f032ad-openshift-default-5f845cc957-5wwbn\" (UID: \"36a9f9d8-95e7-4a96-8cb0-0e02e47a4d9a\") " pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-5wwbn" Oct 31 04:14:25.229614 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:25.229540 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36a9f9d8-95e7-4a96-8cb0-0e02e47a4d9a-istio-envoy\") pod \"e2e-ait-44f032ad-openshift-default-5f845cc957-5wwbn\" (UID: \"36a9f9d8-95e7-4a96-8cb0-0e02e47a4d9a\") " pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-5wwbn" Oct 31 04:14:25.229614 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:25.229578 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36a9f9d8-95e7-4a96-8cb0-0e02e47a4d9a-workload-socket\") pod \"e2e-ait-44f032ad-openshift-default-5f845cc957-5wwbn\" (UID: \"36a9f9d8-95e7-4a96-8cb0-0e02e47a4d9a\") " pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-5wwbn" Oct 31 04:14:25.229765 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:25.229744 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vrm2\" (UniqueName: \"kubernetes.io/projected/36a9f9d8-95e7-4a96-8cb0-0e02e47a4d9a-kube-api-access-7vrm2\") pod \"e2e-ait-44f032ad-openshift-default-5f845cc957-5wwbn\" (UID: \"36a9f9d8-95e7-4a96-8cb0-0e02e47a4d9a\") " pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-5wwbn" Oct 31 04:14:25.229822 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:25.229792 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36a9f9d8-95e7-4a96-8cb0-0e02e47a4d9a-credential-socket\") pod \"e2e-ait-44f032ad-openshift-default-5f845cc957-5wwbn\" (UID: \"36a9f9d8-95e7-4a96-8cb0-0e02e47a4d9a\") " pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-5wwbn" Oct 31 04:14:25.229884 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:25.229849 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36a9f9d8-95e7-4a96-8cb0-0e02e47a4d9a-istio-podinfo\") pod \"e2e-ait-44f032ad-openshift-default-5f845cc957-5wwbn\" (UID: \"36a9f9d8-95e7-4a96-8cb0-0e02e47a4d9a\") " pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-5wwbn" Oct 31 04:14:25.229940 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:25.229927 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36a9f9d8-95e7-4a96-8cb0-0e02e47a4d9a-workload-certs\") pod \"e2e-ait-44f032ad-openshift-default-5f845cc957-5wwbn\" (UID: \"36a9f9d8-95e7-4a96-8cb0-0e02e47a4d9a\") " pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-5wwbn" Oct 31 04:14:25.330696 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:25.330663 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36a9f9d8-95e7-4a96-8cb0-0e02e47a4d9a-istio-envoy\") pod \"e2e-ait-44f032ad-openshift-default-5f845cc957-5wwbn\" (UID: \"36a9f9d8-95e7-4a96-8cb0-0e02e47a4d9a\") " pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-5wwbn" Oct 31 04:14:25.330696 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:25.330702 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36a9f9d8-95e7-4a96-8cb0-0e02e47a4d9a-workload-socket\") pod \"e2e-ait-44f032ad-openshift-default-5f845cc957-5wwbn\" (UID: \"36a9f9d8-95e7-4a96-8cb0-0e02e47a4d9a\") " pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-5wwbn" Oct 31 04:14:25.330936 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:25.330745 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vrm2\" (UniqueName: \"kubernetes.io/projected/36a9f9d8-95e7-4a96-8cb0-0e02e47a4d9a-kube-api-access-7vrm2\") pod \"e2e-ait-44f032ad-openshift-default-5f845cc957-5wwbn\" (UID: \"36a9f9d8-95e7-4a96-8cb0-0e02e47a4d9a\") " pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-5wwbn" Oct 31 04:14:25.330936 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:25.330764 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36a9f9d8-95e7-4a96-8cb0-0e02e47a4d9a-credential-socket\") pod \"e2e-ait-44f032ad-openshift-default-5f845cc957-5wwbn\" (UID: \"36a9f9d8-95e7-4a96-8cb0-0e02e47a4d9a\") " pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-5wwbn" Oct 31 04:14:25.330936 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:25.330779 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36a9f9d8-95e7-4a96-8cb0-0e02e47a4d9a-istio-podinfo\") pod \"e2e-ait-44f032ad-openshift-default-5f845cc957-5wwbn\" (UID: \"36a9f9d8-95e7-4a96-8cb0-0e02e47a4d9a\") " pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-5wwbn" Oct 31 04:14:25.330936 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:25.330802 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36a9f9d8-95e7-4a96-8cb0-0e02e47a4d9a-workload-certs\") pod \"e2e-ait-44f032ad-openshift-default-5f845cc957-5wwbn\" (UID: \"36a9f9d8-95e7-4a96-8cb0-0e02e47a4d9a\") " pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-5wwbn" Oct 31 04:14:25.331144 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:25.330947 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36a9f9d8-95e7-4a96-8cb0-0e02e47a4d9a-istio-data\") pod \"e2e-ait-44f032ad-openshift-default-5f845cc957-5wwbn\" (UID: \"36a9f9d8-95e7-4a96-8cb0-0e02e47a4d9a\") " pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-5wwbn" Oct 31 04:14:25.331144 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:25.331016 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36a9f9d8-95e7-4a96-8cb0-0e02e47a4d9a-istiod-ca-cert\") pod \"e2e-ait-44f032ad-openshift-default-5f845cc957-5wwbn\" (UID: \"36a9f9d8-95e7-4a96-8cb0-0e02e47a4d9a\") " pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-5wwbn" Oct 31 04:14:25.331144 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:25.331089 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36a9f9d8-95e7-4a96-8cb0-0e02e47a4d9a-istio-token\") pod \"e2e-ait-44f032ad-openshift-default-5f845cc957-5wwbn\" (UID: \"36a9f9d8-95e7-4a96-8cb0-0e02e47a4d9a\") " pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-5wwbn" Oct 31 04:14:25.331390 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:25.331258 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36a9f9d8-95e7-4a96-8cb0-0e02e47a4d9a-workload-socket\") pod \"e2e-ait-44f032ad-openshift-default-5f845cc957-5wwbn\" (UID: \"36a9f9d8-95e7-4a96-8cb0-0e02e47a4d9a\") " pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-5wwbn" Oct 31 04:14:25.331390 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:25.331295 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36a9f9d8-95e7-4a96-8cb0-0e02e47a4d9a-credential-socket\") pod \"e2e-ait-44f032ad-openshift-default-5f845cc957-5wwbn\" (UID: \"36a9f9d8-95e7-4a96-8cb0-0e02e47a4d9a\") " pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-5wwbn" Oct 31 04:14:25.331390 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:25.331314 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36a9f9d8-95e7-4a96-8cb0-0e02e47a4d9a-workload-certs\") pod \"e2e-ait-44f032ad-openshift-default-5f845cc957-5wwbn\" (UID: \"36a9f9d8-95e7-4a96-8cb0-0e02e47a4d9a\") " pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-5wwbn" Oct 31 04:14:25.331557 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:25.331401 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36a9f9d8-95e7-4a96-8cb0-0e02e47a4d9a-istio-data\") pod \"e2e-ait-44f032ad-openshift-default-5f845cc957-5wwbn\" (UID: \"36a9f9d8-95e7-4a96-8cb0-0e02e47a4d9a\") " pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-5wwbn" Oct 31 04:14:25.331881 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:25.331857 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36a9f9d8-95e7-4a96-8cb0-0e02e47a4d9a-istiod-ca-cert\") pod \"e2e-ait-44f032ad-openshift-default-5f845cc957-5wwbn\" (UID: \"36a9f9d8-95e7-4a96-8cb0-0e02e47a4d9a\") " pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-5wwbn" Oct 31 04:14:25.332959 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:25.332937 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36a9f9d8-95e7-4a96-8cb0-0e02e47a4d9a-istio-envoy\") pod \"e2e-ait-44f032ad-openshift-default-5f845cc957-5wwbn\" (UID: \"36a9f9d8-95e7-4a96-8cb0-0e02e47a4d9a\") " pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-5wwbn" Oct 31 04:14:25.333455 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:25.333433 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36a9f9d8-95e7-4a96-8cb0-0e02e47a4d9a-istio-podinfo\") pod \"e2e-ait-44f032ad-openshift-default-5f845cc957-5wwbn\" (UID: \"36a9f9d8-95e7-4a96-8cb0-0e02e47a4d9a\") " pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-5wwbn" Oct 31 04:14:25.367923 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:25.367891 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36a9f9d8-95e7-4a96-8cb0-0e02e47a4d9a-istio-token\") pod \"e2e-ait-44f032ad-openshift-default-5f845cc957-5wwbn\" (UID: \"36a9f9d8-95e7-4a96-8cb0-0e02e47a4d9a\") " pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-5wwbn" Oct 31 04:14:25.370028 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:25.369997 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vrm2\" (UniqueName: \"kubernetes.io/projected/36a9f9d8-95e7-4a96-8cb0-0e02e47a4d9a-kube-api-access-7vrm2\") pod \"e2e-ait-44f032ad-openshift-default-5f845cc957-5wwbn\" (UID: \"36a9f9d8-95e7-4a96-8cb0-0e02e47a4d9a\") " pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-5wwbn" Oct 31 04:14:25.426584 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:25.426553 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-5wwbn" Oct 31 04:14:25.766755 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:25.766035 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-5wwbn"] Oct 31 04:14:25.768717 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:14:25.768686 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36a9f9d8_95e7_4a96_8cb0_0e02e47a4d9a.slice/crio-fd8d7330b5a2b0fd55e557f77dd949137108b6e973dcec1644ff24dbf2ee2e95 WatchSource:0}: Error finding container fd8d7330b5a2b0fd55e557f77dd949137108b6e973dcec1644ff24dbf2ee2e95: Status 404 returned error can't find the container with id fd8d7330b5a2b0fd55e557f77dd949137108b6e973dcec1644ff24dbf2ee2e95 Oct 31 04:14:25.771032 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:25.770997 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:14:25.771114 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:25.771067 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:14:25.771114 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:25.771094 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:14:25.974108 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:25.974055 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-5wwbn" event={"ID":"36a9f9d8-95e7-4a96-8cb0-0e02e47a4d9a","Type":"ContainerStarted","Data":"a4b23dd48e851953e2475c4025c6e7873043ab45b87e6fe414c122b8b2d5f18b"} Oct 31 04:14:25.974108 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:25.974112 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-5wwbn" event={"ID":"36a9f9d8-95e7-4a96-8cb0-0e02e47a4d9a","Type":"ContainerStarted","Data":"fd8d7330b5a2b0fd55e557f77dd949137108b6e973dcec1644ff24dbf2ee2e95"} Oct 31 04:14:26.026407 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:26.026242 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-5wwbn" podStartSLOduration=1.026226357 podStartE2EDuration="1.026226357s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:14:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:14:26.025116923 +0000 UTC m=+613.739451556" watchObservedRunningTime="2026-10-31 04:14:26.026226357 +0000 UTC m=+613.740561031" Oct 31 04:14:26.077067 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:26.077037 4376 patch_prober.go:29] interesting pod/e2e-ait-4805aacc-openshift-default-78cc674747-nljxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Oct 31 04:14:26.077299 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:26.077275 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-4805aacc-openshift-default-78cc674747-nljxj" podUID="3f870408-c032-4347-af32-6a0a573aa5fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Oct 31 04:14:26.426876 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:26.426841 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-5wwbn" Oct 31 04:14:26.428317 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:26.428284 4376 patch_prober.go:29] interesting pod/e2e-ait-44f032ad-openshift-default-5f845cc957-5wwbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Oct 31 04:14:26.428462 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:26.428367 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-5wwbn" podUID="36a9f9d8-95e7-4a96-8cb0-0e02e47a4d9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Oct 31 04:14:27.077359 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:27.076722 4376 patch_prober.go:29] interesting pod/e2e-ait-4805aacc-openshift-default-78cc674747-nljxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Oct 31 04:14:27.077359 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:27.076790 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-4805aacc-openshift-default-78cc674747-nljxj" podUID="3f870408-c032-4347-af32-6a0a573aa5fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Oct 31 04:14:27.427417 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:27.427366 4376 patch_prober.go:29] interesting pod/e2e-ait-44f032ad-openshift-default-5f845cc957-5wwbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Oct 31 04:14:27.427630 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:27.427425 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-5wwbn" podUID="36a9f9d8-95e7-4a96-8cb0-0e02e47a4d9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Oct 31 04:14:28.078353 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:28.078065 4376 patch_prober.go:29] interesting pod/e2e-ait-4805aacc-openshift-default-78cc674747-nljxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Oct 31 04:14:28.078353 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:28.078161 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-4805aacc-openshift-default-78cc674747-nljxj" podUID="3f870408-c032-4347-af32-6a0a573aa5fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Oct 31 04:14:28.427978 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:28.427936 4376 patch_prober.go:29] interesting pod/e2e-ait-44f032ad-openshift-default-5f845cc957-5wwbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Oct 31 04:14:28.428152 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:28.427997 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-5wwbn" podUID="36a9f9d8-95e7-4a96-8cb0-0e02e47a4d9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Oct 31 04:14:28.749544 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:28.749441 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-4805aacc-openshift-default-78cc674747-nljxj" Oct 31 04:14:28.999950 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:28.995612 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-16cda44a-589975f687-78vsg" podUID="ab382e64-a6ee-4a64-8e7d-b99274571ab6" containerName="payload-processing" containerID="cri-o://c33f21f7efb4c1753bf498896d541350214367b38100640551aeb12e24fd5036" gracePeriod=25 Oct 31 04:14:29.046136 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:29.045409 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-16cda44a-59f4f898bc-lhxq5" podUID="dd0487ed-0a23-45ad-828d-114d26ec873e" containerName="payload-pre-processing" containerID="cri-o://0a9f2038e707a9da5d67e2458c35ab7d279e87cf86f3cdcba96ab2c40fe29357" gracePeriod=25 Oct 31 04:14:29.079224 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:29.078744 4376 patch_prober.go:29] interesting pod/e2e-ait-4805aacc-openshift-default-78cc674747-nljxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Oct 31 04:14:29.079224 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:29.078810 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-4805aacc-openshift-default-78cc674747-nljxj" podUID="3f870408-c032-4347-af32-6a0a573aa5fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Oct 31 04:14:29.423317 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:29.423290 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-16cda44a-589975f687-78vsg" Oct 31 04:14:29.427770 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:29.427676 4376 patch_prober.go:29] interesting pod/e2e-ait-44f032ad-openshift-default-5f845cc957-5wwbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Oct 31 04:14:29.427770 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:29.427723 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-5wwbn" podUID="36a9f9d8-95e7-4a96-8cb0-0e02e47a4d9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Oct 31 04:14:29.521954 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:29.521918 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-16cda44a-59f4f898bc-lhxq5" Oct 31 04:14:29.594096 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:29.594045 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-a0d1a61d-58fd568b6c-kch6z"] Oct 31 04:14:29.618944 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:29.618904 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab382e64-a6ee-4a64-8e7d-b99274571ab6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab382e64-a6ee-4a64-8e7d-b99274571ab6-plugins-config-volume\") pod \"ab382e64-a6ee-4a64-8e7d-b99274571ab6\" (UID: \"ab382e64-a6ee-4a64-8e7d-b99274571ab6\") " Oct 31 04:14:29.619137 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:29.619023 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab382e64-a6ee-4a64-8e7d-b99274571ab6-kube-api-access-w4vsc\" (UniqueName: \"kubernetes.io/projected/ab382e64-a6ee-4a64-8e7d-b99274571ab6-kube-api-access-w4vsc\") pod \"ab382e64-a6ee-4a64-8e7d-b99274571ab6\" (UID: \"ab382e64-a6ee-4a64-8e7d-b99274571ab6\") " Oct 31 04:14:29.619932 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:29.619899 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab382e64-a6ee-4a64-8e7d-b99274571ab6-plugins-config-volume" pod "ab382e64-a6ee-4a64-8e7d-b99274571ab6" (UID: "ab382e64-a6ee-4a64-8e7d-b99274571ab6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:14:29.622133 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:29.622107 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab382e64-a6ee-4a64-8e7d-b99274571ab6-kube-api-access-w4vsc" pod "ab382e64-a6ee-4a64-8e7d-b99274571ab6" (UID: "ab382e64-a6ee-4a64-8e7d-b99274571ab6"). InnerVolumeSpecName "kube-api-access-w4vsc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:14:29.720134 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:29.720031 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd0487ed-0a23-45ad-828d-114d26ec873e-kube-api-access-phrw4\" (UniqueName: \"kubernetes.io/projected/dd0487ed-0a23-45ad-828d-114d26ec873e-kube-api-access-phrw4\") pod \"dd0487ed-0a23-45ad-828d-114d26ec873e\" (UID: \"dd0487ed-0a23-45ad-828d-114d26ec873e\") " Oct 31 04:14:29.720322 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:29.720159 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd0487ed-0a23-45ad-828d-114d26ec873e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd0487ed-0a23-45ad-828d-114d26ec873e-plugins-config-volume\") pod \"dd0487ed-0a23-45ad-828d-114d26ec873e\" (UID: \"dd0487ed-0a23-45ad-828d-114d26ec873e\") " Oct 31 04:14:29.720543 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:29.720514 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd0487ed-0a23-45ad-828d-114d26ec873e-plugins-config-volume" pod "dd0487ed-0a23-45ad-828d-114d26ec873e" (UID: "dd0487ed-0a23-45ad-828d-114d26ec873e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:14:29.720653 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:29.720541 4376 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab382e64-a6ee-4a64-8e7d-b99274571ab6-plugins-config-volume\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:14:29.720653 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:29.720565 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4vsc\" (UniqueName: \"kubernetes.io/projected/ab382e64-a6ee-4a64-8e7d-b99274571ab6-kube-api-access-w4vsc\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:14:29.722286 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:29.722265 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd0487ed-0a23-45ad-828d-114d26ec873e-kube-api-access-phrw4" pod "dd0487ed-0a23-45ad-828d-114d26ec873e" (UID: "dd0487ed-0a23-45ad-828d-114d26ec873e"). InnerVolumeSpecName "kube-api-access-phrw4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:14:29.821367 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:29.821310 4376 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd0487ed-0a23-45ad-828d-114d26ec873e-plugins-config-volume\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:14:29.821367 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:29.821353 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phrw4\" (UniqueName: \"kubernetes.io/projected/dd0487ed-0a23-45ad-828d-114d26ec873e-kube-api-access-phrw4\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:14:30.016447 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:30.016321 4376 generic.go:356] "Generic (PLEG): container finished" podID="ab382e64-a6ee-4a64-8e7d-b99274571ab6" containerID="c33f21f7efb4c1753bf498896d541350214367b38100640551aeb12e24fd5036" exitCode=0 Oct 31 04:14:30.016447 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:30.016380 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-16cda44a-589975f687-78vsg" Oct 31 04:14:30.016447 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:30.016407 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-16cda44a-589975f687-78vsg" event={"ID":"ab382e64-a6ee-4a64-8e7d-b99274571ab6","Type":"ContainerDied","Data":"c33f21f7efb4c1753bf498896d541350214367b38100640551aeb12e24fd5036"} Oct 31 04:14:30.016727 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:30.016470 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-16cda44a-589975f687-78vsg" event={"ID":"ab382e64-a6ee-4a64-8e7d-b99274571ab6","Type":"ContainerDied","Data":"77bb5f8e11a4eeae60888d32bcf66987f4c53dba90232293bc682702ec725260"} Oct 31 04:14:30.016727 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:30.016497 4376 scope.go:122] "RemoveContainer" containerID="c33f21f7efb4c1753bf498896d541350214367b38100640551aeb12e24fd5036" Oct 31 04:14:30.018121 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:30.018096 4376 generic.go:356] "Generic (PLEG): container finished" podID="dd0487ed-0a23-45ad-828d-114d26ec873e" containerID="0a9f2038e707a9da5d67e2458c35ab7d279e87cf86f3cdcba96ab2c40fe29357" exitCode=0 Oct 31 04:14:30.018700 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:30.018161 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-16cda44a-59f4f898bc-lhxq5" Oct 31 04:14:30.018777 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:30.018189 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-16cda44a-59f4f898bc-lhxq5" event={"ID":"dd0487ed-0a23-45ad-828d-114d26ec873e","Type":"ContainerDied","Data":"0a9f2038e707a9da5d67e2458c35ab7d279e87cf86f3cdcba96ab2c40fe29357"} Oct 31 04:14:30.018864 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:30.018841 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-16cda44a-59f4f898bc-lhxq5" event={"ID":"dd0487ed-0a23-45ad-828d-114d26ec873e","Type":"ContainerDied","Data":"2bc6b1fe7da563bf1dccd103335f709666fe7f29f5db99622edfc17d0549fb10"} Oct 31 04:14:30.027006 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:30.026980 4376 scope.go:122] "RemoveContainer" containerID="c33f21f7efb4c1753bf498896d541350214367b38100640551aeb12e24fd5036" Oct 31 04:14:30.027461 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:14:30.027435 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c33f21f7efb4c1753bf498896d541350214367b38100640551aeb12e24fd5036\": container with ID starting with c33f21f7efb4c1753bf498896d541350214367b38100640551aeb12e24fd5036 not found: ID does not exist" containerID="c33f21f7efb4c1753bf498896d541350214367b38100640551aeb12e24fd5036" Oct 31 04:14:30.027560 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:30.027471 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c33f21f7efb4c1753bf498896d541350214367b38100640551aeb12e24fd5036"} err="failed to get container status \"c33f21f7efb4c1753bf498896d541350214367b38100640551aeb12e24fd5036\": rpc error: code = NotFound desc = could not find container \"c33f21f7efb4c1753bf498896d541350214367b38100640551aeb12e24fd5036\": container with ID starting with c33f21f7efb4c1753bf498896d541350214367b38100640551aeb12e24fd5036 not found: ID does not exist" Oct 31 04:14:30.027560 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:30.027495 4376 scope.go:122] "RemoveContainer" containerID="0a9f2038e707a9da5d67e2458c35ab7d279e87cf86f3cdcba96ab2c40fe29357" Oct 31 04:14:30.036136 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:30.036115 4376 scope.go:122] "RemoveContainer" containerID="0a9f2038e707a9da5d67e2458c35ab7d279e87cf86f3cdcba96ab2c40fe29357" Oct 31 04:14:30.036581 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:14:30.036477 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a9f2038e707a9da5d67e2458c35ab7d279e87cf86f3cdcba96ab2c40fe29357\": container with ID starting with 0a9f2038e707a9da5d67e2458c35ab7d279e87cf86f3cdcba96ab2c40fe29357 not found: ID does not exist" containerID="0a9f2038e707a9da5d67e2458c35ab7d279e87cf86f3cdcba96ab2c40fe29357" Oct 31 04:14:30.036694 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:30.036588 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a9f2038e707a9da5d67e2458c35ab7d279e87cf86f3cdcba96ab2c40fe29357"} err="failed to get container status \"0a9f2038e707a9da5d67e2458c35ab7d279e87cf86f3cdcba96ab2c40fe29357\": rpc error: code = NotFound desc = could not find container \"0a9f2038e707a9da5d67e2458c35ab7d279e87cf86f3cdcba96ab2c40fe29357\": container with ID starting with 0a9f2038e707a9da5d67e2458c35ab7d279e87cf86f3cdcba96ab2c40fe29357 not found: ID does not exist" Oct 31 04:14:30.076752 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:30.076716 4376 patch_prober.go:29] interesting pod/e2e-ait-4805aacc-openshift-default-78cc674747-nljxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Oct 31 04:14:30.076929 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:30.076781 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-4805aacc-openshift-default-78cc674747-nljxj" podUID="3f870408-c032-4347-af32-6a0a573aa5fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Oct 31 04:14:30.078824 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:30.078791 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-16cda44a-59f4f898bc-lhxq5"] Oct 31 04:14:30.084422 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:30.084397 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-16cda44a-59f4f898bc-lhxq5"] Oct 31 04:14:30.124862 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:30.124795 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-16cda44a-589975f687-78vsg"] Oct 31 04:14:30.127485 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:30.127443 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-16cda44a-589975f687-78vsg"] Oct 31 04:14:30.426928 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:30.426897 4376 patch_prober.go:29] interesting pod/e2e-ait-44f032ad-openshift-default-5f845cc957-5wwbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Oct 31 04:14:30.427121 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:30.426945 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-5wwbn" podUID="36a9f9d8-95e7-4a96-8cb0-0e02e47a4d9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Oct 31 04:14:30.466111 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:30.466077 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab382e64-a6ee-4a64-8e7d-b99274571ab6" path="/var/lib/kubelet/pods/ab382e64-a6ee-4a64-8e7d-b99274571ab6/volumes" Oct 31 04:14:30.466493 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:30.466479 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd0487ed-0a23-45ad-828d-114d26ec873e" path="/var/lib/kubelet/pods/dd0487ed-0a23-45ad-828d-114d26ec873e/volumes" Oct 31 04:14:31.081377 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:31.081315 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-4805aacc-openshift-default-78cc674747-nljxj" Oct 31 04:14:31.082256 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:31.082237 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-4805aacc-openshift-default-78cc674747-nljxj" Oct 31 04:14:31.427503 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:31.427464 4376 patch_prober.go:29] interesting pod/e2e-ait-44f032ad-openshift-default-5f845cc957-5wwbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Oct 31 04:14:31.427905 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:31.427522 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-5wwbn" podUID="36a9f9d8-95e7-4a96-8cb0-0e02e47a4d9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Oct 31 04:14:32.431152 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:32.431117 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-5wwbn" Oct 31 04:14:32.431608 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:32.431438 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-5wwbn" Oct 31 04:14:32.432160 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:32.432143 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-5wwbn" Oct 31 04:14:33.820741 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:33.820698 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-16cda44a-openshift-default-778c87d957-665xc"] Oct 31 04:14:33.821249 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:33.820918 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-16cda44a-openshift-default-778c87d957-665xc" podUID="297621de-e8a2-4854-b28a-d3f21dac11b2" containerName="istio-proxy" containerID="cri-o://d0d0a7222098fc9246bf2c005ed204566d0ab5d489cb3d61f8f4bfd8c1461f87" gracePeriod=30 Oct 31 04:14:34.174137 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:34.174102 4376 patch_prober.go:29] interesting pod/e2e-ait-16cda44a-openshift-default-778c87d957-665xc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 04:14:34.174287 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:34.174165 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-16cda44a-openshift-default-778c87d957-665xc" podUID="297621de-e8a2-4854-b28a-d3f21dac11b2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 04:14:34.595240 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:34.595153 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-a0d1a61d-58fd568b6c-kch6z" podUID="850a4333-5224-4732-967d-0b19bbca006f" containerName="payload-pre-processing" containerID="cri-o://2f645c60a21d92fa291d3e499e3f4dfdcad047b04869db9aaafe92545dc89152" gracePeriod=25 Oct 31 04:14:34.857299 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:34.857227 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-a0d1a61d-58fd568b6c-kch6z" Oct 31 04:14:34.922023 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:34.921997 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/850a4333-5224-4732-967d-0b19bbca006f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/850a4333-5224-4732-967d-0b19bbca006f-plugins-config-volume\") pod \"850a4333-5224-4732-967d-0b19bbca006f\" (UID: \"850a4333-5224-4732-967d-0b19bbca006f\") " Oct 31 04:14:34.922185 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:34.922145 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/850a4333-5224-4732-967d-0b19bbca006f-kube-api-access-plz24\" (UniqueName: \"kubernetes.io/projected/850a4333-5224-4732-967d-0b19bbca006f-kube-api-access-plz24\") pod \"850a4333-5224-4732-967d-0b19bbca006f\" (UID: \"850a4333-5224-4732-967d-0b19bbca006f\") " Oct 31 04:14:34.922390 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:34.922369 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/850a4333-5224-4732-967d-0b19bbca006f-plugins-config-volume" pod "850a4333-5224-4732-967d-0b19bbca006f" (UID: "850a4333-5224-4732-967d-0b19bbca006f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:14:34.924580 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:34.924548 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/850a4333-5224-4732-967d-0b19bbca006f-kube-api-access-plz24" pod "850a4333-5224-4732-967d-0b19bbca006f" (UID: "850a4333-5224-4732-967d-0b19bbca006f"). InnerVolumeSpecName "kube-api-access-plz24". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:14:35.023787 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:35.023754 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-plz24\" (UniqueName: \"kubernetes.io/projected/850a4333-5224-4732-967d-0b19bbca006f-kube-api-access-plz24\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:14:35.023787 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:35.023783 4376 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/850a4333-5224-4732-967d-0b19bbca006f-plugins-config-volume\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:14:35.055510 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:35.055472 4376 generic.go:356] "Generic (PLEG): container finished" podID="850a4333-5224-4732-967d-0b19bbca006f" containerID="2f645c60a21d92fa291d3e499e3f4dfdcad047b04869db9aaafe92545dc89152" exitCode=0 Oct 31 04:14:35.055668 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:35.055522 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-a0d1a61d-58fd568b6c-kch6z" Oct 31 04:14:35.055668 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:35.055572 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-a0d1a61d-58fd568b6c-kch6z" event={"ID":"850a4333-5224-4732-967d-0b19bbca006f","Type":"ContainerDied","Data":"2f645c60a21d92fa291d3e499e3f4dfdcad047b04869db9aaafe92545dc89152"} Oct 31 04:14:35.055668 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:35.055635 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-a0d1a61d-58fd568b6c-kch6z" event={"ID":"850a4333-5224-4732-967d-0b19bbca006f","Type":"ContainerDied","Data":"9850bdcd3eec7602a2db2349f7a2d23e2461af9287f4914aea927c7ae3a6fbc2"} Oct 31 04:14:35.055668 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:35.055664 4376 scope.go:122] "RemoveContainer" containerID="2f645c60a21d92fa291d3e499e3f4dfdcad047b04869db9aaafe92545dc89152" Oct 31 04:14:35.065687 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:35.065658 4376 scope.go:122] "RemoveContainer" containerID="2f645c60a21d92fa291d3e499e3f4dfdcad047b04869db9aaafe92545dc89152" Oct 31 04:14:35.072266 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:14:35.072235 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f645c60a21d92fa291d3e499e3f4dfdcad047b04869db9aaafe92545dc89152\": container with ID starting with 2f645c60a21d92fa291d3e499e3f4dfdcad047b04869db9aaafe92545dc89152 not found: ID does not exist" containerID="2f645c60a21d92fa291d3e499e3f4dfdcad047b04869db9aaafe92545dc89152" Oct 31 04:14:35.072411 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:35.072278 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f645c60a21d92fa291d3e499e3f4dfdcad047b04869db9aaafe92545dc89152"} err="failed to get container status \"2f645c60a21d92fa291d3e499e3f4dfdcad047b04869db9aaafe92545dc89152\": rpc error: code = NotFound desc = could not find container \"2f645c60a21d92fa291d3e499e3f4dfdcad047b04869db9aaafe92545dc89152\": container with ID starting with 2f645c60a21d92fa291d3e499e3f4dfdcad047b04869db9aaafe92545dc89152 not found: ID does not exist" Oct 31 04:14:35.114472 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:35.114406 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-a0d1a61d-58fd568b6c-kch6z"] Oct 31 04:14:35.120288 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:35.120264 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-a0d1a61d-58fd568b6c-kch6z"] Oct 31 04:14:36.443591 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:36.443555 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="850a4333-5224-4732-967d-0b19bbca006f" path="/var/lib/kubelet/pods/850a4333-5224-4732-967d-0b19bbca006f/volumes" Oct 31 04:14:36.704898 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:36.704804 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4805aacc-77dfbbd755-hd54k"] Oct 31 04:14:37.489766 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:37.489727 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 04:14:39.069346 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:39.069308 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-16cda44a-openshift-default-778c87d957-665xc" Oct 31 04:14:39.079041 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:39.079012 4376 generic.go:356] "Generic (PLEG): container finished" podID="297621de-e8a2-4854-b28a-d3f21dac11b2" containerID="d0d0a7222098fc9246bf2c005ed204566d0ab5d489cb3d61f8f4bfd8c1461f87" exitCode=0 Oct 31 04:14:39.079168 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:39.079050 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-16cda44a-openshift-default-778c87d957-665xc" event={"ID":"297621de-e8a2-4854-b28a-d3f21dac11b2","Type":"ContainerDied","Data":"d0d0a7222098fc9246bf2c005ed204566d0ab5d489cb3d61f8f4bfd8c1461f87"} Oct 31 04:14:39.079168 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:39.079068 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-16cda44a-openshift-default-778c87d957-665xc" Oct 31 04:14:39.079168 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:39.079116 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-16cda44a-openshift-default-778c87d957-665xc" event={"ID":"297621de-e8a2-4854-b28a-d3f21dac11b2","Type":"ContainerDied","Data":"da38413380a46a9b38c23f38d53ffc78a397415ce6f2ba007a0e7ce82d812735"} Oct 31 04:14:39.079168 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:39.079148 4376 scope.go:122] "RemoveContainer" containerID="d0d0a7222098fc9246bf2c005ed204566d0ab5d489cb3d61f8f4bfd8c1461f87" Oct 31 04:14:39.090051 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:39.090031 4376 scope.go:122] "RemoveContainer" containerID="5e9e7eaaf07d50910d76080d0d0d6e2d64f1429637ab887542ddf4215059be4f" Oct 31 04:14:39.098142 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:39.098120 4376 scope.go:122] "RemoveContainer" containerID="d0d0a7222098fc9246bf2c005ed204566d0ab5d489cb3d61f8f4bfd8c1461f87" Oct 31 04:14:39.098468 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:14:39.098447 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0d0a7222098fc9246bf2c005ed204566d0ab5d489cb3d61f8f4bfd8c1461f87\": container with ID starting with d0d0a7222098fc9246bf2c005ed204566d0ab5d489cb3d61f8f4bfd8c1461f87 not found: ID does not exist" containerID="d0d0a7222098fc9246bf2c005ed204566d0ab5d489cb3d61f8f4bfd8c1461f87" Oct 31 04:14:39.098522 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:39.098477 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0d0a7222098fc9246bf2c005ed204566d0ab5d489cb3d61f8f4bfd8c1461f87"} err="failed to get container status \"d0d0a7222098fc9246bf2c005ed204566d0ab5d489cb3d61f8f4bfd8c1461f87\": rpc error: code = NotFound desc = could not find container \"d0d0a7222098fc9246bf2c005ed204566d0ab5d489cb3d61f8f4bfd8c1461f87\": container with ID starting with d0d0a7222098fc9246bf2c005ed204566d0ab5d489cb3d61f8f4bfd8c1461f87 not found: ID does not exist" Oct 31 04:14:39.098522 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:39.098495 4376 scope.go:122] "RemoveContainer" containerID="5e9e7eaaf07d50910d76080d0d0d6e2d64f1429637ab887542ddf4215059be4f" Oct 31 04:14:39.098739 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:14:39.098719 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e9e7eaaf07d50910d76080d0d0d6e2d64f1429637ab887542ddf4215059be4f\": container with ID starting with 5e9e7eaaf07d50910d76080d0d0d6e2d64f1429637ab887542ddf4215059be4f not found: ID does not exist" containerID="5e9e7eaaf07d50910d76080d0d0d6e2d64f1429637ab887542ddf4215059be4f" Oct 31 04:14:39.098803 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:39.098753 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e9e7eaaf07d50910d76080d0d0d6e2d64f1429637ab887542ddf4215059be4f"} err="failed to get container status \"5e9e7eaaf07d50910d76080d0d0d6e2d64f1429637ab887542ddf4215059be4f\": rpc error: code = NotFound desc = could not find container \"5e9e7eaaf07d50910d76080d0d0d6e2d64f1429637ab887542ddf4215059be4f\": container with ID starting with 5e9e7eaaf07d50910d76080d0d0d6e2d64f1429637ab887542ddf4215059be4f not found: ID does not exist" Oct 31 04:14:39.174914 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:39.173953 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/297621de-e8a2-4854-b28a-d3f21dac11b2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/297621de-e8a2-4854-b28a-d3f21dac11b2-istiod-ca-cert\") pod \"297621de-e8a2-4854-b28a-d3f21dac11b2\" (UID: \"297621de-e8a2-4854-b28a-d3f21dac11b2\") " Oct 31 04:14:39.174914 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:39.174005 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/297621de-e8a2-4854-b28a-d3f21dac11b2-kube-api-access-skq7z\" (UniqueName: \"kubernetes.io/projected/297621de-e8a2-4854-b28a-d3f21dac11b2-kube-api-access-skq7z\") pod \"297621de-e8a2-4854-b28a-d3f21dac11b2\" (UID: \"297621de-e8a2-4854-b28a-d3f21dac11b2\") " Oct 31 04:14:39.174914 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:39.174091 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/297621de-e8a2-4854-b28a-d3f21dac11b2-istio-token\" (UniqueName: \"kubernetes.io/projected/297621de-e8a2-4854-b28a-d3f21dac11b2-istio-token\") pod \"297621de-e8a2-4854-b28a-d3f21dac11b2\" (UID: \"297621de-e8a2-4854-b28a-d3f21dac11b2\") " Oct 31 04:14:39.174914 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:39.174133 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/297621de-e8a2-4854-b28a-d3f21dac11b2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/297621de-e8a2-4854-b28a-d3f21dac11b2-istio-data\") pod \"297621de-e8a2-4854-b28a-d3f21dac11b2\" (UID: \"297621de-e8a2-4854-b28a-d3f21dac11b2\") " Oct 31 04:14:39.174914 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:39.174169 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/297621de-e8a2-4854-b28a-d3f21dac11b2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/297621de-e8a2-4854-b28a-d3f21dac11b2-workload-certs\") pod \"297621de-e8a2-4854-b28a-d3f21dac11b2\" (UID: \"297621de-e8a2-4854-b28a-d3f21dac11b2\") " Oct 31 04:14:39.174914 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:39.174227 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/297621de-e8a2-4854-b28a-d3f21dac11b2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/297621de-e8a2-4854-b28a-d3f21dac11b2-istio-envoy\") pod \"297621de-e8a2-4854-b28a-d3f21dac11b2\" (UID: \"297621de-e8a2-4854-b28a-d3f21dac11b2\") " Oct 31 04:14:39.174914 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:39.174294 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/297621de-e8a2-4854-b28a-d3f21dac11b2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/297621de-e8a2-4854-b28a-d3f21dac11b2-credential-socket\") pod \"297621de-e8a2-4854-b28a-d3f21dac11b2\" (UID: \"297621de-e8a2-4854-b28a-d3f21dac11b2\") " Oct 31 04:14:39.174914 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:39.174352 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/297621de-e8a2-4854-b28a-d3f21dac11b2-istiod-ca-cert" pod "297621de-e8a2-4854-b28a-d3f21dac11b2" (UID: "297621de-e8a2-4854-b28a-d3f21dac11b2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:14:39.174914 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:39.174367 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/297621de-e8a2-4854-b28a-d3f21dac11b2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/297621de-e8a2-4854-b28a-d3f21dac11b2-istio-podinfo\") pod \"297621de-e8a2-4854-b28a-d3f21dac11b2\" (UID: \"297621de-e8a2-4854-b28a-d3f21dac11b2\") " Oct 31 04:14:39.174914 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:39.174400 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/297621de-e8a2-4854-b28a-d3f21dac11b2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/297621de-e8a2-4854-b28a-d3f21dac11b2-workload-socket\") pod \"297621de-e8a2-4854-b28a-d3f21dac11b2\" (UID: \"297621de-e8a2-4854-b28a-d3f21dac11b2\") " Oct 31 04:14:39.174914 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:39.174665 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/297621de-e8a2-4854-b28a-d3f21dac11b2-credential-socket" pod "297621de-e8a2-4854-b28a-d3f21dac11b2" (UID: "297621de-e8a2-4854-b28a-d3f21dac11b2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:14:39.174914 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:39.174803 4376 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/297621de-e8a2-4854-b28a-d3f21dac11b2-credential-socket\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:14:39.174914 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:39.174846 4376 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/297621de-e8a2-4854-b28a-d3f21dac11b2-istiod-ca-cert\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:14:39.174914 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:39.174878 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/297621de-e8a2-4854-b28a-d3f21dac11b2-workload-socket" pod "297621de-e8a2-4854-b28a-d3f21dac11b2" (UID: "297621de-e8a2-4854-b28a-d3f21dac11b2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:14:39.175760 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:39.175125 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/297621de-e8a2-4854-b28a-d3f21dac11b2-workload-certs" pod "297621de-e8a2-4854-b28a-d3f21dac11b2" (UID: "297621de-e8a2-4854-b28a-d3f21dac11b2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:14:39.175760 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:39.175185 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/297621de-e8a2-4854-b28a-d3f21dac11b2-istio-data" pod "297621de-e8a2-4854-b28a-d3f21dac11b2" (UID: "297621de-e8a2-4854-b28a-d3f21dac11b2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:14:39.176696 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:39.176584 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/297621de-e8a2-4854-b28a-d3f21dac11b2-istio-token" pod "297621de-e8a2-4854-b28a-d3f21dac11b2" (UID: "297621de-e8a2-4854-b28a-d3f21dac11b2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:14:39.177065 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:39.177044 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/297621de-e8a2-4854-b28a-d3f21dac11b2-istio-podinfo" pod "297621de-e8a2-4854-b28a-d3f21dac11b2" (UID: "297621de-e8a2-4854-b28a-d3f21dac11b2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 04:14:39.178099 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:39.178076 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/297621de-e8a2-4854-b28a-d3f21dac11b2-istio-envoy" pod "297621de-e8a2-4854-b28a-d3f21dac11b2" (UID: "297621de-e8a2-4854-b28a-d3f21dac11b2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:14:39.178208 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:39.178076 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/297621de-e8a2-4854-b28a-d3f21dac11b2-kube-api-access-skq7z" pod "297621de-e8a2-4854-b28a-d3f21dac11b2" (UID: "297621de-e8a2-4854-b28a-d3f21dac11b2"). InnerVolumeSpecName "kube-api-access-skq7z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:14:39.276257 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:39.276219 4376 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/297621de-e8a2-4854-b28a-d3f21dac11b2-istio-envoy\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:14:39.276257 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:39.276252 4376 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/297621de-e8a2-4854-b28a-d3f21dac11b2-istio-podinfo\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:14:39.276257 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:39.276266 4376 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/297621de-e8a2-4854-b28a-d3f21dac11b2-workload-socket\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:14:39.276559 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:39.276285 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-skq7z\" (UniqueName: \"kubernetes.io/projected/297621de-e8a2-4854-b28a-d3f21dac11b2-kube-api-access-skq7z\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:14:39.276559 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:39.276296 4376 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/297621de-e8a2-4854-b28a-d3f21dac11b2-istio-token\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:14:39.276559 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:39.276305 4376 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/297621de-e8a2-4854-b28a-d3f21dac11b2-istio-data\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:14:39.276559 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:39.276312 4376 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/297621de-e8a2-4854-b28a-d3f21dac11b2-workload-certs\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:14:39.429681 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:39.429639 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-16cda44a-openshift-default-778c87d957-665xc"] Oct 31 04:14:39.433607 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:39.433574 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-16cda44a-openshift-default-778c87d957-665xc"] Oct 31 04:14:40.441745 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:40.441709 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="297621de-e8a2-4854-b28a-d3f21dac11b2" path="/var/lib/kubelet/pods/297621de-e8a2-4854-b28a-d3f21dac11b2/volumes" Oct 31 04:14:40.783663 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:40.783567 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-09a20a-b4f7d6d49-2lp58"] Oct 31 04:14:40.784066 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:40.784045 4376 cpu_manager.go:414] "containerMap: removing container" podUID="297621de-e8a2-4854-b28a-d3f21dac11b2" containerName="istio-proxy" Oct 31 04:14:40.784151 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:40.784068 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="297621de-e8a2-4854-b28a-d3f21dac11b2" containerName="istio-proxy" Oct 31 04:14:40.784151 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:40.784091 4376 cpu_manager.go:414] "containerMap: removing container" podUID="dd0487ed-0a23-45ad-828d-114d26ec873e" containerName="payload-pre-processing" Oct 31 04:14:40.784151 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:40.784099 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd0487ed-0a23-45ad-828d-114d26ec873e" containerName="payload-pre-processing" Oct 31 04:14:40.784297 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:40.784153 4376 cpu_manager.go:414] "containerMap: removing container" podUID="ab382e64-a6ee-4a64-8e7d-b99274571ab6" containerName="payload-processing" Oct 31 04:14:40.784297 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:40.784160 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab382e64-a6ee-4a64-8e7d-b99274571ab6" containerName="payload-processing" Oct 31 04:14:40.784297 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:40.784172 4376 cpu_manager.go:414] "containerMap: removing container" podUID="850a4333-5224-4732-967d-0b19bbca006f" containerName="payload-pre-processing" Oct 31 04:14:40.784297 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:40.784180 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="850a4333-5224-4732-967d-0b19bbca006f" containerName="payload-pre-processing" Oct 31 04:14:40.784297 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:40.784265 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="297621de-e8a2-4854-b28a-d3f21dac11b2" containerName="istio-proxy" Oct 31 04:14:40.784297 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:40.784277 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab382e64-a6ee-4a64-8e7d-b99274571ab6" containerName="payload-processing" Oct 31 04:14:40.784297 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:40.784288 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="850a4333-5224-4732-967d-0b19bbca006f" containerName="payload-pre-processing" Oct 31 04:14:40.784297 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:40.784297 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="297621de-e8a2-4854-b28a-d3f21dac11b2" containerName="istio-proxy" Oct 31 04:14:40.784678 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:40.784307 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd0487ed-0a23-45ad-828d-114d26ec873e" containerName="payload-pre-processing" Oct 31 04:14:40.789321 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:40.789297 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-09a20a-b4f7d6d49-2lp58" Oct 31 04:14:40.805863 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:40.805834 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-09a20a-b4f7d6d49-2lp58"] Oct 31 04:14:40.818035 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:40.817999 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-09a20a-dockercfg-j6txs\"" Oct 31 04:14:40.818035 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:40.818031 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-09a20a\"" Oct 31 04:14:40.844449 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:40.844417 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-09a20a-5bf985cbc9-scqzr"] Oct 31 04:14:40.845013 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:40.844992 4376 cpu_manager.go:414] "containerMap: removing container" podUID="297621de-e8a2-4854-b28a-d3f21dac11b2" containerName="istio-proxy" Oct 31 04:14:40.845013 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:40.845011 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="297621de-e8a2-4854-b28a-d3f21dac11b2" containerName="istio-proxy" Oct 31 04:14:40.848287 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:40.848266 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-09a20a-5bf985cbc9-scqzr" Oct 31 04:14:40.865983 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:40.865945 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-09a20a-5bf985cbc9-scqzr"] Oct 31 04:14:40.888631 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:40.888595 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb75359a-8880-406f-955c-5a3fe0fb796c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-09a20a-5bf985cbc9-scqzr\" (UID: \"cb75359a-8880-406f-955c-5a3fe0fb796c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-09a20a-5bf985cbc9-scqzr" Oct 31 04:14:40.888840 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:40.888657 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmjhn\" (UniqueName: \"kubernetes.io/projected/cb75359a-8880-406f-955c-5a3fe0fb796c-kube-api-access-zmjhn\") pod \"payload-pre-processing-e2e-worker-w2-09a20a-5bf985cbc9-scqzr\" (UID: \"cb75359a-8880-406f-955c-5a3fe0fb796c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-09a20a-5bf985cbc9-scqzr" Oct 31 04:14:40.888840 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:40.888782 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df1df0a3-2701-4f32-a6f7-3f7ce203cfee-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-09a20a-b4f7d6d49-2lp58\" (UID: \"df1df0a3-2701-4f32-a6f7-3f7ce203cfee\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-09a20a-b4f7d6d49-2lp58" Oct 31 04:14:40.888840 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:40.888805 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jddmw\" (UniqueName: \"kubernetes.io/projected/df1df0a3-2701-4f32-a6f7-3f7ce203cfee-kube-api-access-jddmw\") pod \"payload-processing-e2e-worker-w2-09a20a-b4f7d6d49-2lp58\" (UID: \"df1df0a3-2701-4f32-a6f7-3f7ce203cfee\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-09a20a-b4f7d6d49-2lp58" Oct 31 04:14:40.990007 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:40.989969 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmjhn\" (UniqueName: \"kubernetes.io/projected/cb75359a-8880-406f-955c-5a3fe0fb796c-kube-api-access-zmjhn\") pod \"payload-pre-processing-e2e-worker-w2-09a20a-5bf985cbc9-scqzr\" (UID: \"cb75359a-8880-406f-955c-5a3fe0fb796c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-09a20a-5bf985cbc9-scqzr" Oct 31 04:14:40.990007 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:40.990022 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df1df0a3-2701-4f32-a6f7-3f7ce203cfee-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-09a20a-b4f7d6d49-2lp58\" (UID: \"df1df0a3-2701-4f32-a6f7-3f7ce203cfee\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-09a20a-b4f7d6d49-2lp58" Oct 31 04:14:40.990277 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:40.990043 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jddmw\" (UniqueName: \"kubernetes.io/projected/df1df0a3-2701-4f32-a6f7-3f7ce203cfee-kube-api-access-jddmw\") pod \"payload-processing-e2e-worker-w2-09a20a-b4f7d6d49-2lp58\" (UID: \"df1df0a3-2701-4f32-a6f7-3f7ce203cfee\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-09a20a-b4f7d6d49-2lp58" Oct 31 04:14:40.990277 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:40.990134 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb75359a-8880-406f-955c-5a3fe0fb796c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-09a20a-5bf985cbc9-scqzr\" (UID: \"cb75359a-8880-406f-955c-5a3fe0fb796c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-09a20a-5bf985cbc9-scqzr" Oct 31 04:14:40.990872 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:40.990841 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df1df0a3-2701-4f32-a6f7-3f7ce203cfee-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-09a20a-b4f7d6d49-2lp58\" (UID: \"df1df0a3-2701-4f32-a6f7-3f7ce203cfee\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-09a20a-b4f7d6d49-2lp58" Oct 31 04:14:40.991235 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:40.991212 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb75359a-8880-406f-955c-5a3fe0fb796c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-09a20a-5bf985cbc9-scqzr\" (UID: \"cb75359a-8880-406f-955c-5a3fe0fb796c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-09a20a-5bf985cbc9-scqzr" Oct 31 04:14:41.001415 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:14:41.001380 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 04:14:41.020661 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:41.020629 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jddmw\" (UniqueName: \"kubernetes.io/projected/df1df0a3-2701-4f32-a6f7-3f7ce203cfee-kube-api-access-jddmw\") pod \"payload-processing-e2e-worker-w2-09a20a-b4f7d6d49-2lp58\" (UID: \"df1df0a3-2701-4f32-a6f7-3f7ce203cfee\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-09a20a-b4f7d6d49-2lp58" Oct 31 04:14:41.020820 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:41.020749 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmjhn\" (UniqueName: \"kubernetes.io/projected/cb75359a-8880-406f-955c-5a3fe0fb796c-kube-api-access-zmjhn\") pod \"payload-pre-processing-e2e-worker-w2-09a20a-5bf985cbc9-scqzr\" (UID: \"cb75359a-8880-406f-955c-5a3fe0fb796c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-09a20a-5bf985cbc9-scqzr" Oct 31 04:14:41.103143 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:41.103054 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-09a20a-b4f7d6d49-2lp58" Oct 31 04:14:41.164364 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:41.163052 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-09a20a-5bf985cbc9-scqzr" Oct 31 04:14:41.402949 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:14:41.402919 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf1df0a3_2701_4f32_a6f7_3f7ce203cfee.slice/crio-b5589f6f6b2d9a7d696d51b7895a2713e777eac1fd617ea3d87049f9a7f7ec87 WatchSource:0}: Error finding container b5589f6f6b2d9a7d696d51b7895a2713e777eac1fd617ea3d87049f9a7f7ec87: Status 404 returned error can't find the container with id b5589f6f6b2d9a7d696d51b7895a2713e777eac1fd617ea3d87049f9a7f7ec87 Oct 31 04:14:41.403526 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:41.403491 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-09a20a-b4f7d6d49-2lp58"] Oct 31 04:14:41.484028 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:41.483998 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-4805aacc-openshift-default-78cc674747-nljxj"] Oct 31 04:14:41.484544 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:41.484215 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-4805aacc-openshift-default-78cc674747-nljxj" podUID="3f870408-c032-4347-af32-6a0a573aa5fe" containerName="istio-proxy" containerID="cri-o://84dffa99310042b0addbd2d2d69f194d7fcb0644bc66c3f892e3ff1485c82763" gracePeriod=30 Oct 31 04:14:41.497828 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:41.497799 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-09a20a-5bf985cbc9-scqzr"] Oct 31 04:14:41.706021 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:41.705925 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-4805aacc-77dfbbd755-hd54k" podUID="79fd2619-b05f-4da8-862d-a0e3e224bea3" containerName="payload-pre-processing" containerID="cri-o://4b50dd51ef27bc5278a060c961a185a93ebbf3f5745bda546804443cf786076a" gracePeriod=25 Oct 31 04:14:41.965252 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:41.965174 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-4805aacc-77dfbbd755-hd54k" Oct 31 04:14:42.002878 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:42.002840 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79fd2619-b05f-4da8-862d-a0e3e224bea3-kube-api-access-gnjm9\" (UniqueName: \"kubernetes.io/projected/79fd2619-b05f-4da8-862d-a0e3e224bea3-kube-api-access-gnjm9\") pod \"79fd2619-b05f-4da8-862d-a0e3e224bea3\" (UID: \"79fd2619-b05f-4da8-862d-a0e3e224bea3\") " Oct 31 04:14:42.003067 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:42.003044 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79fd2619-b05f-4da8-862d-a0e3e224bea3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79fd2619-b05f-4da8-862d-a0e3e224bea3-plugins-config-volume\") pod \"79fd2619-b05f-4da8-862d-a0e3e224bea3\" (UID: \"79fd2619-b05f-4da8-862d-a0e3e224bea3\") " Oct 31 04:14:42.003459 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:42.003437 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79fd2619-b05f-4da8-862d-a0e3e224bea3-plugins-config-volume" pod "79fd2619-b05f-4da8-862d-a0e3e224bea3" (UID: "79fd2619-b05f-4da8-862d-a0e3e224bea3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:14:42.004967 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:42.004946 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79fd2619-b05f-4da8-862d-a0e3e224bea3-kube-api-access-gnjm9" pod "79fd2619-b05f-4da8-862d-a0e3e224bea3" (UID: "79fd2619-b05f-4da8-862d-a0e3e224bea3"). InnerVolumeSpecName "kube-api-access-gnjm9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:14:42.098145 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:42.098113 4376 generic.go:356] "Generic (PLEG): container finished" podID="79fd2619-b05f-4da8-862d-a0e3e224bea3" containerID="4b50dd51ef27bc5278a060c961a185a93ebbf3f5745bda546804443cf786076a" exitCode=0 Oct 31 04:14:42.098347 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:42.098156 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-4805aacc-77dfbbd755-hd54k" Oct 31 04:14:42.098347 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:42.098202 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-4805aacc-77dfbbd755-hd54k" event={"ID":"79fd2619-b05f-4da8-862d-a0e3e224bea3","Type":"ContainerDied","Data":"4b50dd51ef27bc5278a060c961a185a93ebbf3f5745bda546804443cf786076a"} Oct 31 04:14:42.098347 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:42.098259 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-4805aacc-77dfbbd755-hd54k" event={"ID":"79fd2619-b05f-4da8-862d-a0e3e224bea3","Type":"ContainerDied","Data":"dd2f9817345a85c87fbe50a0c8985df12c51c7da48a13107a1dd9f1fd4c27d69"} Oct 31 04:14:42.098347 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:42.098281 4376 scope.go:122] "RemoveContainer" containerID="4b50dd51ef27bc5278a060c961a185a93ebbf3f5745bda546804443cf786076a" Oct 31 04:14:42.099861 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:42.099832 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-09a20a-5bf985cbc9-scqzr" event={"ID":"cb75359a-8880-406f-955c-5a3fe0fb796c","Type":"ContainerStarted","Data":"ebedea044c8eaa87c68fd97ebb83131a5fe25f4f04c65881135cdf40f0894b32"} Oct 31 04:14:42.099982 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:42.099878 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-09a20a-5bf985cbc9-scqzr" event={"ID":"cb75359a-8880-406f-955c-5a3fe0fb796c","Type":"ContainerStarted","Data":"e7776281bc2c028430ed8cb58519eb60f2b838ac0fa9062671dac7eb53fc5b1f"} Oct 31 04:14:42.099982 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:42.099915 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-09a20a-5bf985cbc9-scqzr" Oct 31 04:14:42.102037 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:42.102006 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-09a20a-b4f7d6d49-2lp58" event={"ID":"df1df0a3-2701-4f32-a6f7-3f7ce203cfee","Type":"ContainerStarted","Data":"85e13b0c3148e086c5aa422ad780b541df3510fb08bfb98b9a3ba3db31fac305"} Oct 31 04:14:42.102167 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:42.102072 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-09a20a-b4f7d6d49-2lp58" event={"ID":"df1df0a3-2701-4f32-a6f7-3f7ce203cfee","Type":"ContainerStarted","Data":"b5589f6f6b2d9a7d696d51b7895a2713e777eac1fd617ea3d87049f9a7f7ec87"} Oct 31 04:14:42.102167 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:42.102100 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-09a20a-b4f7d6d49-2lp58" Oct 31 04:14:42.104124 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:42.104104 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gnjm9\" (UniqueName: \"kubernetes.io/projected/79fd2619-b05f-4da8-862d-a0e3e224bea3-kube-api-access-gnjm9\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:14:42.104124 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:42.104123 4376 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79fd2619-b05f-4da8-862d-a0e3e224bea3-plugins-config-volume\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:14:42.108344 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:42.108142 4376 scope.go:122] "RemoveContainer" containerID="4b50dd51ef27bc5278a060c961a185a93ebbf3f5745bda546804443cf786076a" Oct 31 04:14:42.108473 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:14:42.108455 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b50dd51ef27bc5278a060c961a185a93ebbf3f5745bda546804443cf786076a\": container with ID starting with 4b50dd51ef27bc5278a060c961a185a93ebbf3f5745bda546804443cf786076a not found: ID does not exist" containerID="4b50dd51ef27bc5278a060c961a185a93ebbf3f5745bda546804443cf786076a" Oct 31 04:14:42.108519 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:42.108481 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b50dd51ef27bc5278a060c961a185a93ebbf3f5745bda546804443cf786076a"} err="failed to get container status \"4b50dd51ef27bc5278a060c961a185a93ebbf3f5745bda546804443cf786076a\": rpc error: code = NotFound desc = could not find container \"4b50dd51ef27bc5278a060c961a185a93ebbf3f5745bda546804443cf786076a\": container with ID starting with 4b50dd51ef27bc5278a060c961a185a93ebbf3f5745bda546804443cf786076a not found: ID does not exist" Oct 31 04:14:42.160464 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:42.160423 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-09a20a-b4f7d6d49-2lp58" podStartSLOduration=2.160410897 podStartE2EDuration="2.160410897s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:14:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:14:42.160057594 +0000 UTC m=+629.874392210" watchObservedRunningTime="2026-10-31 04:14:42.160410897 +0000 UTC m=+629.874745513" Oct 31 04:14:42.204067 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:42.204034 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4805aacc-77dfbbd755-hd54k"] Oct 31 04:14:42.210517 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:42.210482 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4805aacc-77dfbbd755-hd54k"] Oct 31 04:14:42.253212 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:42.253117 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-09a20a-5bf985cbc9-scqzr" podStartSLOduration=2.253104486 podStartE2EDuration="2.253104486s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:14:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:14:42.25227047 +0000 UTC m=+629.966605087" watchObservedRunningTime="2026-10-31 04:14:42.253104486 +0000 UTC m=+629.967439123" Oct 31 04:14:42.443670 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:42.443638 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79fd2619-b05f-4da8-862d-a0e3e224bea3" path="/var/lib/kubelet/pods/79fd2619-b05f-4da8-862d-a0e3e224bea3/volumes" Oct 31 04:14:42.540406 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:42.539715 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-44f032ad-868c96f9c9-fvrrj"] Oct 31 04:14:42.576651 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:42.576609 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-44f032ad-7b89c4449f-gjfv8"] Oct 31 04:14:42.753449 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:42.753421 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-zmth9"] Oct 31 04:14:42.753851 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:42.753838 4376 cpu_manager.go:414] "containerMap: removing container" podUID="79fd2619-b05f-4da8-862d-a0e3e224bea3" containerName="payload-pre-processing" Oct 31 04:14:42.753907 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:42.753853 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79fd2619-b05f-4da8-862d-a0e3e224bea3" containerName="payload-pre-processing" Oct 31 04:14:42.753947 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:42.753934 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="79fd2619-b05f-4da8-862d-a0e3e224bea3" containerName="payload-pre-processing" Oct 31 04:14:42.758560 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:42.758536 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-zmth9" Oct 31 04:14:42.778809 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:42.778778 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-965f61-openshift-default-dockercfg-ggnnv\"" Oct 31 04:14:42.782015 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:42.781986 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-zmth9"] Oct 31 04:14:42.810513 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:42.810418 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d825a50-865d-46c6-b88d-73f7170963c5-istio-data\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-zmth9\" (UID: \"5d825a50-865d-46c6-b88d-73f7170963c5\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-zmth9" Oct 31 04:14:42.810513 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:42.810475 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d825a50-865d-46c6-b88d-73f7170963c5-credential-socket\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-zmth9\" (UID: \"5d825a50-865d-46c6-b88d-73f7170963c5\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-zmth9" Oct 31 04:14:42.810750 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:42.810536 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d825a50-865d-46c6-b88d-73f7170963c5-istiod-ca-cert\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-zmth9\" (UID: \"5d825a50-865d-46c6-b88d-73f7170963c5\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-zmth9" Oct 31 04:14:42.810750 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:42.810635 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d825a50-865d-46c6-b88d-73f7170963c5-istio-podinfo\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-zmth9\" (UID: \"5d825a50-865d-46c6-b88d-73f7170963c5\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-zmth9" Oct 31 04:14:42.810750 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:42.810690 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d825a50-865d-46c6-b88d-73f7170963c5-workload-socket\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-zmth9\" (UID: \"5d825a50-865d-46c6-b88d-73f7170963c5\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-zmth9" Oct 31 04:14:42.810922 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:42.810759 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d825a50-865d-46c6-b88d-73f7170963c5-workload-certs\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-zmth9\" (UID: \"5d825a50-865d-46c6-b88d-73f7170963c5\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-zmth9" Oct 31 04:14:42.810922 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:42.810868 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d825a50-865d-46c6-b88d-73f7170963c5-istio-token\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-zmth9\" (UID: \"5d825a50-865d-46c6-b88d-73f7170963c5\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-zmth9" Oct 31 04:14:42.810922 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:42.810895 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d825a50-865d-46c6-b88d-73f7170963c5-istio-envoy\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-zmth9\" (UID: \"5d825a50-865d-46c6-b88d-73f7170963c5\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-zmth9" Oct 31 04:14:42.811078 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:42.810921 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84mr7\" (UniqueName: \"kubernetes.io/projected/5d825a50-865d-46c6-b88d-73f7170963c5-kube-api-access-84mr7\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-zmth9\" (UID: \"5d825a50-865d-46c6-b88d-73f7170963c5\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-zmth9" Oct 31 04:14:42.911747 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:42.911719 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d825a50-865d-46c6-b88d-73f7170963c5-istio-token\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-zmth9\" (UID: \"5d825a50-865d-46c6-b88d-73f7170963c5\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-zmth9" Oct 31 04:14:42.911747 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:42.911753 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d825a50-865d-46c6-b88d-73f7170963c5-istio-envoy\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-zmth9\" (UID: \"5d825a50-865d-46c6-b88d-73f7170963c5\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-zmth9" Oct 31 04:14:42.911951 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:42.911917 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84mr7\" (UniqueName: \"kubernetes.io/projected/5d825a50-865d-46c6-b88d-73f7170963c5-kube-api-access-84mr7\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-zmth9\" (UID: \"5d825a50-865d-46c6-b88d-73f7170963c5\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-zmth9" Oct 31 04:14:42.911993 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:42.911954 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d825a50-865d-46c6-b88d-73f7170963c5-istio-data\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-zmth9\" (UID: \"5d825a50-865d-46c6-b88d-73f7170963c5\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-zmth9" Oct 31 04:14:42.912046 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:42.912006 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d825a50-865d-46c6-b88d-73f7170963c5-credential-socket\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-zmth9\" (UID: \"5d825a50-865d-46c6-b88d-73f7170963c5\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-zmth9" Oct 31 04:14:42.912101 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:42.912062 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d825a50-865d-46c6-b88d-73f7170963c5-istiod-ca-cert\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-zmth9\" (UID: \"5d825a50-865d-46c6-b88d-73f7170963c5\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-zmth9" Oct 31 04:14:42.912159 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:42.912113 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d825a50-865d-46c6-b88d-73f7170963c5-istio-podinfo\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-zmth9\" (UID: \"5d825a50-865d-46c6-b88d-73f7170963c5\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-zmth9" Oct 31 04:14:42.912159 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:42.912154 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d825a50-865d-46c6-b88d-73f7170963c5-workload-socket\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-zmth9\" (UID: \"5d825a50-865d-46c6-b88d-73f7170963c5\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-zmth9" Oct 31 04:14:42.912268 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:42.912191 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d825a50-865d-46c6-b88d-73f7170963c5-workload-certs\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-zmth9\" (UID: \"5d825a50-865d-46c6-b88d-73f7170963c5\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-zmth9" Oct 31 04:14:42.912447 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:42.912416 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d825a50-865d-46c6-b88d-73f7170963c5-istio-data\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-zmth9\" (UID: \"5d825a50-865d-46c6-b88d-73f7170963c5\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-zmth9" Oct 31 04:14:42.912578 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:42.912431 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d825a50-865d-46c6-b88d-73f7170963c5-credential-socket\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-zmth9\" (UID: \"5d825a50-865d-46c6-b88d-73f7170963c5\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-zmth9" Oct 31 04:14:42.912578 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:42.912522 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d825a50-865d-46c6-b88d-73f7170963c5-workload-socket\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-zmth9\" (UID: \"5d825a50-865d-46c6-b88d-73f7170963c5\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-zmth9" Oct 31 04:14:42.912736 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:42.912588 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d825a50-865d-46c6-b88d-73f7170963c5-workload-certs\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-zmth9\" (UID: \"5d825a50-865d-46c6-b88d-73f7170963c5\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-zmth9" Oct 31 04:14:42.912865 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:42.912840 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d825a50-865d-46c6-b88d-73f7170963c5-istiod-ca-cert\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-zmth9\" (UID: \"5d825a50-865d-46c6-b88d-73f7170963c5\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-zmth9" Oct 31 04:14:42.914471 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:42.914449 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d825a50-865d-46c6-b88d-73f7170963c5-istio-envoy\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-zmth9\" (UID: \"5d825a50-865d-46c6-b88d-73f7170963c5\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-zmth9" Oct 31 04:14:42.914599 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:42.914581 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d825a50-865d-46c6-b88d-73f7170963c5-istio-podinfo\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-zmth9\" (UID: \"5d825a50-865d-46c6-b88d-73f7170963c5\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-zmth9" Oct 31 04:14:42.946118 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:42.946083 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84mr7\" (UniqueName: \"kubernetes.io/projected/5d825a50-865d-46c6-b88d-73f7170963c5-kube-api-access-84mr7\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-zmth9\" (UID: \"5d825a50-865d-46c6-b88d-73f7170963c5\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-zmth9" Oct 31 04:14:42.950711 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:42.950682 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d825a50-865d-46c6-b88d-73f7170963c5-istio-token\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-zmth9\" (UID: \"5d825a50-865d-46c6-b88d-73f7170963c5\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-zmth9" Oct 31 04:14:43.074356 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:43.074247 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-zmth9" Oct 31 04:14:43.399737 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:14:43.399704 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d825a50_865d_46c6_b88d_73f7170963c5.slice/crio-dcab1365bcd83b6138d68270ff4f2f214909b6070b689eb55c67d6c2400239b3 WatchSource:0}: Error finding container dcab1365bcd83b6138d68270ff4f2f214909b6070b689eb55c67d6c2400239b3: Status 404 returned error can't find the container with id dcab1365bcd83b6138d68270ff4f2f214909b6070b689eb55c67d6c2400239b3 Oct 31 04:14:43.400174 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:43.400148 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-zmth9"] Oct 31 04:14:43.402043 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:43.402006 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:14:43.402146 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:43.402071 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:14:43.402146 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:43.402119 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:14:44.118419 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:44.118385 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-zmth9" event={"ID":"5d825a50-865d-46c6-b88d-73f7170963c5","Type":"ContainerStarted","Data":"ba4b20e19ffef90a458a50db30fa6d872dfa311726d28f320c05c268046f4bac"} Oct 31 04:14:44.118938 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:44.118444 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-zmth9" event={"ID":"5d825a50-865d-46c6-b88d-73f7170963c5","Type":"ContainerStarted","Data":"dcab1365bcd83b6138d68270ff4f2f214909b6070b689eb55c67d6c2400239b3"} Oct 31 04:14:44.183979 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:44.183929 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-zmth9" podStartSLOduration=2.183915138 podStartE2EDuration="2.183915138s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:14:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:14:44.183738851 +0000 UTC m=+631.898073486" watchObservedRunningTime="2026-10-31 04:14:44.183915138 +0000 UTC m=+631.898249754" Oct 31 04:14:45.074534 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:45.074500 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-zmth9" Oct 31 04:14:45.079669 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:45.079640 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-zmth9" Oct 31 04:14:45.123650 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:45.123625 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-zmth9" Oct 31 04:14:45.124605 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:45.124585 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-zmth9" Oct 31 04:14:46.084628 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:46.084585 4376 patch_prober.go:29] interesting pod/e2e-ait-4805aacc-openshift-default-78cc674747-nljxj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 04:14:46.084802 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:46.084644 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-4805aacc-openshift-default-78cc674747-nljxj" podUID="3f870408-c032-4347-af32-6a0a573aa5fe" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 04:14:46.416108 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:46.416070 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-f8a8088d-7d6557f575-7vg47"] Oct 31 04:14:46.416602 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:46.416366 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f8a8088d-7d6557f575-7vg47" podUID="19b4fe89-322d-460f-8ccd-d5b950cbda65" containerName="maas-api" containerID="cri-o://c9c74826ced90edf3433ecc3762b3c961070917aabd532268b189e51fecb9abb" gracePeriod=30 Oct 31 04:14:46.770024 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:46.769997 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f8a8088d-7d6557f575-7vg47" Oct 31 04:14:46.804751 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:46.804717 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-4805aacc-openshift-default-78cc674747-nljxj" Oct 31 04:14:46.866951 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:46.866917 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/19b4fe89-322d-460f-8ccd-d5b950cbda65-metrics-certs\" (UniqueName: \"kubernetes.io/secret/19b4fe89-322d-460f-8ccd-d5b950cbda65-metrics-certs\") pod \"19b4fe89-322d-460f-8ccd-d5b950cbda65\" (UID: \"19b4fe89-322d-460f-8ccd-d5b950cbda65\") " Oct 31 04:14:46.867147 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:46.866964 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f870408-c032-4347-af32-6a0a573aa5fe-istio-token\" (UniqueName: \"kubernetes.io/projected/3f870408-c032-4347-af32-6a0a573aa5fe-istio-token\") pod \"3f870408-c032-4347-af32-6a0a573aa5fe\" (UID: \"3f870408-c032-4347-af32-6a0a573aa5fe\") " Oct 31 04:14:46.867147 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:46.867030 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f870408-c032-4347-af32-6a0a573aa5fe-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f870408-c032-4347-af32-6a0a573aa5fe-workload-socket\") pod \"3f870408-c032-4347-af32-6a0a573aa5fe\" (UID: \"3f870408-c032-4347-af32-6a0a573aa5fe\") " Oct 31 04:14:46.867147 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:46.867080 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f870408-c032-4347-af32-6a0a573aa5fe-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f870408-c032-4347-af32-6a0a573aa5fe-istio-data\") pod \"3f870408-c032-4347-af32-6a0a573aa5fe\" (UID: \"3f870408-c032-4347-af32-6a0a573aa5fe\") " Oct 31 04:14:46.867147 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:46.867129 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3f870408-c032-4347-af32-6a0a573aa5fe-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f870408-c032-4347-af32-6a0a573aa5fe-istio-podinfo\") pod \"3f870408-c032-4347-af32-6a0a573aa5fe\" (UID: \"3f870408-c032-4347-af32-6a0a573aa5fe\") " Oct 31 04:14:46.867530 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:46.867158 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f870408-c032-4347-af32-6a0a573aa5fe-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f870408-c032-4347-af32-6a0a573aa5fe-credential-socket\") pod \"3f870408-c032-4347-af32-6a0a573aa5fe\" (UID: \"3f870408-c032-4347-af32-6a0a573aa5fe\") " Oct 31 04:14:46.867530 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:46.867178 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19b4fe89-322d-460f-8ccd-d5b950cbda65-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/19b4fe89-322d-460f-8ccd-d5b950cbda65-openshift-service-ca\") pod \"19b4fe89-322d-460f-8ccd-d5b950cbda65\" (UID: \"19b4fe89-322d-460f-8ccd-d5b950cbda65\") " Oct 31 04:14:46.867530 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:46.867247 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/19b4fe89-322d-460f-8ccd-d5b950cbda65-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/19b4fe89-322d-460f-8ccd-d5b950cbda65-maas-api-tls\") pod \"19b4fe89-322d-460f-8ccd-d5b950cbda65\" (UID: \"19b4fe89-322d-460f-8ccd-d5b950cbda65\") " Oct 31 04:14:46.867530 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:46.867289 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f870408-c032-4347-af32-6a0a573aa5fe-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f870408-c032-4347-af32-6a0a573aa5fe-istiod-ca-cert\") pod \"3f870408-c032-4347-af32-6a0a573aa5fe\" (UID: \"3f870408-c032-4347-af32-6a0a573aa5fe\") " Oct 31 04:14:46.867530 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:46.867357 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f870408-c032-4347-af32-6a0a573aa5fe-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f870408-c032-4347-af32-6a0a573aa5fe-istio-envoy\") pod \"3f870408-c032-4347-af32-6a0a573aa5fe\" (UID: \"3f870408-c032-4347-af32-6a0a573aa5fe\") " Oct 31 04:14:46.867530 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:46.867388 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f870408-c032-4347-af32-6a0a573aa5fe-kube-api-access-9j466\" (UniqueName: \"kubernetes.io/projected/3f870408-c032-4347-af32-6a0a573aa5fe-kube-api-access-9j466\") pod \"3f870408-c032-4347-af32-6a0a573aa5fe\" (UID: \"3f870408-c032-4347-af32-6a0a573aa5fe\") " Oct 31 04:14:46.867530 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:46.867420 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f870408-c032-4347-af32-6a0a573aa5fe-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f870408-c032-4347-af32-6a0a573aa5fe-workload-certs\") pod \"3f870408-c032-4347-af32-6a0a573aa5fe\" (UID: \"3f870408-c032-4347-af32-6a0a573aa5fe\") " Oct 31 04:14:46.868735 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:46.867566 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19b4fe89-322d-460f-8ccd-d5b950cbda65-kube-api-access-bwxm7\" (UniqueName: \"kubernetes.io/projected/19b4fe89-322d-460f-8ccd-d5b950cbda65-kube-api-access-bwxm7\") pod \"19b4fe89-322d-460f-8ccd-d5b950cbda65\" (UID: \"19b4fe89-322d-460f-8ccd-d5b950cbda65\") " Oct 31 04:14:46.868735 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:46.868374 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19b4fe89-322d-460f-8ccd-d5b950cbda65-openshift-service-ca" pod "19b4fe89-322d-460f-8ccd-d5b950cbda65" (UID: "19b4fe89-322d-460f-8ccd-d5b950cbda65"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:14:46.868735 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:46.868608 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f870408-c032-4347-af32-6a0a573aa5fe-istio-data" pod "3f870408-c032-4347-af32-6a0a573aa5fe" (UID: "3f870408-c032-4347-af32-6a0a573aa5fe"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:14:46.868901 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:46.868809 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f870408-c032-4347-af32-6a0a573aa5fe-workload-socket" pod "3f870408-c032-4347-af32-6a0a573aa5fe" (UID: "3f870408-c032-4347-af32-6a0a573aa5fe"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:14:46.869445 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:46.869078 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f870408-c032-4347-af32-6a0a573aa5fe-credential-socket" pod "3f870408-c032-4347-af32-6a0a573aa5fe" (UID: "3f870408-c032-4347-af32-6a0a573aa5fe"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:14:46.869445 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:46.869419 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f870408-c032-4347-af32-6a0a573aa5fe-istiod-ca-cert" pod "3f870408-c032-4347-af32-6a0a573aa5fe" (UID: "3f870408-c032-4347-af32-6a0a573aa5fe"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:14:46.869689 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:46.869660 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f870408-c032-4347-af32-6a0a573aa5fe-workload-certs" pod "3f870408-c032-4347-af32-6a0a573aa5fe" (UID: "3f870408-c032-4347-af32-6a0a573aa5fe"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:14:46.871682 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:46.871643 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3f870408-c032-4347-af32-6a0a573aa5fe-istio-podinfo" pod "3f870408-c032-4347-af32-6a0a573aa5fe" (UID: "3f870408-c032-4347-af32-6a0a573aa5fe"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 04:14:46.871799 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:46.871731 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f870408-c032-4347-af32-6a0a573aa5fe-istio-envoy" pod "3f870408-c032-4347-af32-6a0a573aa5fe" (UID: "3f870408-c032-4347-af32-6a0a573aa5fe"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:14:46.872049 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:46.872028 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f870408-c032-4347-af32-6a0a573aa5fe-kube-api-access-9j466" pod "3f870408-c032-4347-af32-6a0a573aa5fe" (UID: "3f870408-c032-4347-af32-6a0a573aa5fe"). InnerVolumeSpecName "kube-api-access-9j466". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:14:46.872131 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:46.872057 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f870408-c032-4347-af32-6a0a573aa5fe-istio-token" pod "3f870408-c032-4347-af32-6a0a573aa5fe" (UID: "3f870408-c032-4347-af32-6a0a573aa5fe"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:14:46.872441 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:46.872411 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19b4fe89-322d-460f-8ccd-d5b950cbda65-metrics-certs" pod "19b4fe89-322d-460f-8ccd-d5b950cbda65" (UID: "19b4fe89-322d-460f-8ccd-d5b950cbda65"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:14:46.872523 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:46.872461 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19b4fe89-322d-460f-8ccd-d5b950cbda65-maas-api-tls" pod "19b4fe89-322d-460f-8ccd-d5b950cbda65" (UID: "19b4fe89-322d-460f-8ccd-d5b950cbda65"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:14:46.873055 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:46.873036 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19b4fe89-322d-460f-8ccd-d5b950cbda65-kube-api-access-bwxm7" pod "19b4fe89-322d-460f-8ccd-d5b950cbda65" (UID: "19b4fe89-322d-460f-8ccd-d5b950cbda65"). InnerVolumeSpecName "kube-api-access-bwxm7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:14:46.972248 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:46.972149 4376 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f870408-c032-4347-af32-6a0a573aa5fe-istiod-ca-cert\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:14:46.972549 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:46.972527 4376 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f870408-c032-4347-af32-6a0a573aa5fe-istio-envoy\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:14:46.972694 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:46.972678 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9j466\" (UniqueName: \"kubernetes.io/projected/3f870408-c032-4347-af32-6a0a573aa5fe-kube-api-access-9j466\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:14:46.972818 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:46.972790 4376 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f870408-c032-4347-af32-6a0a573aa5fe-workload-certs\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:14:46.972818 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:46.972814 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bwxm7\" (UniqueName: \"kubernetes.io/projected/19b4fe89-322d-460f-8ccd-d5b950cbda65-kube-api-access-bwxm7\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:14:46.972945 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:46.972830 4376 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/19b4fe89-322d-460f-8ccd-d5b950cbda65-metrics-certs\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:14:46.972945 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:46.972842 4376 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f870408-c032-4347-af32-6a0a573aa5fe-istio-token\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:14:46.972945 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:46.972854 4376 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f870408-c032-4347-af32-6a0a573aa5fe-workload-socket\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:14:46.972945 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:46.972868 4376 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f870408-c032-4347-af32-6a0a573aa5fe-istio-data\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:14:46.972945 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:46.972882 4376 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f870408-c032-4347-af32-6a0a573aa5fe-istio-podinfo\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:14:46.972945 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:46.972896 4376 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f870408-c032-4347-af32-6a0a573aa5fe-credential-socket\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:14:46.972945 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:46.972909 4376 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/19b4fe89-322d-460f-8ccd-d5b950cbda65-openshift-service-ca\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:14:46.972945 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:46.972923 4376 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/19b4fe89-322d-460f-8ccd-d5b950cbda65-maas-api-tls\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:14:47.135708 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:47.135669 4376 generic.go:356] "Generic (PLEG): container finished" podID="19b4fe89-322d-460f-8ccd-d5b950cbda65" containerID="c9c74826ced90edf3433ecc3762b3c961070917aabd532268b189e51fecb9abb" exitCode=0 Oct 31 04:14:47.135899 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:47.135723 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f8a8088d-7d6557f575-7vg47" Oct 31 04:14:47.135899 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:47.135753 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f8a8088d-7d6557f575-7vg47" event={"ID":"19b4fe89-322d-460f-8ccd-d5b950cbda65","Type":"ContainerDied","Data":"c9c74826ced90edf3433ecc3762b3c961070917aabd532268b189e51fecb9abb"} Oct 31 04:14:47.135899 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:47.135816 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f8a8088d-7d6557f575-7vg47" event={"ID":"19b4fe89-322d-460f-8ccd-d5b950cbda65","Type":"ContainerDied","Data":"5ee7759cb33ec04e740854b468314a62d448fd316520e50fa645e6a4a88796a4"} Oct 31 04:14:47.135899 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:47.135848 4376 scope.go:122] "RemoveContainer" containerID="c9c74826ced90edf3433ecc3762b3c961070917aabd532268b189e51fecb9abb" Oct 31 04:14:47.137804 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:47.137776 4376 generic.go:356] "Generic (PLEG): container finished" podID="3f870408-c032-4347-af32-6a0a573aa5fe" containerID="84dffa99310042b0addbd2d2d69f194d7fcb0644bc66c3f892e3ff1485c82763" exitCode=0 Oct 31 04:14:47.137978 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:47.137943 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4805aacc-openshift-default-78cc674747-nljxj" event={"ID":"3f870408-c032-4347-af32-6a0a573aa5fe","Type":"ContainerDied","Data":"84dffa99310042b0addbd2d2d69f194d7fcb0644bc66c3f892e3ff1485c82763"} Oct 31 04:14:47.138075 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:47.137998 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4805aacc-openshift-default-78cc674747-nljxj" event={"ID":"3f870408-c032-4347-af32-6a0a573aa5fe","Type":"ContainerDied","Data":"fe8c0d2d9a71a597baa03c8d6299ea47db0528f9a276fa68bfa05be49db7e97d"} Oct 31 04:14:47.138075 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:47.138013 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-4805aacc-openshift-default-78cc674747-nljxj" Oct 31 04:14:47.151218 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:47.151192 4376 scope.go:122] "RemoveContainer" containerID="c9c74826ced90edf3433ecc3762b3c961070917aabd532268b189e51fecb9abb" Oct 31 04:14:47.151603 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:14:47.151581 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9c74826ced90edf3433ecc3762b3c961070917aabd532268b189e51fecb9abb\": container with ID starting with c9c74826ced90edf3433ecc3762b3c961070917aabd532268b189e51fecb9abb not found: ID does not exist" containerID="c9c74826ced90edf3433ecc3762b3c961070917aabd532268b189e51fecb9abb" Oct 31 04:14:47.151694 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:47.151616 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9c74826ced90edf3433ecc3762b3c961070917aabd532268b189e51fecb9abb"} err="failed to get container status \"c9c74826ced90edf3433ecc3762b3c961070917aabd532268b189e51fecb9abb\": rpc error: code = NotFound desc = could not find container \"c9c74826ced90edf3433ecc3762b3c961070917aabd532268b189e51fecb9abb\": container with ID starting with c9c74826ced90edf3433ecc3762b3c961070917aabd532268b189e51fecb9abb not found: ID does not exist" Oct 31 04:14:47.151694 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:47.151644 4376 scope.go:122] "RemoveContainer" containerID="84dffa99310042b0addbd2d2d69f194d7fcb0644bc66c3f892e3ff1485c82763" Oct 31 04:14:47.162434 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:47.162413 4376 scope.go:122] "RemoveContainer" containerID="8e1655be4ab5787d2a2b1f7d8e19850d2afb27ac91d330e34c268a7cdb30b3f1" Oct 31 04:14:47.171443 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:47.171418 4376 scope.go:122] "RemoveContainer" containerID="84dffa99310042b0addbd2d2d69f194d7fcb0644bc66c3f892e3ff1485c82763" Oct 31 04:14:47.171786 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:14:47.171753 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84dffa99310042b0addbd2d2d69f194d7fcb0644bc66c3f892e3ff1485c82763\": container with ID starting with 84dffa99310042b0addbd2d2d69f194d7fcb0644bc66c3f892e3ff1485c82763 not found: ID does not exist" containerID="84dffa99310042b0addbd2d2d69f194d7fcb0644bc66c3f892e3ff1485c82763" Oct 31 04:14:47.171860 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:47.171799 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84dffa99310042b0addbd2d2d69f194d7fcb0644bc66c3f892e3ff1485c82763"} err="failed to get container status \"84dffa99310042b0addbd2d2d69f194d7fcb0644bc66c3f892e3ff1485c82763\": rpc error: code = NotFound desc = could not find container \"84dffa99310042b0addbd2d2d69f194d7fcb0644bc66c3f892e3ff1485c82763\": container with ID starting with 84dffa99310042b0addbd2d2d69f194d7fcb0644bc66c3f892e3ff1485c82763 not found: ID does not exist" Oct 31 04:14:47.171860 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:47.171826 4376 scope.go:122] "RemoveContainer" containerID="8e1655be4ab5787d2a2b1f7d8e19850d2afb27ac91d330e34c268a7cdb30b3f1" Oct 31 04:14:47.172122 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:14:47.172102 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e1655be4ab5787d2a2b1f7d8e19850d2afb27ac91d330e34c268a7cdb30b3f1\": container with ID starting with 8e1655be4ab5787d2a2b1f7d8e19850d2afb27ac91d330e34c268a7cdb30b3f1 not found: ID does not exist" containerID="8e1655be4ab5787d2a2b1f7d8e19850d2afb27ac91d330e34c268a7cdb30b3f1" Oct 31 04:14:47.172193 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:47.172127 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e1655be4ab5787d2a2b1f7d8e19850d2afb27ac91d330e34c268a7cdb30b3f1"} err="failed to get container status \"8e1655be4ab5787d2a2b1f7d8e19850d2afb27ac91d330e34c268a7cdb30b3f1\": rpc error: code = NotFound desc = could not find container \"8e1655be4ab5787d2a2b1f7d8e19850d2afb27ac91d330e34c268a7cdb30b3f1\": container with ID starting with 8e1655be4ab5787d2a2b1f7d8e19850d2afb27ac91d330e34c268a7cdb30b3f1 not found: ID does not exist" Oct 31 04:14:47.194240 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:47.194203 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-4805aacc-openshift-default-78cc674747-nljxj"] Oct 31 04:14:47.208599 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:47.208568 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-4805aacc-openshift-default-78cc674747-nljxj"] Oct 31 04:14:47.239509 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:47.239465 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-f8a8088d-7d6557f575-7vg47"] Oct 31 04:14:47.242741 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:47.242716 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-f8a8088d-7d6557f575-7vg47"] Oct 31 04:14:47.540412 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:47.540372 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-44f032ad-868c96f9c9-fvrrj" podUID="cd9af128-7cf1-4c41-96f1-c159eaa13692" containerName="payload-processing" containerID="cri-o://88d1c735e4c49ecbb2717bed6b8c4d719b62ff35eeece0965899c73929d0b809" gracePeriod=25 Oct 31 04:14:47.577839 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:47.577797 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-44f032ad-7b89c4449f-gjfv8" podUID="9b428b05-ad88-44cc-b719-85d7cd8ae9b2" containerName="payload-pre-processing" containerID="cri-o://6dea152092f5b56b46ba44229b2cbb8615777cb24ece1601cbe10ee6b3c7871b" gracePeriod=25 Oct 31 04:14:47.823954 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:47.823932 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-44f032ad-868c96f9c9-fvrrj" Oct 31 04:14:47.849502 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:47.849478 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-44f032ad-7b89c4449f-gjfv8" Oct 31 04:14:47.986498 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:47.986403 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b428b05-ad88-44cc-b719-85d7cd8ae9b2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b428b05-ad88-44cc-b719-85d7cd8ae9b2-plugins-config-volume\") pod \"9b428b05-ad88-44cc-b719-85d7cd8ae9b2\" (UID: \"9b428b05-ad88-44cc-b719-85d7cd8ae9b2\") " Oct 31 04:14:47.986498 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:47.986470 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b428b05-ad88-44cc-b719-85d7cd8ae9b2-kube-api-access-vvtzx\" (UniqueName: \"kubernetes.io/projected/9b428b05-ad88-44cc-b719-85d7cd8ae9b2-kube-api-access-vvtzx\") pod \"9b428b05-ad88-44cc-b719-85d7cd8ae9b2\" (UID: \"9b428b05-ad88-44cc-b719-85d7cd8ae9b2\") " Oct 31 04:14:47.986498 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:47.986496 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd9af128-7cf1-4c41-96f1-c159eaa13692-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd9af128-7cf1-4c41-96f1-c159eaa13692-plugins-config-volume\") pod \"cd9af128-7cf1-4c41-96f1-c159eaa13692\" (UID: \"cd9af128-7cf1-4c41-96f1-c159eaa13692\") " Oct 31 04:14:47.986775 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:47.986602 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd9af128-7cf1-4c41-96f1-c159eaa13692-kube-api-access-nmbd2\" (UniqueName: \"kubernetes.io/projected/cd9af128-7cf1-4c41-96f1-c159eaa13692-kube-api-access-nmbd2\") pod \"cd9af128-7cf1-4c41-96f1-c159eaa13692\" (UID: \"cd9af128-7cf1-4c41-96f1-c159eaa13692\") " Oct 31 04:14:47.986829 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:47.986766 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b428b05-ad88-44cc-b719-85d7cd8ae9b2-plugins-config-volume" pod "9b428b05-ad88-44cc-b719-85d7cd8ae9b2" (UID: "9b428b05-ad88-44cc-b719-85d7cd8ae9b2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:14:47.986955 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:47.986932 4376 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b428b05-ad88-44cc-b719-85d7cd8ae9b2-plugins-config-volume\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:14:47.987101 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:47.986992 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd9af128-7cf1-4c41-96f1-c159eaa13692-plugins-config-volume" pod "cd9af128-7cf1-4c41-96f1-c159eaa13692" (UID: "cd9af128-7cf1-4c41-96f1-c159eaa13692"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:14:47.988700 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:47.988680 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd9af128-7cf1-4c41-96f1-c159eaa13692-kube-api-access-nmbd2" pod "cd9af128-7cf1-4c41-96f1-c159eaa13692" (UID: "cd9af128-7cf1-4c41-96f1-c159eaa13692"). InnerVolumeSpecName "kube-api-access-nmbd2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:14:47.988760 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:47.988711 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b428b05-ad88-44cc-b719-85d7cd8ae9b2-kube-api-access-vvtzx" pod "9b428b05-ad88-44cc-b719-85d7cd8ae9b2" (UID: "9b428b05-ad88-44cc-b719-85d7cd8ae9b2"). InnerVolumeSpecName "kube-api-access-vvtzx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:14:48.087605 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:48.087575 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nmbd2\" (UniqueName: \"kubernetes.io/projected/cd9af128-7cf1-4c41-96f1-c159eaa13692-kube-api-access-nmbd2\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:14:48.087605 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:48.087601 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvtzx\" (UniqueName: \"kubernetes.io/projected/9b428b05-ad88-44cc-b719-85d7cd8ae9b2-kube-api-access-vvtzx\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:14:48.087605 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:48.087610 4376 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd9af128-7cf1-4c41-96f1-c159eaa13692-plugins-config-volume\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:14:48.145465 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:48.145429 4376 generic.go:356] "Generic (PLEG): container finished" podID="cd9af128-7cf1-4c41-96f1-c159eaa13692" containerID="88d1c735e4c49ecbb2717bed6b8c4d719b62ff35eeece0965899c73929d0b809" exitCode=0 Oct 31 04:14:48.145618 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:48.145475 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-44f032ad-868c96f9c9-fvrrj" Oct 31 04:14:48.145618 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:48.145518 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-44f032ad-868c96f9c9-fvrrj" event={"ID":"cd9af128-7cf1-4c41-96f1-c159eaa13692","Type":"ContainerDied","Data":"88d1c735e4c49ecbb2717bed6b8c4d719b62ff35eeece0965899c73929d0b809"} Oct 31 04:14:48.145618 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:48.145577 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-44f032ad-868c96f9c9-fvrrj" event={"ID":"cd9af128-7cf1-4c41-96f1-c159eaa13692","Type":"ContainerDied","Data":"92ebf5bbf0af4abb8d61a6159cee6795848aee89e441a6c7f4554aea51fd63b9"} Oct 31 04:14:48.145618 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:48.145614 4376 scope.go:122] "RemoveContainer" containerID="88d1c735e4c49ecbb2717bed6b8c4d719b62ff35eeece0965899c73929d0b809" Oct 31 04:14:48.147962 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:48.147932 4376 generic.go:356] "Generic (PLEG): container finished" podID="9b428b05-ad88-44cc-b719-85d7cd8ae9b2" containerID="6dea152092f5b56b46ba44229b2cbb8615777cb24ece1601cbe10ee6b3c7871b" exitCode=0 Oct 31 04:14:48.148078 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:48.147972 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-44f032ad-7b89c4449f-gjfv8" Oct 31 04:14:48.148078 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:48.147984 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-44f032ad-7b89c4449f-gjfv8" event={"ID":"9b428b05-ad88-44cc-b719-85d7cd8ae9b2","Type":"ContainerDied","Data":"6dea152092f5b56b46ba44229b2cbb8615777cb24ece1601cbe10ee6b3c7871b"} Oct 31 04:14:48.148078 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:48.148019 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-44f032ad-7b89c4449f-gjfv8" event={"ID":"9b428b05-ad88-44cc-b719-85d7cd8ae9b2","Type":"ContainerDied","Data":"562b10620e58c9c580410786b6675d3ab0ef0bf927a1a03dbab5c5b0588b647b"} Oct 31 04:14:48.155249 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:48.155227 4376 scope.go:122] "RemoveContainer" containerID="88d1c735e4c49ecbb2717bed6b8c4d719b62ff35eeece0965899c73929d0b809" Oct 31 04:14:48.155547 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:14:48.155526 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88d1c735e4c49ecbb2717bed6b8c4d719b62ff35eeece0965899c73929d0b809\": container with ID starting with 88d1c735e4c49ecbb2717bed6b8c4d719b62ff35eeece0965899c73929d0b809 not found: ID does not exist" containerID="88d1c735e4c49ecbb2717bed6b8c4d719b62ff35eeece0965899c73929d0b809" Oct 31 04:14:48.155621 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:48.155552 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88d1c735e4c49ecbb2717bed6b8c4d719b62ff35eeece0965899c73929d0b809"} err="failed to get container status \"88d1c735e4c49ecbb2717bed6b8c4d719b62ff35eeece0965899c73929d0b809\": rpc error: code = NotFound desc = could not find container \"88d1c735e4c49ecbb2717bed6b8c4d719b62ff35eeece0965899c73929d0b809\": container with ID starting with 88d1c735e4c49ecbb2717bed6b8c4d719b62ff35eeece0965899c73929d0b809 not found: ID does not exist" Oct 31 04:14:48.155621 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:48.155569 4376 scope.go:122] "RemoveContainer" containerID="6dea152092f5b56b46ba44229b2cbb8615777cb24ece1601cbe10ee6b3c7871b" Oct 31 04:14:48.163916 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:48.163899 4376 scope.go:122] "RemoveContainer" containerID="6dea152092f5b56b46ba44229b2cbb8615777cb24ece1601cbe10ee6b3c7871b" Oct 31 04:14:48.164186 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:14:48.164165 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6dea152092f5b56b46ba44229b2cbb8615777cb24ece1601cbe10ee6b3c7871b\": container with ID starting with 6dea152092f5b56b46ba44229b2cbb8615777cb24ece1601cbe10ee6b3c7871b not found: ID does not exist" containerID="6dea152092f5b56b46ba44229b2cbb8615777cb24ece1601cbe10ee6b3c7871b" Oct 31 04:14:48.164254 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:48.164196 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6dea152092f5b56b46ba44229b2cbb8615777cb24ece1601cbe10ee6b3c7871b"} err="failed to get container status \"6dea152092f5b56b46ba44229b2cbb8615777cb24ece1601cbe10ee6b3c7871b\": rpc error: code = NotFound desc = could not find container \"6dea152092f5b56b46ba44229b2cbb8615777cb24ece1601cbe10ee6b3c7871b\": container with ID starting with 6dea152092f5b56b46ba44229b2cbb8615777cb24ece1601cbe10ee6b3c7871b not found: ID does not exist" Oct 31 04:14:48.203302 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:48.203267 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-44f032ad-7b89c4449f-gjfv8"] Oct 31 04:14:48.204932 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:48.204910 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-44f032ad-7b89c4449f-gjfv8"] Oct 31 04:14:48.242348 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:48.242254 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-44f032ad-868c96f9c9-fvrrj"] Oct 31 04:14:48.253257 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:48.253229 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-44f032ad-868c96f9c9-fvrrj"] Oct 31 04:14:48.442240 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:48.442211 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19b4fe89-322d-460f-8ccd-d5b950cbda65" path="/var/lib/kubelet/pods/19b4fe89-322d-460f-8ccd-d5b950cbda65/volumes" Oct 31 04:14:48.442678 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:48.442663 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f870408-c032-4347-af32-6a0a573aa5fe" path="/var/lib/kubelet/pods/3f870408-c032-4347-af32-6a0a573aa5fe/volumes" Oct 31 04:14:48.442990 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:48.442978 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b428b05-ad88-44cc-b719-85d7cd8ae9b2" path="/var/lib/kubelet/pods/9b428b05-ad88-44cc-b719-85d7cd8ae9b2/volumes" Oct 31 04:14:48.443595 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:48.443579 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd9af128-7cf1-4c41-96f1-c159eaa13692" path="/var/lib/kubelet/pods/cd9af128-7cf1-4c41-96f1-c159eaa13692/volumes" Oct 31 04:14:48.706547 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:48.706507 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-96rgs"] Oct 31 04:14:48.707116 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:48.707095 4376 cpu_manager.go:414] "containerMap: removing container" podUID="3f870408-c032-4347-af32-6a0a573aa5fe" containerName="istio-proxy" Oct 31 04:14:48.707212 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:48.707118 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f870408-c032-4347-af32-6a0a573aa5fe" containerName="istio-proxy" Oct 31 04:14:48.707212 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:48.707140 4376 cpu_manager.go:414] "containerMap: removing container" podUID="3f870408-c032-4347-af32-6a0a573aa5fe" containerName="istio-proxy" Oct 31 04:14:48.707212 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:48.707147 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f870408-c032-4347-af32-6a0a573aa5fe" containerName="istio-proxy" Oct 31 04:14:48.707212 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:48.707168 4376 cpu_manager.go:414] "containerMap: removing container" podUID="19b4fe89-322d-460f-8ccd-d5b950cbda65" containerName="maas-api" Oct 31 04:14:48.707212 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:48.707176 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19b4fe89-322d-460f-8ccd-d5b950cbda65" containerName="maas-api" Oct 31 04:14:48.707212 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:48.707191 4376 cpu_manager.go:414] "containerMap: removing container" podUID="cd9af128-7cf1-4c41-96f1-c159eaa13692" containerName="payload-processing" Oct 31 04:14:48.707212 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:48.707198 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd9af128-7cf1-4c41-96f1-c159eaa13692" containerName="payload-processing" Oct 31 04:14:48.707212 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:48.707212 4376 cpu_manager.go:414] "containerMap: removing container" podUID="9b428b05-ad88-44cc-b719-85d7cd8ae9b2" containerName="payload-pre-processing" Oct 31 04:14:48.707682 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:48.707219 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b428b05-ad88-44cc-b719-85d7cd8ae9b2" containerName="payload-pre-processing" Oct 31 04:14:48.707682 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:48.707319 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="19b4fe89-322d-460f-8ccd-d5b950cbda65" containerName="maas-api" Oct 31 04:14:48.707682 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:48.707369 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd9af128-7cf1-4c41-96f1-c159eaa13692" containerName="payload-processing" Oct 31 04:14:48.707682 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:48.707380 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b428b05-ad88-44cc-b719-85d7cd8ae9b2" containerName="payload-pre-processing" Oct 31 04:14:48.707682 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:48.707394 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f870408-c032-4347-af32-6a0a573aa5fe" containerName="istio-proxy" Oct 31 04:14:48.707682 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:48.707404 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f870408-c032-4347-af32-6a0a573aa5fe" containerName="istio-proxy" Oct 31 04:14:48.712078 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:48.712058 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-96rgs" Oct 31 04:14:48.732202 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:48.732170 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-96rgs"] Oct 31 04:14:48.733864 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:48.733842 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-439989-openshift-default-dockercfg-9247r\"" Oct 31 04:14:48.761386 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:48.761353 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-965f61-84cf84d479-gq7q5"] Oct 31 04:14:48.765135 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:48.765104 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-965f61-84cf84d479-gq7q5" Oct 31 04:14:48.787660 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:48.787631 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-965f61-84cf84d479-gq7q5"] Oct 31 04:14:48.794538 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:48.794512 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b3b5806-b3d7-48a8-b2d5-fb202004605d-istio-podinfo\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-96rgs\" (UID: \"2b3b5806-b3d7-48a8-b2d5-fb202004605d\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-96rgs" Oct 31 04:14:48.794684 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:48.794548 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b3b5806-b3d7-48a8-b2d5-fb202004605d-workload-certs\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-96rgs\" (UID: \"2b3b5806-b3d7-48a8-b2d5-fb202004605d\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-96rgs" Oct 31 04:14:48.794684 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:48.794638 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b3b5806-b3d7-48a8-b2d5-fb202004605d-istio-data\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-96rgs\" (UID: \"2b3b5806-b3d7-48a8-b2d5-fb202004605d\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-96rgs" Oct 31 04:14:48.794843 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:48.794816 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b3b5806-b3d7-48a8-b2d5-fb202004605d-istio-envoy\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-96rgs\" (UID: \"2b3b5806-b3d7-48a8-b2d5-fb202004605d\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-96rgs" Oct 31 04:14:48.794964 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:48.794892 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b3b5806-b3d7-48a8-b2d5-fb202004605d-credential-socket\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-96rgs\" (UID: \"2b3b5806-b3d7-48a8-b2d5-fb202004605d\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-96rgs" Oct 31 04:14:48.794964 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:48.794932 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbjt2\" (UniqueName: \"kubernetes.io/projected/2b3b5806-b3d7-48a8-b2d5-fb202004605d-kube-api-access-tbjt2\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-96rgs\" (UID: \"2b3b5806-b3d7-48a8-b2d5-fb202004605d\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-96rgs" Oct 31 04:14:48.795072 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:48.794992 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b3b5806-b3d7-48a8-b2d5-fb202004605d-workload-socket\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-96rgs\" (UID: \"2b3b5806-b3d7-48a8-b2d5-fb202004605d\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-96rgs" Oct 31 04:14:48.795072 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:48.795051 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b3b5806-b3d7-48a8-b2d5-fb202004605d-istiod-ca-cert\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-96rgs\" (UID: \"2b3b5806-b3d7-48a8-b2d5-fb202004605d\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-96rgs" Oct 31 04:14:48.795151 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:48.795085 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b3b5806-b3d7-48a8-b2d5-fb202004605d-istio-token\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-96rgs\" (UID: \"2b3b5806-b3d7-48a8-b2d5-fb202004605d\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-96rgs" Oct 31 04:14:48.795432 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:48.795412 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-965f61\"" Oct 31 04:14:48.826780 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:48.826742 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 04:14:48.833774 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:48.833745 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-965f61-dockercfg-hvtlg\"" Oct 31 04:14:48.843424 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:48.843389 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-965f61-576d68cfbf-ffkvw"] Oct 31 04:14:48.847189 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:48.847165 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-965f61-576d68cfbf-ffkvw" Oct 31 04:14:48.866036 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:48.866004 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-965f61-576d68cfbf-ffkvw"] Oct 31 04:14:48.895605 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:48.895570 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b3b5806-b3d7-48a8-b2d5-fb202004605d-workload-certs\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-96rgs\" (UID: \"2b3b5806-b3d7-48a8-b2d5-fb202004605d\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-96rgs" Oct 31 04:14:48.895798 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:48.895611 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b3b5806-b3d7-48a8-b2d5-fb202004605d-istio-data\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-96rgs\" (UID: \"2b3b5806-b3d7-48a8-b2d5-fb202004605d\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-96rgs" Oct 31 04:14:48.895864 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:48.895824 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7blqt\" (UniqueName: \"kubernetes.io/projected/2219d6d2-75e4-4fd8-ae05-901e3ac891a1-kube-api-access-7blqt\") pod \"payload-pre-processing-e2e-worker-w0-965f61-84cf84d479-gq7q5\" (UID: \"2219d6d2-75e4-4fd8-ae05-901e3ac891a1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-965f61-84cf84d479-gq7q5" Oct 31 04:14:48.895919 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:48.895876 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b3b5806-b3d7-48a8-b2d5-fb202004605d-istio-envoy\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-96rgs\" (UID: \"2b3b5806-b3d7-48a8-b2d5-fb202004605d\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-96rgs" Oct 31 04:14:48.895987 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:48.895957 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b3b5806-b3d7-48a8-b2d5-fb202004605d-credential-socket\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-96rgs\" (UID: \"2b3b5806-b3d7-48a8-b2d5-fb202004605d\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-96rgs" Oct 31 04:14:48.896126 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:48.896011 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b3b5806-b3d7-48a8-b2d5-fb202004605d-workload-certs\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-96rgs\" (UID: \"2b3b5806-b3d7-48a8-b2d5-fb202004605d\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-96rgs" Oct 31 04:14:48.896126 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:48.896030 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbjt2\" (UniqueName: \"kubernetes.io/projected/2b3b5806-b3d7-48a8-b2d5-fb202004605d-kube-api-access-tbjt2\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-96rgs\" (UID: \"2b3b5806-b3d7-48a8-b2d5-fb202004605d\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-96rgs" Oct 31 04:14:48.896126 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:48.896075 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b3b5806-b3d7-48a8-b2d5-fb202004605d-workload-socket\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-96rgs\" (UID: \"2b3b5806-b3d7-48a8-b2d5-fb202004605d\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-96rgs" Oct 31 04:14:48.896126 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:48.896101 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b3b5806-b3d7-48a8-b2d5-fb202004605d-istiod-ca-cert\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-96rgs\" (UID: \"2b3b5806-b3d7-48a8-b2d5-fb202004605d\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-96rgs" Oct 31 04:14:48.896126 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:48.895967 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b3b5806-b3d7-48a8-b2d5-fb202004605d-istio-data\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-96rgs\" (UID: \"2b3b5806-b3d7-48a8-b2d5-fb202004605d\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-96rgs" Oct 31 04:14:48.896126 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:48.896124 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b3b5806-b3d7-48a8-b2d5-fb202004605d-istio-token\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-96rgs\" (UID: \"2b3b5806-b3d7-48a8-b2d5-fb202004605d\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-96rgs" Oct 31 04:14:48.896450 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:48.896188 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2219d6d2-75e4-4fd8-ae05-901e3ac891a1-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-965f61-84cf84d479-gq7q5\" (UID: \"2219d6d2-75e4-4fd8-ae05-901e3ac891a1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-965f61-84cf84d479-gq7q5" Oct 31 04:14:48.896450 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:48.896232 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b3b5806-b3d7-48a8-b2d5-fb202004605d-credential-socket\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-96rgs\" (UID: \"2b3b5806-b3d7-48a8-b2d5-fb202004605d\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-96rgs" Oct 31 04:14:48.896450 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:48.896251 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b3b5806-b3d7-48a8-b2d5-fb202004605d-istio-podinfo\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-96rgs\" (UID: \"2b3b5806-b3d7-48a8-b2d5-fb202004605d\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-96rgs" Oct 31 04:14:48.896450 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:48.896395 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b3b5806-b3d7-48a8-b2d5-fb202004605d-workload-socket\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-96rgs\" (UID: \"2b3b5806-b3d7-48a8-b2d5-fb202004605d\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-96rgs" Oct 31 04:14:48.896768 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:48.896746 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b3b5806-b3d7-48a8-b2d5-fb202004605d-istiod-ca-cert\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-96rgs\" (UID: \"2b3b5806-b3d7-48a8-b2d5-fb202004605d\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-96rgs" Oct 31 04:14:48.898267 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:48.898246 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b3b5806-b3d7-48a8-b2d5-fb202004605d-istio-envoy\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-96rgs\" (UID: \"2b3b5806-b3d7-48a8-b2d5-fb202004605d\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-96rgs" Oct 31 04:14:48.898831 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:48.898815 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b3b5806-b3d7-48a8-b2d5-fb202004605d-istio-podinfo\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-96rgs\" (UID: \"2b3b5806-b3d7-48a8-b2d5-fb202004605d\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-96rgs" Oct 31 04:14:48.921669 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:48.921639 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b3b5806-b3d7-48a8-b2d5-fb202004605d-istio-token\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-96rgs\" (UID: \"2b3b5806-b3d7-48a8-b2d5-fb202004605d\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-96rgs" Oct 31 04:14:48.921822 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:48.921737 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbjt2\" (UniqueName: \"kubernetes.io/projected/2b3b5806-b3d7-48a8-b2d5-fb202004605d-kube-api-access-tbjt2\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-96rgs\" (UID: \"2b3b5806-b3d7-48a8-b2d5-fb202004605d\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-96rgs" Oct 31 04:14:49.001118 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:48.998639 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2219d6d2-75e4-4fd8-ae05-901e3ac891a1-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-965f61-84cf84d479-gq7q5\" (UID: \"2219d6d2-75e4-4fd8-ae05-901e3ac891a1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-965f61-84cf84d479-gq7q5" Oct 31 04:14:49.001118 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:48.998756 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5z6w\" (UniqueName: \"kubernetes.io/projected/d2870b2c-9c85-43d1-a726-38edfcf6ae2b-kube-api-access-w5z6w\") pod \"payload-processing-e2e-worker-w0-965f61-576d68cfbf-ffkvw\" (UID: \"d2870b2c-9c85-43d1-a726-38edfcf6ae2b\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-965f61-576d68cfbf-ffkvw" Oct 31 04:14:49.001118 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:48.998843 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7blqt\" (UniqueName: \"kubernetes.io/projected/2219d6d2-75e4-4fd8-ae05-901e3ac891a1-kube-api-access-7blqt\") pod \"payload-pre-processing-e2e-worker-w0-965f61-84cf84d479-gq7q5\" (UID: \"2219d6d2-75e4-4fd8-ae05-901e3ac891a1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-965f61-84cf84d479-gq7q5" Oct 31 04:14:49.001118 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:48.998920 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2870b2c-9c85-43d1-a726-38edfcf6ae2b-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-965f61-576d68cfbf-ffkvw\" (UID: \"d2870b2c-9c85-43d1-a726-38edfcf6ae2b\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-965f61-576d68cfbf-ffkvw" Oct 31 04:14:49.001118 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:49.000250 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2219d6d2-75e4-4fd8-ae05-901e3ac891a1-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-965f61-84cf84d479-gq7q5\" (UID: \"2219d6d2-75e4-4fd8-ae05-901e3ac891a1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-965f61-84cf84d479-gq7q5" Oct 31 04:14:49.027038 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:49.026988 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-96rgs" Oct 31 04:14:49.029700 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:49.029668 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7blqt\" (UniqueName: \"kubernetes.io/projected/2219d6d2-75e4-4fd8-ae05-901e3ac891a1-kube-api-access-7blqt\") pod \"payload-pre-processing-e2e-worker-w0-965f61-84cf84d479-gq7q5\" (UID: \"2219d6d2-75e4-4fd8-ae05-901e3ac891a1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-965f61-84cf84d479-gq7q5" Oct 31 04:14:49.072676 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:49.072631 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-44f032ad-666df5c575-kxg5z"] Oct 31 04:14:49.072938 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:49.072910 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-44f032ad-666df5c575-kxg5z" podUID="f8e1248b-7093-4b13-997d-4c3005c02110" containerName="maas-api" containerID="cri-o://f5f07db5caf75271c375e5a5bdc31a8d79f5368cb9937d70ea5f0dbb73f7a95a" gracePeriod=30 Oct 31 04:14:49.080988 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:49.080763 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-965f61-84cf84d479-gq7q5" Oct 31 04:14:49.100507 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:49.099903 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5z6w\" (UniqueName: \"kubernetes.io/projected/d2870b2c-9c85-43d1-a726-38edfcf6ae2b-kube-api-access-w5z6w\") pod \"payload-processing-e2e-worker-w0-965f61-576d68cfbf-ffkvw\" (UID: \"d2870b2c-9c85-43d1-a726-38edfcf6ae2b\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-965f61-576d68cfbf-ffkvw" Oct 31 04:14:49.100507 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:49.100287 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2870b2c-9c85-43d1-a726-38edfcf6ae2b-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-965f61-576d68cfbf-ffkvw\" (UID: \"d2870b2c-9c85-43d1-a726-38edfcf6ae2b\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-965f61-576d68cfbf-ffkvw" Oct 31 04:14:49.101025 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:49.101003 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2870b2c-9c85-43d1-a726-38edfcf6ae2b-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-965f61-576d68cfbf-ffkvw\" (UID: \"d2870b2c-9c85-43d1-a726-38edfcf6ae2b\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-965f61-576d68cfbf-ffkvw" Oct 31 04:14:49.126803 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:49.126739 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5z6w\" (UniqueName: \"kubernetes.io/projected/d2870b2c-9c85-43d1-a726-38edfcf6ae2b-kube-api-access-w5z6w\") pod \"payload-processing-e2e-worker-w0-965f61-576d68cfbf-ffkvw\" (UID: \"d2870b2c-9c85-43d1-a726-38edfcf6ae2b\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-965f61-576d68cfbf-ffkvw" Oct 31 04:14:49.163250 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:49.161506 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-965f61-576d68cfbf-ffkvw" Oct 31 04:14:49.360947 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:49.360918 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-44f032ad-666df5c575-kxg5z" Oct 31 04:14:49.393634 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:14:49.393607 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b3b5806_b3d7_48a8_b2d5_fb202004605d.slice/crio-8e2d0d016fc10ffc1a10ebeb873fdd6fa4f59e1a8aa687e691505622300ff885 WatchSource:0}: Error finding container 8e2d0d016fc10ffc1a10ebeb873fdd6fa4f59e1a8aa687e691505622300ff885: Status 404 returned error can't find the container with id 8e2d0d016fc10ffc1a10ebeb873fdd6fa4f59e1a8aa687e691505622300ff885 Oct 31 04:14:49.394349 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:49.394307 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-96rgs"] Oct 31 04:14:49.395933 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:49.395901 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:14:49.396024 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:49.395976 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:14:49.396092 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:49.396019 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:14:49.403983 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:49.403960 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-965f61-84cf84d479-gq7q5"] Oct 31 04:14:49.404917 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:14:49.404894 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2219d6d2_75e4_4fd8_ae05_901e3ac891a1.slice/crio-ff435e3dfb7eb0ddcd685d214c5d930d01f1dff36886705124784852c0d3a58a WatchSource:0}: Error finding container ff435e3dfb7eb0ddcd685d214c5d930d01f1dff36886705124784852c0d3a58a: Status 404 returned error can't find the container with id ff435e3dfb7eb0ddcd685d214c5d930d01f1dff36886705124784852c0d3a58a Oct 31 04:14:49.478757 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:49.478503 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-965f61-576d68cfbf-ffkvw"] Oct 31 04:14:49.506281 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:49.505223 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f8e1248b-7093-4b13-997d-4c3005c02110-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8e1248b-7093-4b13-997d-4c3005c02110-metrics-certs\") pod \"f8e1248b-7093-4b13-997d-4c3005c02110\" (UID: \"f8e1248b-7093-4b13-997d-4c3005c02110\") " Oct 31 04:14:49.506281 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:49.505277 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8e1248b-7093-4b13-997d-4c3005c02110-kube-api-access-zkknc\" (UniqueName: \"kubernetes.io/projected/f8e1248b-7093-4b13-997d-4c3005c02110-kube-api-access-zkknc\") pod \"f8e1248b-7093-4b13-997d-4c3005c02110\" (UID: \"f8e1248b-7093-4b13-997d-4c3005c02110\") " Oct 31 04:14:49.506281 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:49.505495 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f8e1248b-7093-4b13-997d-4c3005c02110-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8e1248b-7093-4b13-997d-4c3005c02110-maas-api-tls\") pod \"f8e1248b-7093-4b13-997d-4c3005c02110\" (UID: \"f8e1248b-7093-4b13-997d-4c3005c02110\") " Oct 31 04:14:49.506281 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:49.505527 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8e1248b-7093-4b13-997d-4c3005c02110-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8e1248b-7093-4b13-997d-4c3005c02110-openshift-service-ca\") pod \"f8e1248b-7093-4b13-997d-4c3005c02110\" (UID: \"f8e1248b-7093-4b13-997d-4c3005c02110\") " Oct 31 04:14:49.506281 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:49.506099 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8e1248b-7093-4b13-997d-4c3005c02110-openshift-service-ca" pod "f8e1248b-7093-4b13-997d-4c3005c02110" (UID: "f8e1248b-7093-4b13-997d-4c3005c02110"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:14:49.508875 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:49.508824 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8e1248b-7093-4b13-997d-4c3005c02110-metrics-certs" pod "f8e1248b-7093-4b13-997d-4c3005c02110" (UID: "f8e1248b-7093-4b13-997d-4c3005c02110"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:14:49.509019 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:49.508999 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8e1248b-7093-4b13-997d-4c3005c02110-maas-api-tls" pod "f8e1248b-7093-4b13-997d-4c3005c02110" (UID: "f8e1248b-7093-4b13-997d-4c3005c02110"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:14:49.509860 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:49.509836 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8e1248b-7093-4b13-997d-4c3005c02110-kube-api-access-zkknc" pod "f8e1248b-7093-4b13-997d-4c3005c02110" (UID: "f8e1248b-7093-4b13-997d-4c3005c02110"). InnerVolumeSpecName "kube-api-access-zkknc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:14:49.607542 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:49.607508 4376 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8e1248b-7093-4b13-997d-4c3005c02110-maas-api-tls\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:14:49.607695 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:49.607544 4376 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8e1248b-7093-4b13-997d-4c3005c02110-openshift-service-ca\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:14:49.607695 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:49.607563 4376 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8e1248b-7093-4b13-997d-4c3005c02110-metrics-certs\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:14:49.607695 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:49.607578 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkknc\" (UniqueName: \"kubernetes.io/projected/f8e1248b-7093-4b13-997d-4c3005c02110-kube-api-access-zkknc\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:14:50.166018 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:50.165973 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-965f61-84cf84d479-gq7q5" event={"ID":"2219d6d2-75e4-4fd8-ae05-901e3ac891a1","Type":"ContainerStarted","Data":"8c2df74c6a9dad654e97f0509fd434a7aacb58e86933be659e7592df76eed988"} Oct 31 04:14:50.166445 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:50.166035 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-965f61-84cf84d479-gq7q5" event={"ID":"2219d6d2-75e4-4fd8-ae05-901e3ac891a1","Type":"ContainerStarted","Data":"ff435e3dfb7eb0ddcd685d214c5d930d01f1dff36886705124784852c0d3a58a"} Oct 31 04:14:50.166445 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:50.166088 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-965f61-84cf84d479-gq7q5" Oct 31 04:14:50.167761 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:50.167732 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-96rgs" event={"ID":"2b3b5806-b3d7-48a8-b2d5-fb202004605d","Type":"ContainerStarted","Data":"903f8ec8804650023256d4b08dbb9e0cf0299163173d3e83a7ae97003a6ac292"} Oct 31 04:14:50.167874 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:50.167775 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-96rgs" event={"ID":"2b3b5806-b3d7-48a8-b2d5-fb202004605d","Type":"ContainerStarted","Data":"8e2d0d016fc10ffc1a10ebeb873fdd6fa4f59e1a8aa687e691505622300ff885"} Oct 31 04:14:50.168924 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:50.168905 4376 generic.go:356] "Generic (PLEG): container finished" podID="f8e1248b-7093-4b13-997d-4c3005c02110" containerID="f5f07db5caf75271c375e5a5bdc31a8d79f5368cb9937d70ea5f0dbb73f7a95a" exitCode=0 Oct 31 04:14:50.169032 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:50.168942 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-44f032ad-666df5c575-kxg5z" Oct 31 04:14:50.169032 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:50.168961 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-44f032ad-666df5c575-kxg5z" event={"ID":"f8e1248b-7093-4b13-997d-4c3005c02110","Type":"ContainerDied","Data":"f5f07db5caf75271c375e5a5bdc31a8d79f5368cb9937d70ea5f0dbb73f7a95a"} Oct 31 04:14:50.169032 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:50.168995 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-44f032ad-666df5c575-kxg5z" event={"ID":"f8e1248b-7093-4b13-997d-4c3005c02110","Type":"ContainerDied","Data":"57540015739a89767677e907991d4a3c8a14177432a0bb5bfb86037ad6641ce7"} Oct 31 04:14:50.169032 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:50.169021 4376 scope.go:122] "RemoveContainer" containerID="f5f07db5caf75271c375e5a5bdc31a8d79f5368cb9937d70ea5f0dbb73f7a95a" Oct 31 04:14:50.170628 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:50.170596 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-965f61-576d68cfbf-ffkvw" event={"ID":"d2870b2c-9c85-43d1-a726-38edfcf6ae2b","Type":"ContainerStarted","Data":"a8d41f023bf21e1e86500f2f77165da0241e08a284c500d343e05fcfede33883"} Oct 31 04:14:50.170725 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:50.170645 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-965f61-576d68cfbf-ffkvw" event={"ID":"d2870b2c-9c85-43d1-a726-38edfcf6ae2b","Type":"ContainerStarted","Data":"30aee2111452a27e7e496737b3d4a48a2fdfbbf354154af2bdd40416f66ac000"} Oct 31 04:14:50.170725 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:50.170675 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-965f61-576d68cfbf-ffkvw" Oct 31 04:14:50.177955 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:50.177936 4376 scope.go:122] "RemoveContainer" containerID="f5f07db5caf75271c375e5a5bdc31a8d79f5368cb9937d70ea5f0dbb73f7a95a" Oct 31 04:14:50.178231 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:14:50.178211 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5f07db5caf75271c375e5a5bdc31a8d79f5368cb9937d70ea5f0dbb73f7a95a\": container with ID starting with f5f07db5caf75271c375e5a5bdc31a8d79f5368cb9937d70ea5f0dbb73f7a95a not found: ID does not exist" containerID="f5f07db5caf75271c375e5a5bdc31a8d79f5368cb9937d70ea5f0dbb73f7a95a" Oct 31 04:14:50.178272 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:50.178239 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5f07db5caf75271c375e5a5bdc31a8d79f5368cb9937d70ea5f0dbb73f7a95a"} err="failed to get container status \"f5f07db5caf75271c375e5a5bdc31a8d79f5368cb9937d70ea5f0dbb73f7a95a\": rpc error: code = NotFound desc = could not find container \"f5f07db5caf75271c375e5a5bdc31a8d79f5368cb9937d70ea5f0dbb73f7a95a\": container with ID starting with f5f07db5caf75271c375e5a5bdc31a8d79f5368cb9937d70ea5f0dbb73f7a95a not found: ID does not exist" Oct 31 04:14:50.214284 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:50.214233 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-965f61-84cf84d479-gq7q5" podStartSLOduration=2.214221289 podStartE2EDuration="2.214221289s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:14:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:14:50.212680905 +0000 UTC m=+637.927015535" watchObservedRunningTime="2026-10-31 04:14:50.214221289 +0000 UTC m=+637.928555907" Oct 31 04:14:50.254251 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:50.254227 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-44f032ad-666df5c575-kxg5z"] Oct 31 04:14:50.264604 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:50.264572 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-44f032ad-666df5c575-kxg5z"] Oct 31 04:14:50.306574 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:50.306527 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-96rgs" podStartSLOduration=2.306516274 podStartE2EDuration="2.306516274s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:14:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:14:50.305661148 +0000 UTC m=+638.019995784" watchObservedRunningTime="2026-10-31 04:14:50.306516274 +0000 UTC m=+638.020850890" Oct 31 04:14:50.350374 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:50.350309 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-965f61-576d68cfbf-ffkvw" podStartSLOduration=2.350296647 podStartE2EDuration="2.350296647s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:14:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:14:50.349037336 +0000 UTC m=+638.063371971" watchObservedRunningTime="2026-10-31 04:14:50.350296647 +0000 UTC m=+638.064631264" Oct 31 04:14:50.442209 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:50.442130 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8e1248b-7093-4b13-997d-4c3005c02110" path="/var/lib/kubelet/pods/f8e1248b-7093-4b13-997d-4c3005c02110/volumes" Oct 31 04:14:51.027847 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:51.027818 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-96rgs" Oct 31 04:14:51.032658 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:51.032631 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-96rgs" Oct 31 04:14:51.176805 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:51.176775 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-96rgs" Oct 31 04:14:51.177577 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:51.177555 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-96rgs" Oct 31 04:14:51.451177 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:51.451143 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-5wwbn"] Oct 31 04:14:51.451430 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:51.451403 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-5wwbn" podUID="36a9f9d8-95e7-4a96-8cb0-0e02e47a4d9a" containerName="istio-proxy" containerID="cri-o://a4b23dd48e851953e2475c4025c6e7873043ab45b87e6fe414c122b8b2d5f18b" gracePeriod=30 Oct 31 04:14:52.703321 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:52.703288 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-brpvl"] Oct 31 04:14:52.703761 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:52.703745 4376 cpu_manager.go:414] "containerMap: removing container" podUID="f8e1248b-7093-4b13-997d-4c3005c02110" containerName="maas-api" Oct 31 04:14:52.703804 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:52.703761 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8e1248b-7093-4b13-997d-4c3005c02110" containerName="maas-api" Oct 31 04:14:52.703863 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:52.703854 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8e1248b-7093-4b13-997d-4c3005c02110" containerName="maas-api" Oct 31 04:14:52.708453 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:52.708426 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-brpvl" Oct 31 04:14:52.730258 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:52.730230 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-brpvl"] Oct 31 04:14:52.742081 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:52.742048 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-c4006d-openshift-default-dockercfg-h2s7k\"" Oct 31 04:14:52.839911 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:52.839869 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6957320f-9d86-4430-bca7-85db00d7a099-workload-certs\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-brpvl\" (UID: \"6957320f-9d86-4430-bca7-85db00d7a099\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-brpvl" Oct 31 04:14:52.840148 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:52.839974 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6957320f-9d86-4430-bca7-85db00d7a099-workload-socket\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-brpvl\" (UID: \"6957320f-9d86-4430-bca7-85db00d7a099\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-brpvl" Oct 31 04:14:52.840148 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:52.840008 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dq2k2\" (UniqueName: \"kubernetes.io/projected/6957320f-9d86-4430-bca7-85db00d7a099-kube-api-access-dq2k2\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-brpvl\" (UID: \"6957320f-9d86-4430-bca7-85db00d7a099\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-brpvl" Oct 31 04:14:52.840148 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:52.840049 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6957320f-9d86-4430-bca7-85db00d7a099-istio-envoy\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-brpvl\" (UID: \"6957320f-9d86-4430-bca7-85db00d7a099\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-brpvl" Oct 31 04:14:52.840148 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:52.840112 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6957320f-9d86-4430-bca7-85db00d7a099-istio-token\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-brpvl\" (UID: \"6957320f-9d86-4430-bca7-85db00d7a099\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-brpvl" Oct 31 04:14:52.840473 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:52.840181 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6957320f-9d86-4430-bca7-85db00d7a099-credential-socket\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-brpvl\" (UID: \"6957320f-9d86-4430-bca7-85db00d7a099\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-brpvl" Oct 31 04:14:52.840473 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:52.840232 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6957320f-9d86-4430-bca7-85db00d7a099-istiod-ca-cert\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-brpvl\" (UID: \"6957320f-9d86-4430-bca7-85db00d7a099\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-brpvl" Oct 31 04:14:52.840473 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:52.840259 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6957320f-9d86-4430-bca7-85db00d7a099-istio-podinfo\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-brpvl\" (UID: \"6957320f-9d86-4430-bca7-85db00d7a099\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-brpvl" Oct 31 04:14:52.840473 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:52.840290 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6957320f-9d86-4430-bca7-85db00d7a099-istio-data\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-brpvl\" (UID: \"6957320f-9d86-4430-bca7-85db00d7a099\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-brpvl" Oct 31 04:14:52.941829 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:52.941794 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6957320f-9d86-4430-bca7-85db00d7a099-istiod-ca-cert\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-brpvl\" (UID: \"6957320f-9d86-4430-bca7-85db00d7a099\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-brpvl" Oct 31 04:14:52.941829 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:52.941830 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6957320f-9d86-4430-bca7-85db00d7a099-istio-podinfo\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-brpvl\" (UID: \"6957320f-9d86-4430-bca7-85db00d7a099\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-brpvl" Oct 31 04:14:52.942090 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:52.941854 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6957320f-9d86-4430-bca7-85db00d7a099-istio-data\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-brpvl\" (UID: \"6957320f-9d86-4430-bca7-85db00d7a099\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-brpvl" Oct 31 04:14:52.942090 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:52.941902 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6957320f-9d86-4430-bca7-85db00d7a099-workload-certs\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-brpvl\" (UID: \"6957320f-9d86-4430-bca7-85db00d7a099\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-brpvl" Oct 31 04:14:52.942090 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:52.941951 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6957320f-9d86-4430-bca7-85db00d7a099-workload-socket\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-brpvl\" (UID: \"6957320f-9d86-4430-bca7-85db00d7a099\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-brpvl" Oct 31 04:14:52.942221 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:52.942110 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dq2k2\" (UniqueName: \"kubernetes.io/projected/6957320f-9d86-4430-bca7-85db00d7a099-kube-api-access-dq2k2\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-brpvl\" (UID: \"6957320f-9d86-4430-bca7-85db00d7a099\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-brpvl" Oct 31 04:14:52.942221 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:52.942171 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6957320f-9d86-4430-bca7-85db00d7a099-istio-envoy\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-brpvl\" (UID: \"6957320f-9d86-4430-bca7-85db00d7a099\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-brpvl" Oct 31 04:14:52.942221 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:52.942197 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6957320f-9d86-4430-bca7-85db00d7a099-istio-token\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-brpvl\" (UID: \"6957320f-9d86-4430-bca7-85db00d7a099\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-brpvl" Oct 31 04:14:52.942309 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:52.942258 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6957320f-9d86-4430-bca7-85db00d7a099-credential-socket\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-brpvl\" (UID: \"6957320f-9d86-4430-bca7-85db00d7a099\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-brpvl" Oct 31 04:14:52.942725 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:52.942692 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6957320f-9d86-4430-bca7-85db00d7a099-istiod-ca-cert\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-brpvl\" (UID: \"6957320f-9d86-4430-bca7-85db00d7a099\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-brpvl" Oct 31 04:14:52.942887 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:52.942763 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6957320f-9d86-4430-bca7-85db00d7a099-workload-socket\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-brpvl\" (UID: \"6957320f-9d86-4430-bca7-85db00d7a099\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-brpvl" Oct 31 04:14:52.942887 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:52.942806 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6957320f-9d86-4430-bca7-85db00d7a099-workload-certs\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-brpvl\" (UID: \"6957320f-9d86-4430-bca7-85db00d7a099\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-brpvl" Oct 31 04:14:52.942887 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:52.942865 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6957320f-9d86-4430-bca7-85db00d7a099-istio-data\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-brpvl\" (UID: \"6957320f-9d86-4430-bca7-85db00d7a099\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-brpvl" Oct 31 04:14:52.943181 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:52.943141 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6957320f-9d86-4430-bca7-85db00d7a099-credential-socket\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-brpvl\" (UID: \"6957320f-9d86-4430-bca7-85db00d7a099\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-brpvl" Oct 31 04:14:52.944822 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:52.944793 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6957320f-9d86-4430-bca7-85db00d7a099-istio-envoy\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-brpvl\" (UID: \"6957320f-9d86-4430-bca7-85db00d7a099\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-brpvl" Oct 31 04:14:52.944928 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:52.944878 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6957320f-9d86-4430-bca7-85db00d7a099-istio-podinfo\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-brpvl\" (UID: \"6957320f-9d86-4430-bca7-85db00d7a099\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-brpvl" Oct 31 04:14:52.973026 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:52.972957 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dq2k2\" (UniqueName: \"kubernetes.io/projected/6957320f-9d86-4430-bca7-85db00d7a099-kube-api-access-dq2k2\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-brpvl\" (UID: \"6957320f-9d86-4430-bca7-85db00d7a099\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-brpvl" Oct 31 04:14:52.974473 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:52.974455 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6957320f-9d86-4430-bca7-85db00d7a099-istio-token\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-brpvl\" (UID: \"6957320f-9d86-4430-bca7-85db00d7a099\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-brpvl" Oct 31 04:14:53.023479 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:53.023440 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-brpvl" Oct 31 04:14:53.133755 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:53.122971 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-09a20a-b4f7d6d49-2lp58" Oct 31 04:14:53.133755 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:53.125719 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-09a20a-5bf985cbc9-scqzr" Oct 31 04:14:53.373345 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:14:53.373305 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6957320f_9d86_4430_bca7_85db00d7a099.slice/crio-8e850ede4e9ea44b5e1aae8f38af2dfe694dfc8e51562cf708e65ad31b6103db WatchSource:0}: Error finding container 8e850ede4e9ea44b5e1aae8f38af2dfe694dfc8e51562cf708e65ad31b6103db: Status 404 returned error can't find the container with id 8e850ede4e9ea44b5e1aae8f38af2dfe694dfc8e51562cf708e65ad31b6103db Oct 31 04:14:53.374649 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:53.374627 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-brpvl"] Oct 31 04:14:53.376229 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:53.376200 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:14:53.376283 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:53.376266 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:14:53.376317 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:53.376294 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:14:54.197039 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:54.197002 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-brpvl" event={"ID":"6957320f-9d86-4430-bca7-85db00d7a099","Type":"ContainerStarted","Data":"6f62c4f1a8beb2890f61a48c09ca5cd6a4e392a731114cdf4c1babee7bfda920"} Oct 31 04:14:54.197493 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:54.197088 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-brpvl" event={"ID":"6957320f-9d86-4430-bca7-85db00d7a099","Type":"ContainerStarted","Data":"8e850ede4e9ea44b5e1aae8f38af2dfe694dfc8e51562cf708e65ad31b6103db"} Oct 31 04:14:54.276204 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:54.276134 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-brpvl" podStartSLOduration=2.276120636 podStartE2EDuration="2.276120636s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:14:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:14:54.260323866 +0000 UTC m=+641.974658503" watchObservedRunningTime="2026-10-31 04:14:54.276120636 +0000 UTC m=+641.990455313" Oct 31 04:14:54.462599 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:54.462495 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 04:14:55.024268 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:55.024230 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-brpvl" Oct 31 04:14:55.029422 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:55.029399 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-brpvl" Oct 31 04:14:55.202812 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:55.202783 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-brpvl" Oct 31 04:14:55.204740 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:55.204702 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-brpvl" Oct 31 04:14:56.712952 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:56.712926 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-5wwbn" Oct 31 04:14:56.800681 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:56.800591 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36a9f9d8-95e7-4a96-8cb0-0e02e47a4d9a-kube-api-access-7vrm2\" (UniqueName: \"kubernetes.io/projected/36a9f9d8-95e7-4a96-8cb0-0e02e47a4d9a-kube-api-access-7vrm2\") pod \"36a9f9d8-95e7-4a96-8cb0-0e02e47a4d9a\" (UID: \"36a9f9d8-95e7-4a96-8cb0-0e02e47a4d9a\") " Oct 31 04:14:56.800681 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:56.800632 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36a9f9d8-95e7-4a96-8cb0-0e02e47a4d9a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36a9f9d8-95e7-4a96-8cb0-0e02e47a4d9a-credential-socket\") pod \"36a9f9d8-95e7-4a96-8cb0-0e02e47a4d9a\" (UID: \"36a9f9d8-95e7-4a96-8cb0-0e02e47a4d9a\") " Oct 31 04:14:56.800681 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:56.800654 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/36a9f9d8-95e7-4a96-8cb0-0e02e47a4d9a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36a9f9d8-95e7-4a96-8cb0-0e02e47a4d9a-istio-podinfo\") pod \"36a9f9d8-95e7-4a96-8cb0-0e02e47a4d9a\" (UID: \"36a9f9d8-95e7-4a96-8cb0-0e02e47a4d9a\") " Oct 31 04:14:56.800974 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:56.800714 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36a9f9d8-95e7-4a96-8cb0-0e02e47a4d9a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36a9f9d8-95e7-4a96-8cb0-0e02e47a4d9a-istiod-ca-cert\") pod \"36a9f9d8-95e7-4a96-8cb0-0e02e47a4d9a\" (UID: \"36a9f9d8-95e7-4a96-8cb0-0e02e47a4d9a\") " Oct 31 04:14:56.800974 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:56.800650 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-439989-5d779f9865-lc48d"] Oct 31 04:14:56.800974 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:56.800783 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36a9f9d8-95e7-4a96-8cb0-0e02e47a4d9a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36a9f9d8-95e7-4a96-8cb0-0e02e47a4d9a-istio-data\") pod \"36a9f9d8-95e7-4a96-8cb0-0e02e47a4d9a\" (UID: \"36a9f9d8-95e7-4a96-8cb0-0e02e47a4d9a\") " Oct 31 04:14:56.800974 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:56.800846 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36a9f9d8-95e7-4a96-8cb0-0e02e47a4d9a-istio-token\" (UniqueName: \"kubernetes.io/projected/36a9f9d8-95e7-4a96-8cb0-0e02e47a4d9a-istio-token\") pod \"36a9f9d8-95e7-4a96-8cb0-0e02e47a4d9a\" (UID: \"36a9f9d8-95e7-4a96-8cb0-0e02e47a4d9a\") " Oct 31 04:14:56.800974 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:56.800876 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36a9f9d8-95e7-4a96-8cb0-0e02e47a4d9a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36a9f9d8-95e7-4a96-8cb0-0e02e47a4d9a-workload-socket\") pod \"36a9f9d8-95e7-4a96-8cb0-0e02e47a4d9a\" (UID: \"36a9f9d8-95e7-4a96-8cb0-0e02e47a4d9a\") " Oct 31 04:14:56.800974 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:56.800909 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36a9f9d8-95e7-4a96-8cb0-0e02e47a4d9a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36a9f9d8-95e7-4a96-8cb0-0e02e47a4d9a-istio-envoy\") pod \"36a9f9d8-95e7-4a96-8cb0-0e02e47a4d9a\" (UID: \"36a9f9d8-95e7-4a96-8cb0-0e02e47a4d9a\") " Oct 31 04:14:56.800974 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:56.800938 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36a9f9d8-95e7-4a96-8cb0-0e02e47a4d9a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36a9f9d8-95e7-4a96-8cb0-0e02e47a4d9a-workload-certs\") pod \"36a9f9d8-95e7-4a96-8cb0-0e02e47a4d9a\" (UID: \"36a9f9d8-95e7-4a96-8cb0-0e02e47a4d9a\") " Oct 31 04:14:56.800974 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:56.800953 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36a9f9d8-95e7-4a96-8cb0-0e02e47a4d9a-credential-socket" pod "36a9f9d8-95e7-4a96-8cb0-0e02e47a4d9a" (UID: "36a9f9d8-95e7-4a96-8cb0-0e02e47a4d9a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:14:56.801400 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:56.801173 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36a9f9d8-95e7-4a96-8cb0-0e02e47a4d9a-workload-socket" pod "36a9f9d8-95e7-4a96-8cb0-0e02e47a4d9a" (UID: "36a9f9d8-95e7-4a96-8cb0-0e02e47a4d9a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:14:56.801400 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:56.801181 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36a9f9d8-95e7-4a96-8cb0-0e02e47a4d9a-istiod-ca-cert" pod "36a9f9d8-95e7-4a96-8cb0-0e02e47a4d9a" (UID: "36a9f9d8-95e7-4a96-8cb0-0e02e47a4d9a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:14:56.801400 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:56.801355 4376 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36a9f9d8-95e7-4a96-8cb0-0e02e47a4d9a-workload-socket\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:14:56.801400 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:56.801371 4376 cpu_manager.go:414] "containerMap: removing container" podUID="36a9f9d8-95e7-4a96-8cb0-0e02e47a4d9a" containerName="istio-proxy" Oct 31 04:14:56.801400 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:56.801376 4376 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36a9f9d8-95e7-4a96-8cb0-0e02e47a4d9a-credential-socket\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:14:56.801400 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:56.801385 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36a9f9d8-95e7-4a96-8cb0-0e02e47a4d9a" containerName="istio-proxy" Oct 31 04:14:56.801400 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:56.801391 4376 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36a9f9d8-95e7-4a96-8cb0-0e02e47a4d9a-istiod-ca-cert\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:14:56.801400 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:56.801393 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36a9f9d8-95e7-4a96-8cb0-0e02e47a4d9a-istio-data" pod "36a9f9d8-95e7-4a96-8cb0-0e02e47a4d9a" (UID: "36a9f9d8-95e7-4a96-8cb0-0e02e47a4d9a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:14:56.802522 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:56.801440 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36a9f9d8-95e7-4a96-8cb0-0e02e47a4d9a-workload-certs" pod "36a9f9d8-95e7-4a96-8cb0-0e02e47a4d9a" (UID: "36a9f9d8-95e7-4a96-8cb0-0e02e47a4d9a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:14:56.802522 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:56.801510 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="36a9f9d8-95e7-4a96-8cb0-0e02e47a4d9a" containerName="istio-proxy" Oct 31 04:14:56.805639 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:56.805608 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/36a9f9d8-95e7-4a96-8cb0-0e02e47a4d9a-istio-podinfo" pod "36a9f9d8-95e7-4a96-8cb0-0e02e47a4d9a" (UID: "36a9f9d8-95e7-4a96-8cb0-0e02e47a4d9a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 04:14:56.805958 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:56.805934 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-439989-5d779f9865-lc48d" Oct 31 04:14:56.806163 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:56.806140 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36a9f9d8-95e7-4a96-8cb0-0e02e47a4d9a-istio-token" pod "36a9f9d8-95e7-4a96-8cb0-0e02e47a4d9a" (UID: "36a9f9d8-95e7-4a96-8cb0-0e02e47a4d9a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:14:56.806669 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:56.806648 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36a9f9d8-95e7-4a96-8cb0-0e02e47a4d9a-kube-api-access-7vrm2" pod "36a9f9d8-95e7-4a96-8cb0-0e02e47a4d9a" (UID: "36a9f9d8-95e7-4a96-8cb0-0e02e47a4d9a"). InnerVolumeSpecName "kube-api-access-7vrm2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:14:56.807845 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:56.807821 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36a9f9d8-95e7-4a96-8cb0-0e02e47a4d9a-istio-envoy" pod "36a9f9d8-95e7-4a96-8cb0-0e02e47a4d9a" (UID: "36a9f9d8-95e7-4a96-8cb0-0e02e47a4d9a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:14:56.821177 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:56.821147 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-439989-5d779f9865-lc48d"] Oct 31 04:14:56.852240 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:56.852203 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-439989\"" Oct 31 04:14:56.852429 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:56.852347 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-439989-dockercfg-hwjxl\"" Oct 31 04:14:56.902391 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:56.902321 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a252fc6-14bf-4524-907a-f8ee294acda5-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-439989-5d779f9865-lc48d\" (UID: \"0a252fc6-14bf-4524-907a-f8ee294acda5\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-439989-5d779f9865-lc48d" Oct 31 04:14:56.902708 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:56.902576 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzl69\" (UniqueName: \"kubernetes.io/projected/0a252fc6-14bf-4524-907a-f8ee294acda5-kube-api-access-pzl69\") pod \"payload-processing-e2e-worker-w1-439989-5d779f9865-lc48d\" (UID: \"0a252fc6-14bf-4524-907a-f8ee294acda5\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-439989-5d779f9865-lc48d" Oct 31 04:14:56.902794 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:56.902719 4376 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36a9f9d8-95e7-4a96-8cb0-0e02e47a4d9a-istio-token\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:14:56.902794 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:56.902738 4376 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36a9f9d8-95e7-4a96-8cb0-0e02e47a4d9a-istio-envoy\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:14:56.902794 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:56.902752 4376 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36a9f9d8-95e7-4a96-8cb0-0e02e47a4d9a-workload-certs\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:14:56.902794 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:56.902766 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7vrm2\" (UniqueName: \"kubernetes.io/projected/36a9f9d8-95e7-4a96-8cb0-0e02e47a4d9a-kube-api-access-7vrm2\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:14:56.902794 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:56.902780 4376 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36a9f9d8-95e7-4a96-8cb0-0e02e47a4d9a-istio-podinfo\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:14:56.902951 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:56.902794 4376 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36a9f9d8-95e7-4a96-8cb0-0e02e47a4d9a-istio-data\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:14:57.004182 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:57.004140 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzl69\" (UniqueName: \"kubernetes.io/projected/0a252fc6-14bf-4524-907a-f8ee294acda5-kube-api-access-pzl69\") pod \"payload-processing-e2e-worker-w1-439989-5d779f9865-lc48d\" (UID: \"0a252fc6-14bf-4524-907a-f8ee294acda5\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-439989-5d779f9865-lc48d" Oct 31 04:14:57.004387 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:57.004218 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a252fc6-14bf-4524-907a-f8ee294acda5-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-439989-5d779f9865-lc48d\" (UID: \"0a252fc6-14bf-4524-907a-f8ee294acda5\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-439989-5d779f9865-lc48d" Oct 31 04:14:57.005072 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:57.005050 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a252fc6-14bf-4524-907a-f8ee294acda5-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-439989-5d779f9865-lc48d\" (UID: \"0a252fc6-14bf-4524-907a-f8ee294acda5\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-439989-5d779f9865-lc48d" Oct 31 04:14:57.032294 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:57.032261 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzl69\" (UniqueName: \"kubernetes.io/projected/0a252fc6-14bf-4524-907a-f8ee294acda5-kube-api-access-pzl69\") pod \"payload-processing-e2e-worker-w1-439989-5d779f9865-lc48d\" (UID: \"0a252fc6-14bf-4524-907a-f8ee294acda5\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-439989-5d779f9865-lc48d" Oct 31 04:14:57.137396 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:57.137352 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-439989-5d779f9865-lc48d" Oct 31 04:14:57.215643 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:57.215602 4376 generic.go:356] "Generic (PLEG): container finished" podID="36a9f9d8-95e7-4a96-8cb0-0e02e47a4d9a" containerID="a4b23dd48e851953e2475c4025c6e7873043ab45b87e6fe414c122b8b2d5f18b" exitCode=0 Oct 31 04:14:57.215809 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:57.215666 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-5wwbn" Oct 31 04:14:57.215809 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:57.215692 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-5wwbn" event={"ID":"36a9f9d8-95e7-4a96-8cb0-0e02e47a4d9a","Type":"ContainerDied","Data":"a4b23dd48e851953e2475c4025c6e7873043ab45b87e6fe414c122b8b2d5f18b"} Oct 31 04:14:57.215809 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:57.215777 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-5wwbn" event={"ID":"36a9f9d8-95e7-4a96-8cb0-0e02e47a4d9a","Type":"ContainerDied","Data":"fd8d7330b5a2b0fd55e557f77dd949137108b6e973dcec1644ff24dbf2ee2e95"} Oct 31 04:14:57.215933 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:57.215815 4376 scope.go:122] "RemoveContainer" containerID="a4b23dd48e851953e2475c4025c6e7873043ab45b87e6fe414c122b8b2d5f18b" Oct 31 04:14:57.229501 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:57.229483 4376 scope.go:122] "RemoveContainer" containerID="a4b23dd48e851953e2475c4025c6e7873043ab45b87e6fe414c122b8b2d5f18b" Oct 31 04:14:57.229825 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:14:57.229809 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4b23dd48e851953e2475c4025c6e7873043ab45b87e6fe414c122b8b2d5f18b\": container with ID starting with a4b23dd48e851953e2475c4025c6e7873043ab45b87e6fe414c122b8b2d5f18b not found: ID does not exist" containerID="a4b23dd48e851953e2475c4025c6e7873043ab45b87e6fe414c122b8b2d5f18b" Oct 31 04:14:57.229881 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:57.229837 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4b23dd48e851953e2475c4025c6e7873043ab45b87e6fe414c122b8b2d5f18b"} err="failed to get container status \"a4b23dd48e851953e2475c4025c6e7873043ab45b87e6fe414c122b8b2d5f18b\": rpc error: code = NotFound desc = could not find container \"a4b23dd48e851953e2475c4025c6e7873043ab45b87e6fe414c122b8b2d5f18b\": container with ID starting with a4b23dd48e851953e2475c4025c6e7873043ab45b87e6fe414c122b8b2d5f18b not found: ID does not exist" Oct 31 04:14:57.276517 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:57.276483 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-5wwbn"] Oct 31 04:14:57.278615 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:57.278592 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-5wwbn"] Oct 31 04:14:57.433692 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:57.433665 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-439989-5d779f9865-lc48d"] Oct 31 04:14:57.433872 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:14:57.433849 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a252fc6_14bf_4524_907a_f8ee294acda5.slice/crio-309925dfc201351fd53a96db994451a368e43d2fbac67a41944c12ed9965413f WatchSource:0}: Error finding container 309925dfc201351fd53a96db994451a368e43d2fbac67a41944c12ed9965413f: Status 404 returned error can't find the container with id 309925dfc201351fd53a96db994451a368e43d2fbac67a41944c12ed9965413f Oct 31 04:14:58.223723 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:58.223684 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-439989-5d779f9865-lc48d" event={"ID":"0a252fc6-14bf-4524-907a-f8ee294acda5","Type":"ContainerStarted","Data":"f676360a316ac36141aebd4e14dc1b541d41b2d44de27e267b4315a9b52ac0d7"} Oct 31 04:14:58.224140 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:58.223774 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-439989-5d779f9865-lc48d" event={"ID":"0a252fc6-14bf-4524-907a-f8ee294acda5","Type":"ContainerStarted","Data":"309925dfc201351fd53a96db994451a368e43d2fbac67a41944c12ed9965413f"} Oct 31 04:14:58.224140 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:58.223830 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-439989-5d779f9865-lc48d" Oct 31 04:14:58.269602 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:58.269543 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-439989-5d779f9865-lc48d" podStartSLOduration=2.269530062 podStartE2EDuration="2.269530062s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:14:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:14:58.268971472 +0000 UTC m=+645.983306089" watchObservedRunningTime="2026-10-31 04:14:58.269530062 +0000 UTC m=+645.983864701" Oct 31 04:14:58.441774 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:14:58.441744 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36a9f9d8-95e7-4a96-8cb0-0e02e47a4d9a" path="/var/lib/kubelet/pods/36a9f9d8-95e7-4a96-8cb0-0e02e47a4d9a/volumes" Oct 31 04:15:00.171910 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:00.171880 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-965f61-29890335-df4dh"] Oct 31 04:15:00.175490 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:00.175452 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-965f61-29890335-df4dh" Oct 31 04:15:00.192530 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:00.192505 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-965f61-29890335-df4dh"] Oct 31 04:15:00.231859 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:00.231806 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzb8r\"" Oct 31 04:15:00.345439 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:00.345405 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/822172d2-2688-49fe-9a6b-628f32ea8b98-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-965f61-29890335-df4dh\" (UID: \"822172d2-2688-49fe-9a6b-628f32ea8b98\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-965f61-29890335-df4dh" Oct 31 04:15:00.345620 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:00.345522 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6n6rb\" (UniqueName: \"kubernetes.io/projected/822172d2-2688-49fe-9a6b-628f32ea8b98-kube-api-access-6n6rb\") pod \"maas-api-key-cleanup-e2e-worker-w0-965f61-29890335-df4dh\" (UID: \"822172d2-2688-49fe-9a6b-628f32ea8b98\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-965f61-29890335-df4dh" Oct 31 04:15:00.446655 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:00.446575 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/822172d2-2688-49fe-9a6b-628f32ea8b98-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-965f61-29890335-df4dh\" (UID: \"822172d2-2688-49fe-9a6b-628f32ea8b98\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-965f61-29890335-df4dh" Oct 31 04:15:00.446655 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:00.446641 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6n6rb\" (UniqueName: \"kubernetes.io/projected/822172d2-2688-49fe-9a6b-628f32ea8b98-kube-api-access-6n6rb\") pod \"maas-api-key-cleanup-e2e-worker-w0-965f61-29890335-df4dh\" (UID: \"822172d2-2688-49fe-9a6b-628f32ea8b98\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-965f61-29890335-df4dh" Oct 31 04:15:00.447212 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:00.447192 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/822172d2-2688-49fe-9a6b-628f32ea8b98-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-965f61-29890335-df4dh\" (UID: \"822172d2-2688-49fe-9a6b-628f32ea8b98\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-965f61-29890335-df4dh" Oct 31 04:15:00.471867 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:00.471834 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6n6rb\" (UniqueName: \"kubernetes.io/projected/822172d2-2688-49fe-9a6b-628f32ea8b98-kube-api-access-6n6rb\") pod \"maas-api-key-cleanup-e2e-worker-w0-965f61-29890335-df4dh\" (UID: \"822172d2-2688-49fe-9a6b-628f32ea8b98\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-965f61-29890335-df4dh" Oct 31 04:15:00.489903 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:00.489875 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-965f61-29890335-df4dh" Oct 31 04:15:00.790895 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:00.790861 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-965f61-29890335-df4dh"] Oct 31 04:15:00.799290 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:15:00.799250 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod822172d2_2688_49fe_9a6b_628f32ea8b98.slice/crio-a4ba2117574b84bea5da81ad274901294b6aeb2ce62e7b900c8f3cadc9e0773b WatchSource:0}: Error finding container a4ba2117574b84bea5da81ad274901294b6aeb2ce62e7b900c8f3cadc9e0773b: Status 404 returned error can't find the container with id a4ba2117574b84bea5da81ad274901294b6aeb2ce62e7b900c8f3cadc9e0773b Oct 31 04:15:00.803050 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:00.802049 4376 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 04:15:01.179300 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:01.179275 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-965f61-576d68cfbf-ffkvw" Oct 31 04:15:01.179701 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:01.179373 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-965f61-84cf84d479-gq7q5" Oct 31 04:15:01.272688 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:01.272647 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-965f61-29890335-df4dh" event={"ID":"822172d2-2688-49fe-9a6b-628f32ea8b98","Type":"ContainerStarted","Data":"a4ba2117574b84bea5da81ad274901294b6aeb2ce62e7b900c8f3cadc9e0773b"} Oct 31 04:15:02.368155 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:02.368116 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 04:15:04.297687 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:04.297647 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-965f61-29890335-df4dh" event={"ID":"822172d2-2688-49fe-9a6b-628f32ea8b98","Type":"ContainerStarted","Data":"c98c0fe4c3bacda4853d989ce4d7b9252209aff830f0054b6a6344d5c3d3626b"} Oct 31 04:15:04.362354 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:04.359245 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-965f61-29890335-df4dh" podStartSLOduration=1.332792177 podStartE2EDuration="4.359226591s" totalImagesPullingTime="3.026434414s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:15:04.355960855 +0000 UTC m=+652.070295474" watchObservedRunningTime="2026-10-31 04:15:04.359226591 +0000 UTC m=+652.073561210" Oct 31 04:15:05.303993 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:05.303960 4376 generic.go:356] "Generic (PLEG): container finished" podID="822172d2-2688-49fe-9a6b-628f32ea8b98" containerID="c98c0fe4c3bacda4853d989ce4d7b9252209aff830f0054b6a6344d5c3d3626b" exitCode=60 Oct 31 04:15:05.304364 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:05.304040 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-965f61-29890335-df4dh" event={"ID":"822172d2-2688-49fe-9a6b-628f32ea8b98","Type":"ContainerDied","Data":"c98c0fe4c3bacda4853d989ce4d7b9252209aff830f0054b6a6344d5c3d3626b"} Oct 31 04:15:05.304364 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:05.304257 4376 scope.go:122] "RemoveContainer" containerID="c98c0fe4c3bacda4853d989ce4d7b9252209aff830f0054b6a6344d5c3d3626b" Oct 31 04:15:06.190767 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:06.190734 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-c4006d-b7ccb647b-v7tsw"] Oct 31 04:15:06.194288 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:06.194263 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c4006d-b7ccb647b-v7tsw" Oct 31 04:15:06.217302 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:06.217274 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-c4006d-b7ccb647b-v7tsw"] Oct 31 04:15:06.223927 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:06.223772 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bp65b\" (UniqueName: \"kubernetes.io/projected/ba6f1fd8-1985-4ea5-9097-36263d48aa13-kube-api-access-bp65b\") pod \"maas-api-e2e-worker-w4-c4006d-b7ccb647b-v7tsw\" (UID: \"ba6f1fd8-1985-4ea5-9097-36263d48aa13\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c4006d-b7ccb647b-v7tsw" Oct 31 04:15:06.223927 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:06.223820 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ba6f1fd8-1985-4ea5-9097-36263d48aa13-maas-api-tls\") pod \"maas-api-e2e-worker-w4-c4006d-b7ccb647b-v7tsw\" (UID: \"ba6f1fd8-1985-4ea5-9097-36263d48aa13\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c4006d-b7ccb647b-v7tsw" Oct 31 04:15:06.223927 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:06.223906 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba6f1fd8-1985-4ea5-9097-36263d48aa13-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-c4006d-b7ccb647b-v7tsw\" (UID: \"ba6f1fd8-1985-4ea5-9097-36263d48aa13\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c4006d-b7ccb647b-v7tsw" Oct 31 04:15:06.224146 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:06.224047 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba6f1fd8-1985-4ea5-9097-36263d48aa13-metrics-certs\") pod \"maas-api-e2e-worker-w4-c4006d-b7ccb647b-v7tsw\" (UID: \"ba6f1fd8-1985-4ea5-9097-36263d48aa13\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c4006d-b7ccb647b-v7tsw" Oct 31 04:15:06.227573 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:06.227554 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-c4006d-serving-cert\"" Oct 31 04:15:06.311348 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:06.311293 4376 generic.go:356] "Generic (PLEG): container finished" podID="822172d2-2688-49fe-9a6b-628f32ea8b98" containerID="7e7f1c3ef691dff0910bf78ae8554b663a547696b2d8cb1a5eb19ff441457e7f" exitCode=60 Oct 31 04:15:06.311769 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:06.311373 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-965f61-29890335-df4dh" event={"ID":"822172d2-2688-49fe-9a6b-628f32ea8b98","Type":"ContainerDied","Data":"7e7f1c3ef691dff0910bf78ae8554b663a547696b2d8cb1a5eb19ff441457e7f"} Oct 31 04:15:06.311769 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:06.311436 4376 scope.go:122] "RemoveContainer" containerID="c98c0fe4c3bacda4853d989ce4d7b9252209aff830f0054b6a6344d5c3d3626b" Oct 31 04:15:06.311769 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:06.311723 4376 scope.go:122] "RemoveContainer" containerID="7e7f1c3ef691dff0910bf78ae8554b663a547696b2d8cb1a5eb19ff441457e7f" Oct 31 04:15:06.312041 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:15:06.312016 4376 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-965f61-29890335-df4dh_odh-ai-gateway-infra(822172d2-2688-49fe-9a6b-628f32ea8b98)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-965f61-29890335-df4dh" podUID="822172d2-2688-49fe-9a6b-628f32ea8b98" Oct 31 04:15:06.325082 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:06.325062 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba6f1fd8-1985-4ea5-9097-36263d48aa13-metrics-certs\") pod \"maas-api-e2e-worker-w4-c4006d-b7ccb647b-v7tsw\" (UID: \"ba6f1fd8-1985-4ea5-9097-36263d48aa13\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c4006d-b7ccb647b-v7tsw" Oct 31 04:15:06.325196 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:06.325106 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bp65b\" (UniqueName: \"kubernetes.io/projected/ba6f1fd8-1985-4ea5-9097-36263d48aa13-kube-api-access-bp65b\") pod \"maas-api-e2e-worker-w4-c4006d-b7ccb647b-v7tsw\" (UID: \"ba6f1fd8-1985-4ea5-9097-36263d48aa13\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c4006d-b7ccb647b-v7tsw" Oct 31 04:15:06.325196 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:06.325133 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ba6f1fd8-1985-4ea5-9097-36263d48aa13-maas-api-tls\") pod \"maas-api-e2e-worker-w4-c4006d-b7ccb647b-v7tsw\" (UID: \"ba6f1fd8-1985-4ea5-9097-36263d48aa13\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c4006d-b7ccb647b-v7tsw" Oct 31 04:15:06.325196 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:06.325178 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba6f1fd8-1985-4ea5-9097-36263d48aa13-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-c4006d-b7ccb647b-v7tsw\" (UID: \"ba6f1fd8-1985-4ea5-9097-36263d48aa13\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c4006d-b7ccb647b-v7tsw" Oct 31 04:15:06.325379 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:15:06.325274 4376 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w4-c4006d-serving-cert: secret "maas-api-e2e-worker-w4-c4006d-serving-cert" not found Oct 31 04:15:06.325495 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:15:06.325470 4376 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba6f1fd8-1985-4ea5-9097-36263d48aa13-maas-api-tls podName:ba6f1fd8-1985-4ea5-9097-36263d48aa13 nodeName:}" failed. No retries permitted until 2026-10-31 04:15:06.825314859 +0000 UTC m=+654.539649480 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ba6f1fd8-1985-4ea5-9097-36263d48aa13-maas-api-tls") pod "maas-api-e2e-worker-w4-c4006d-b7ccb647b-v7tsw" (UID: "ba6f1fd8-1985-4ea5-9097-36263d48aa13") : secret "maas-api-e2e-worker-w4-c4006d-serving-cert" not found Oct 31 04:15:06.326009 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:06.325980 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba6f1fd8-1985-4ea5-9097-36263d48aa13-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-c4006d-b7ccb647b-v7tsw\" (UID: \"ba6f1fd8-1985-4ea5-9097-36263d48aa13\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c4006d-b7ccb647b-v7tsw" Oct 31 04:15:06.327620 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:06.327595 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba6f1fd8-1985-4ea5-9097-36263d48aa13-metrics-certs\") pod \"maas-api-e2e-worker-w4-c4006d-b7ccb647b-v7tsw\" (UID: \"ba6f1fd8-1985-4ea5-9097-36263d48aa13\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c4006d-b7ccb647b-v7tsw" Oct 31 04:15:06.350608 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:06.350575 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bp65b\" (UniqueName: \"kubernetes.io/projected/ba6f1fd8-1985-4ea5-9097-36263d48aa13-kube-api-access-bp65b\") pod \"maas-api-e2e-worker-w4-c4006d-b7ccb647b-v7tsw\" (UID: \"ba6f1fd8-1985-4ea5-9097-36263d48aa13\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c4006d-b7ccb647b-v7tsw" Oct 31 04:15:06.833710 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:06.833667 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ba6f1fd8-1985-4ea5-9097-36263d48aa13-maas-api-tls\") pod \"maas-api-e2e-worker-w4-c4006d-b7ccb647b-v7tsw\" (UID: \"ba6f1fd8-1985-4ea5-9097-36263d48aa13\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c4006d-b7ccb647b-v7tsw" Oct 31 04:15:06.836855 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:06.836827 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ba6f1fd8-1985-4ea5-9097-36263d48aa13-maas-api-tls\") pod \"maas-api-e2e-worker-w4-c4006d-b7ccb647b-v7tsw\" (UID: \"ba6f1fd8-1985-4ea5-9097-36263d48aa13\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c4006d-b7ccb647b-v7tsw" Oct 31 04:15:07.107140 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:07.107045 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c4006d-b7ccb647b-v7tsw" Oct 31 04:15:07.318153 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:07.318120 4376 scope.go:122] "RemoveContainer" containerID="7e7f1c3ef691dff0910bf78ae8554b663a547696b2d8cb1a5eb19ff441457e7f" Oct 31 04:15:07.318587 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:15:07.318380 4376 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-965f61-29890335-df4dh_odh-ai-gateway-infra(822172d2-2688-49fe-9a6b-628f32ea8b98)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-965f61-29890335-df4dh" podUID="822172d2-2688-49fe-9a6b-628f32ea8b98" Oct 31 04:15:07.436808 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:07.436777 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-c4006d-b7ccb647b-v7tsw"] Oct 31 04:15:08.324056 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:08.324018 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c4006d-b7ccb647b-v7tsw" event={"ID":"ba6f1fd8-1985-4ea5-9097-36263d48aa13","Type":"ContainerStarted","Data":"4e6065771759e51f9b8e0f1cb6ce9ff5710a05f7842243ffa01a13f582b4d6c3"} Oct 31 04:15:08.324502 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:08.324081 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c4006d-b7ccb647b-v7tsw" event={"ID":"ba6f1fd8-1985-4ea5-9097-36263d48aa13","Type":"ContainerStarted","Data":"16850602a80a7fb971fc33fc22e09155f9f1d7a274bb091db47dce0362a26252"} Oct 31 04:15:08.324502 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:08.324218 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c4006d-b7ccb647b-v7tsw" Oct 31 04:15:08.377605 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:08.377544 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c4006d-b7ccb647b-v7tsw" podStartSLOduration=2.04301271 podStartE2EDuration="2.377526842s" totalImagesPullingTime="334.514132ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:15:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:15:08.376082812 +0000 UTC m=+656.090417430" watchObservedRunningTime="2026-10-31 04:15:08.377526842 +0000 UTC m=+656.091861462" Oct 31 04:15:09.230958 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:09.230929 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-439989-5d779f9865-lc48d" Oct 31 04:15:10.998822 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:15:10.998785 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 04:15:11.364082 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:11.363996 4376 generic.go:356] "Generic (PLEG): container finished" podID="5d825a50-865d-46c6-b88d-73f7170963c5" containerID="ba4b20e19ffef90a458a50db30fa6d872dfa311726d28f320c05c268046f4bac" exitCode=0 Oct 31 04:15:11.364223 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:11.364081 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-zmth9" event={"ID":"5d825a50-865d-46c6-b88d-73f7170963c5","Type":"ContainerDied","Data":"ba4b20e19ffef90a458a50db30fa6d872dfa311726d28f320c05c268046f4bac"} Oct 31 04:15:11.364540 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:11.364526 4376 scope.go:122] "RemoveContainer" containerID="ba4b20e19ffef90a458a50db30fa6d872dfa311726d28f320c05c268046f4bac" Oct 31 04:15:11.365770 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:11.365740 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:15:11.365859 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:11.365805 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:15:11.365859 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:11.365837 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:15:11.989319 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:11.989289 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 04:15:12.385189 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:12.385153 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-zmth9" event={"ID":"5d825a50-865d-46c6-b88d-73f7170963c5","Type":"ContainerStarted","Data":"ef3065d9e6b828d32123237c43ff3949d01ad4aa59f2bd3080d3ca6fffe8548f"} Oct 31 04:15:13.088352 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:13.074687 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-zmth9" Oct 31 04:15:13.088352 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:13.076396 4376 patch_prober.go:29] interesting pod/e2e-worker-w0-965f61-openshift-default-55854964c-zmth9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Oct 31 04:15:13.088352 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:13.076447 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-zmth9" podUID="5d825a50-865d-46c6-b88d-73f7170963c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Oct 31 04:15:14.099351 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:14.088726 4376 patch_prober.go:29] interesting pod/e2e-worker-w0-965f61-openshift-default-55854964c-zmth9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Oct 31 04:15:14.099351 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:14.088807 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-zmth9" podUID="5d825a50-865d-46c6-b88d-73f7170963c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Oct 31 04:15:14.347704 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:14.347671 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c4006d-b7ccb647b-v7tsw" Oct 31 04:15:15.076353 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:15.075708 4376 patch_prober.go:29] interesting pod/e2e-worker-w0-965f61-openshift-default-55854964c-zmth9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Oct 31 04:15:15.076353 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:15.075773 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-zmth9" podUID="5d825a50-865d-46c6-b88d-73f7170963c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Oct 31 04:15:15.124356 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:15.124309 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-zmth9" Oct 31 04:15:16.077353 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:16.075781 4376 patch_prober.go:29] interesting pod/e2e-worker-w0-965f61-openshift-default-55854964c-zmth9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Oct 31 04:15:16.077353 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:16.075847 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-zmth9" podUID="5d825a50-865d-46c6-b88d-73f7170963c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Oct 31 04:15:17.080353 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:17.075709 4376 patch_prober.go:29] interesting pod/e2e-worker-w0-965f61-openshift-default-55854964c-zmth9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Oct 31 04:15:17.080353 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:17.075764 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-zmth9" podUID="5d825a50-865d-46c6-b88d-73f7170963c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Oct 31 04:15:17.436133 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:17.436101 4376 scope.go:122] "RemoveContainer" containerID="7e7f1c3ef691dff0910bf78ae8554b663a547696b2d8cb1a5eb19ff441457e7f" Oct 31 04:15:17.569833 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:17.569799 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 04:15:18.080100 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:18.080030 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-zmth9" Oct 31 04:15:18.081245 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:18.081221 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-zmth9" Oct 31 04:15:18.423241 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:18.423210 4376 generic.go:356] "Generic (PLEG): container finished" podID="2b3b5806-b3d7-48a8-b2d5-fb202004605d" containerID="903f8ec8804650023256d4b08dbb9e0cf0299163173d3e83a7ae97003a6ac292" exitCode=0 Oct 31 04:15:18.423425 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:18.423285 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-96rgs" event={"ID":"2b3b5806-b3d7-48a8-b2d5-fb202004605d","Type":"ContainerDied","Data":"903f8ec8804650023256d4b08dbb9e0cf0299163173d3e83a7ae97003a6ac292"} Oct 31 04:15:18.423671 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:18.423652 4376 scope.go:122] "RemoveContainer" containerID="903f8ec8804650023256d4b08dbb9e0cf0299163173d3e83a7ae97003a6ac292" Oct 31 04:15:18.425040 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:18.425008 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:15:18.425142 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:18.425086 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:15:18.425142 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:18.425131 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:15:18.425311 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:18.425293 4376 generic.go:356] "Generic (PLEG): container finished" podID="822172d2-2688-49fe-9a6b-628f32ea8b98" containerID="1561f1cf9ecae4088570f70f11b6211fefcad357ea61a9076f880204403e886c" exitCode=60 Oct 31 04:15:18.425390 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:18.425366 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-965f61-29890335-df4dh" event={"ID":"822172d2-2688-49fe-9a6b-628f32ea8b98","Type":"ContainerDied","Data":"1561f1cf9ecae4088570f70f11b6211fefcad357ea61a9076f880204403e886c"} Oct 31 04:15:18.425452 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:18.425413 4376 scope.go:122] "RemoveContainer" containerID="7e7f1c3ef691dff0910bf78ae8554b663a547696b2d8cb1a5eb19ff441457e7f" Oct 31 04:15:18.425939 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:18.425920 4376 scope.go:122] "RemoveContainer" containerID="1561f1cf9ecae4088570f70f11b6211fefcad357ea61a9076f880204403e886c" Oct 31 04:15:18.426150 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:15:18.426132 4376 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-965f61-29890335-df4dh_odh-ai-gateway-infra(822172d2-2688-49fe-9a6b-628f32ea8b98)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-965f61-29890335-df4dh" podUID="822172d2-2688-49fe-9a6b-628f32ea8b98" Oct 31 04:15:19.434612 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:19.434580 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-96rgs" event={"ID":"2b3b5806-b3d7-48a8-b2d5-fb202004605d","Type":"ContainerStarted","Data":"9c1b014c56cc8c247d4e85c151deb5101b818f50fee2a89082b85dea30d963ce"} Oct 31 04:15:19.570116 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:19.570080 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-965f61-29890335-df4dh"] Oct 31 04:15:19.794956 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:19.794931 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-965f61-29890335-df4dh" Oct 31 04:15:19.874138 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:19.874099 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/822172d2-2688-49fe-9a6b-628f32ea8b98-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/822172d2-2688-49fe-9a6b-628f32ea8b98-maas-api-service-ca\") pod \"822172d2-2688-49fe-9a6b-628f32ea8b98\" (UID: \"822172d2-2688-49fe-9a6b-628f32ea8b98\") " Oct 31 04:15:19.874358 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:19.874257 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/822172d2-2688-49fe-9a6b-628f32ea8b98-kube-api-access-6n6rb\" (UniqueName: \"kubernetes.io/projected/822172d2-2688-49fe-9a6b-628f32ea8b98-kube-api-access-6n6rb\") pod \"822172d2-2688-49fe-9a6b-628f32ea8b98\" (UID: \"822172d2-2688-49fe-9a6b-628f32ea8b98\") " Oct 31 04:15:19.874551 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:19.874525 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/822172d2-2688-49fe-9a6b-628f32ea8b98-maas-api-service-ca" pod "822172d2-2688-49fe-9a6b-628f32ea8b98" (UID: "822172d2-2688-49fe-9a6b-628f32ea8b98"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:15:19.874636 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:19.874592 4376 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/822172d2-2688-49fe-9a6b-628f32ea8b98-maas-api-service-ca\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:15:19.876306 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:19.876281 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/822172d2-2688-49fe-9a6b-628f32ea8b98-kube-api-access-6n6rb" pod "822172d2-2688-49fe-9a6b-628f32ea8b98" (UID: "822172d2-2688-49fe-9a6b-628f32ea8b98"). InnerVolumeSpecName "kube-api-access-6n6rb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:15:19.975465 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:19.975424 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6n6rb\" (UniqueName: \"kubernetes.io/projected/822172d2-2688-49fe-9a6b-628f32ea8b98-kube-api-access-6n6rb\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:15:20.027948 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:20.027870 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-96rgs" Oct 31 04:15:20.028883 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:20.028860 4376 patch_prober.go:29] interesting pod/e2e-worker-w1-439989-openshift-default-8448c6ff99-96rgs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Oct 31 04:15:20.028980 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:20.028908 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-96rgs" podUID="2b3b5806-b3d7-48a8-b2d5-fb202004605d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Oct 31 04:15:20.475453 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:20.475418 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-965f61-29890335-df4dh" Oct 31 04:15:20.475453 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:20.475427 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-965f61-29890335-df4dh" event={"ID":"822172d2-2688-49fe-9a6b-628f32ea8b98","Type":"ContainerDied","Data":"a4ba2117574b84bea5da81ad274901294b6aeb2ce62e7b900c8f3cadc9e0773b"} Oct 31 04:15:20.476086 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:20.475512 4376 scope.go:122] "RemoveContainer" containerID="1561f1cf9ecae4088570f70f11b6211fefcad357ea61a9076f880204403e886c" Oct 31 04:15:20.534701 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:20.534671 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-965f61-29890335-df4dh"] Oct 31 04:15:20.558360 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:20.555380 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-965f61-29890335-df4dh"] Oct 31 04:15:21.028232 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:21.028202 4376 patch_prober.go:29] interesting pod/e2e-worker-w1-439989-openshift-default-8448c6ff99-96rgs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Oct 31 04:15:21.028415 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:21.028252 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-96rgs" podUID="2b3b5806-b3d7-48a8-b2d5-fb202004605d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Oct 31 04:15:21.177041 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:21.177011 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-96rgs" Oct 31 04:15:22.028309 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:22.028276 4376 patch_prober.go:29] interesting pod/e2e-worker-w1-439989-openshift-default-8448c6ff99-96rgs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Oct 31 04:15:22.028680 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:22.028349 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-96rgs" podUID="2b3b5806-b3d7-48a8-b2d5-fb202004605d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Oct 31 04:15:22.456352 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:22.455910 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="822172d2-2688-49fe-9a6b-628f32ea8b98" path="/var/lib/kubelet/pods/822172d2-2688-49fe-9a6b-628f32ea8b98/volumes" Oct 31 04:15:22.489342 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:22.489303 4376 generic.go:356] "Generic (PLEG): container finished" podID="6957320f-9d86-4430-bca7-85db00d7a099" containerID="6f62c4f1a8beb2890f61a48c09ca5cd6a4e392a731114cdf4c1babee7bfda920" exitCode=0 Oct 31 04:15:22.489538 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:22.489392 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-brpvl" event={"ID":"6957320f-9d86-4430-bca7-85db00d7a099","Type":"ContainerDied","Data":"6f62c4f1a8beb2890f61a48c09ca5cd6a4e392a731114cdf4c1babee7bfda920"} Oct 31 04:15:22.490151 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:22.490131 4376 scope.go:122] "RemoveContainer" containerID="6f62c4f1a8beb2890f61a48c09ca5cd6a4e392a731114cdf4c1babee7bfda920" Oct 31 04:15:22.491551 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:22.491525 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:15:22.491733 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:22.491713 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:15:22.491888 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:22.491868 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:15:23.027430 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:23.027397 4376 patch_prober.go:29] interesting pod/e2e-worker-w1-439989-openshift-default-8448c6ff99-96rgs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Oct 31 04:15:23.027600 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:23.027457 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-96rgs" podUID="2b3b5806-b3d7-48a8-b2d5-fb202004605d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Oct 31 04:15:23.497906 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:23.497874 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-brpvl" event={"ID":"6957320f-9d86-4430-bca7-85db00d7a099","Type":"ContainerStarted","Data":"ca88dfd607dbbf2878cd20e8f583c5ecee8952003e85169731c0a12d701e9315"} Oct 31 04:15:24.023658 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:24.023626 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-brpvl" Oct 31 04:15:24.024920 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:24.024892 4376 patch_prober.go:29] interesting pod/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-brpvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Oct 31 04:15:24.025045 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:24.024942 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-brpvl" podUID="6957320f-9d86-4430-bca7-85db00d7a099" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Oct 31 04:15:24.028223 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:24.028204 4376 patch_prober.go:29] interesting pod/e2e-worker-w1-439989-openshift-default-8448c6ff99-96rgs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Oct 31 04:15:24.028270 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:24.028244 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-96rgs" podUID="2b3b5806-b3d7-48a8-b2d5-fb202004605d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Oct 31 04:15:25.023888 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:25.023810 4376 patch_prober.go:29] interesting pod/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-brpvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Oct 31 04:15:25.023888 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:25.023860 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-brpvl" podUID="6957320f-9d86-4430-bca7-85db00d7a099" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Oct 31 04:15:25.032138 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:25.032113 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-96rgs" Oct 31 04:15:25.033132 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:25.033111 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-96rgs" Oct 31 04:15:25.203723 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:25.203697 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-brpvl" Oct 31 04:15:26.024608 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:26.024564 4376 patch_prober.go:29] interesting pod/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-brpvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Oct 31 04:15:26.025020 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:26.024634 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-brpvl" podUID="6957320f-9d86-4430-bca7-85db00d7a099" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Oct 31 04:15:26.050971 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:26.050937 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 04:15:27.024647 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:27.024612 4376 patch_prober.go:29] interesting pod/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-brpvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Oct 31 04:15:27.025054 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:27.024676 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-brpvl" podUID="6957320f-9d86-4430-bca7-85db00d7a099" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Oct 31 04:15:28.024455 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:28.024417 4376 patch_prober.go:29] interesting pod/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-brpvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Oct 31 04:15:28.024730 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:28.024481 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-brpvl" podUID="6957320f-9d86-4430-bca7-85db00d7a099" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Oct 31 04:15:28.089015 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:28.088980 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-09a20a/premium-w2-09a20a-kserve-57c45db74f-s8glf"] Oct 31 04:15:28.089587 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:28.089568 4376 cpu_manager.go:414] "containerMap: removing container" podUID="822172d2-2688-49fe-9a6b-628f32ea8b98" containerName="cleanup" Oct 31 04:15:28.089665 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:28.089588 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="822172d2-2688-49fe-9a6b-628f32ea8b98" containerName="cleanup" Oct 31 04:15:28.089665 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:28.089602 4376 cpu_manager.go:414] "containerMap: removing container" podUID="822172d2-2688-49fe-9a6b-628f32ea8b98" containerName="cleanup" Oct 31 04:15:28.089665 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:28.089607 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="822172d2-2688-49fe-9a6b-628f32ea8b98" containerName="cleanup" Oct 31 04:15:28.089665 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:28.089634 4376 cpu_manager.go:414] "containerMap: removing container" podUID="822172d2-2688-49fe-9a6b-628f32ea8b98" containerName="cleanup" Oct 31 04:15:28.089665 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:28.089642 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="822172d2-2688-49fe-9a6b-628f32ea8b98" containerName="cleanup" Oct 31 04:15:28.089815 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:28.089730 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="822172d2-2688-49fe-9a6b-628f32ea8b98" containerName="cleanup" Oct 31 04:15:28.089815 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:28.089739 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="822172d2-2688-49fe-9a6b-628f32ea8b98" containerName="cleanup" Oct 31 04:15:28.092849 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:28.092827 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-09a20a/premium-w2-09a20a-kserve-57c45db74f-s8glf" Oct 31 04:15:28.116873 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:28.116838 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-09a20a/premium-w2-09a20a-kserve-57c45db74f-s8glf"] Oct 31 04:15:28.126396 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:28.126372 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-09a20a\"/\"premium-w2-09a20a-kserve-self-signed-certs\"" Oct 31 04:15:28.126563 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:28.126372 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-09a20a\"/\"kube-root-ca.crt\"" Oct 31 04:15:28.126650 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:28.126372 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-09a20a\"/\"openshift-service-ca.crt\"" Oct 31 04:15:28.126650 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:28.126389 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-09a20a\"/\"default-dockercfg-qcqb8\"" Oct 31 04:15:28.273899 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:28.273859 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3f9d0a0-a106-48bb-b6f2-a980c3aff4e7-model-cache\") pod \"premium-w2-09a20a-kserve-57c45db74f-s8glf\" (UID: \"a3f9d0a0-a106-48bb-b6f2-a980c3aff4e7\") " pod="e2e-models-e2e-worker-w2-09a20a/premium-w2-09a20a-kserve-57c45db74f-s8glf" Oct 31 04:15:28.274109 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:28.273913 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3f9d0a0-a106-48bb-b6f2-a980c3aff4e7-tls-certs\") pod \"premium-w2-09a20a-kserve-57c45db74f-s8glf\" (UID: \"a3f9d0a0-a106-48bb-b6f2-a980c3aff4e7\") " pod="e2e-models-e2e-worker-w2-09a20a/premium-w2-09a20a-kserve-57c45db74f-s8glf" Oct 31 04:15:28.274109 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:28.274001 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3f9d0a0-a106-48bb-b6f2-a980c3aff4e7-tmp-dir\") pod \"premium-w2-09a20a-kserve-57c45db74f-s8glf\" (UID: \"a3f9d0a0-a106-48bb-b6f2-a980c3aff4e7\") " pod="e2e-models-e2e-worker-w2-09a20a/premium-w2-09a20a-kserve-57c45db74f-s8glf" Oct 31 04:15:28.274204 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:28.274129 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3f9d0a0-a106-48bb-b6f2-a980c3aff4e7-dshm\") pod \"premium-w2-09a20a-kserve-57c45db74f-s8glf\" (UID: \"a3f9d0a0-a106-48bb-b6f2-a980c3aff4e7\") " pod="e2e-models-e2e-worker-w2-09a20a/premium-w2-09a20a-kserve-57c45db74f-s8glf" Oct 31 04:15:28.274204 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:28.274195 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvsj5\" (UniqueName: \"kubernetes.io/projected/a3f9d0a0-a106-48bb-b6f2-a980c3aff4e7-kube-api-access-dvsj5\") pod \"premium-w2-09a20a-kserve-57c45db74f-s8glf\" (UID: \"a3f9d0a0-a106-48bb-b6f2-a980c3aff4e7\") " pod="e2e-models-e2e-worker-w2-09a20a/premium-w2-09a20a-kserve-57c45db74f-s8glf" Oct 31 04:15:28.274306 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:28.274272 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3f9d0a0-a106-48bb-b6f2-a980c3aff4e7-home\") pod \"premium-w2-09a20a-kserve-57c45db74f-s8glf\" (UID: \"a3f9d0a0-a106-48bb-b6f2-a980c3aff4e7\") " pod="e2e-models-e2e-worker-w2-09a20a/premium-w2-09a20a-kserve-57c45db74f-s8glf" Oct 31 04:15:28.375486 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:28.375450 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3f9d0a0-a106-48bb-b6f2-a980c3aff4e7-home\") pod \"premium-w2-09a20a-kserve-57c45db74f-s8glf\" (UID: \"a3f9d0a0-a106-48bb-b6f2-a980c3aff4e7\") " pod="e2e-models-e2e-worker-w2-09a20a/premium-w2-09a20a-kserve-57c45db74f-s8glf" Oct 31 04:15:28.375671 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:28.375497 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3f9d0a0-a106-48bb-b6f2-a980c3aff4e7-model-cache\") pod \"premium-w2-09a20a-kserve-57c45db74f-s8glf\" (UID: \"a3f9d0a0-a106-48bb-b6f2-a980c3aff4e7\") " pod="e2e-models-e2e-worker-w2-09a20a/premium-w2-09a20a-kserve-57c45db74f-s8glf" Oct 31 04:15:28.375671 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:28.375543 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3f9d0a0-a106-48bb-b6f2-a980c3aff4e7-tls-certs\") pod \"premium-w2-09a20a-kserve-57c45db74f-s8glf\" (UID: \"a3f9d0a0-a106-48bb-b6f2-a980c3aff4e7\") " pod="e2e-models-e2e-worker-w2-09a20a/premium-w2-09a20a-kserve-57c45db74f-s8glf" Oct 31 04:15:28.375671 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:28.375579 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3f9d0a0-a106-48bb-b6f2-a980c3aff4e7-tmp-dir\") pod \"premium-w2-09a20a-kserve-57c45db74f-s8glf\" (UID: \"a3f9d0a0-a106-48bb-b6f2-a980c3aff4e7\") " pod="e2e-models-e2e-worker-w2-09a20a/premium-w2-09a20a-kserve-57c45db74f-s8glf" Oct 31 04:15:28.375827 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:28.375677 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3f9d0a0-a106-48bb-b6f2-a980c3aff4e7-dshm\") pod \"premium-w2-09a20a-kserve-57c45db74f-s8glf\" (UID: \"a3f9d0a0-a106-48bb-b6f2-a980c3aff4e7\") " pod="e2e-models-e2e-worker-w2-09a20a/premium-w2-09a20a-kserve-57c45db74f-s8glf" Oct 31 04:15:28.375827 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:28.375780 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvsj5\" (UniqueName: \"kubernetes.io/projected/a3f9d0a0-a106-48bb-b6f2-a980c3aff4e7-kube-api-access-dvsj5\") pod \"premium-w2-09a20a-kserve-57c45db74f-s8glf\" (UID: \"a3f9d0a0-a106-48bb-b6f2-a980c3aff4e7\") " pod="e2e-models-e2e-worker-w2-09a20a/premium-w2-09a20a-kserve-57c45db74f-s8glf" Oct 31 04:15:28.376015 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:28.375995 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3f9d0a0-a106-48bb-b6f2-a980c3aff4e7-home\") pod \"premium-w2-09a20a-kserve-57c45db74f-s8glf\" (UID: \"a3f9d0a0-a106-48bb-b6f2-a980c3aff4e7\") " pod="e2e-models-e2e-worker-w2-09a20a/premium-w2-09a20a-kserve-57c45db74f-s8glf" Oct 31 04:15:28.376082 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:28.376058 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3f9d0a0-a106-48bb-b6f2-a980c3aff4e7-model-cache\") pod \"premium-w2-09a20a-kserve-57c45db74f-s8glf\" (UID: \"a3f9d0a0-a106-48bb-b6f2-a980c3aff4e7\") " pod="e2e-models-e2e-worker-w2-09a20a/premium-w2-09a20a-kserve-57c45db74f-s8glf" Oct 31 04:15:28.376129 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:28.376079 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3f9d0a0-a106-48bb-b6f2-a980c3aff4e7-tmp-dir\") pod \"premium-w2-09a20a-kserve-57c45db74f-s8glf\" (UID: \"a3f9d0a0-a106-48bb-b6f2-a980c3aff4e7\") " pod="e2e-models-e2e-worker-w2-09a20a/premium-w2-09a20a-kserve-57c45db74f-s8glf" Oct 31 04:15:28.378034 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:28.378011 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3f9d0a0-a106-48bb-b6f2-a980c3aff4e7-dshm\") pod \"premium-w2-09a20a-kserve-57c45db74f-s8glf\" (UID: \"a3f9d0a0-a106-48bb-b6f2-a980c3aff4e7\") " pod="e2e-models-e2e-worker-w2-09a20a/premium-w2-09a20a-kserve-57c45db74f-s8glf" Oct 31 04:15:28.378136 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:28.378092 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3f9d0a0-a106-48bb-b6f2-a980c3aff4e7-tls-certs\") pod \"premium-w2-09a20a-kserve-57c45db74f-s8glf\" (UID: \"a3f9d0a0-a106-48bb-b6f2-a980c3aff4e7\") " pod="e2e-models-e2e-worker-w2-09a20a/premium-w2-09a20a-kserve-57c45db74f-s8glf" Oct 31 04:15:28.401219 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:28.401186 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvsj5\" (UniqueName: \"kubernetes.io/projected/a3f9d0a0-a106-48bb-b6f2-a980c3aff4e7-kube-api-access-dvsj5\") pod \"premium-w2-09a20a-kserve-57c45db74f-s8glf\" (UID: \"a3f9d0a0-a106-48bb-b6f2-a980c3aff4e7\") " pod="e2e-models-e2e-worker-w2-09a20a/premium-w2-09a20a-kserve-57c45db74f-s8glf" Oct 31 04:15:28.409086 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:28.409068 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-09a20a/premium-w2-09a20a-kserve-57c45db74f-s8glf" Oct 31 04:15:28.736748 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:28.736711 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-09a20a/premium-w2-09a20a-kserve-57c45db74f-s8glf"] Oct 31 04:15:29.028629 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:29.028553 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-brpvl" Oct 31 04:15:29.029594 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:29.029576 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-brpvl" Oct 31 04:15:29.530148 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:29.530111 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-09a20a/premium-w2-09a20a-kserve-57c45db74f-s8glf" event={"ID":"a3f9d0a0-a106-48bb-b6f2-a980c3aff4e7","Type":"ContainerStarted","Data":"0ecb5f1038cfbba39690602f96218fc175fbff09c119a5928389d7fd428ac72d"} Oct 31 04:15:29.530360 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:29.530156 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-09a20a/premium-w2-09a20a-kserve-57c45db74f-s8glf" event={"ID":"a3f9d0a0-a106-48bb-b6f2-a980c3aff4e7","Type":"ContainerStarted","Data":"0c11a845978de64293a2fcc20dabe52f0c11773ca6c699862471f7bf28c5d440"} Oct 31 04:15:29.578194 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:29.578141 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-09a20a/premium-w2-09a20a-kserve-57c45db74f-s8glf" podStartSLOduration=1.5781239 podStartE2EDuration="1.5781239s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:15:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:15:29.576846811 +0000 UTC m=+677.291181428" watchObservedRunningTime="2026-10-31 04:15:29.5781239 +0000 UTC m=+677.292458522" Oct 31 04:15:30.409702 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:30.409672 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-09a20a/premium-w2-09a20a-kserve-57c45db74f-s8glf" Oct 31 04:15:33.560793 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:33.560757 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-c4006d/simulator-w4-c4006d-kserve-595455485b-4h46q"] Oct 31 04:15:33.561662 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:33.561643 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="822172d2-2688-49fe-9a6b-628f32ea8b98" containerName="cleanup" Oct 31 04:15:33.565562 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:33.565536 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c4006d/simulator-w4-c4006d-kserve-595455485b-4h46q" Oct 31 04:15:33.581468 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:33.581443 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-c4006d/simulator-w4-c4006d-kserve-595455485b-4h46q"] Oct 31 04:15:33.593526 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:33.593497 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-c4006d\"/\"openshift-service-ca.crt\"" Oct 31 04:15:33.593526 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:33.593499 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-c4006d\"/\"kube-root-ca.crt\"" Oct 31 04:15:33.596543 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:33.596525 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-c4006d\"/\"simulator-w4-c4006d-kserve-self-signed-certs\"" Oct 31 04:15:33.596669 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:33.596544 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-c4006d\"/\"default-dockercfg-kj674\"" Oct 31 04:15:33.620348 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:33.620309 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/576e2c1f-1c80-43a9-a136-3692776642ae-model-cache\") pod \"simulator-w4-c4006d-kserve-595455485b-4h46q\" (UID: \"576e2c1f-1c80-43a9-a136-3692776642ae\") " pod="e2e-models-e2e-worker-w4-c4006d/simulator-w4-c4006d-kserve-595455485b-4h46q" Oct 31 04:15:33.620478 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:33.620427 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/576e2c1f-1c80-43a9-a136-3692776642ae-tls-certs\") pod \"simulator-w4-c4006d-kserve-595455485b-4h46q\" (UID: \"576e2c1f-1c80-43a9-a136-3692776642ae\") " pod="e2e-models-e2e-worker-w4-c4006d/simulator-w4-c4006d-kserve-595455485b-4h46q" Oct 31 04:15:33.620541 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:33.620482 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/576e2c1f-1c80-43a9-a136-3692776642ae-tmp-dir\") pod \"simulator-w4-c4006d-kserve-595455485b-4h46q\" (UID: \"576e2c1f-1c80-43a9-a136-3692776642ae\") " pod="e2e-models-e2e-worker-w4-c4006d/simulator-w4-c4006d-kserve-595455485b-4h46q" Oct 31 04:15:33.620541 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:33.620501 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/576e2c1f-1c80-43a9-a136-3692776642ae-home\") pod \"simulator-w4-c4006d-kserve-595455485b-4h46q\" (UID: \"576e2c1f-1c80-43a9-a136-3692776642ae\") " pod="e2e-models-e2e-worker-w4-c4006d/simulator-w4-c4006d-kserve-595455485b-4h46q" Oct 31 04:15:33.620646 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:33.620562 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/576e2c1f-1c80-43a9-a136-3692776642ae-dshm\") pod \"simulator-w4-c4006d-kserve-595455485b-4h46q\" (UID: \"576e2c1f-1c80-43a9-a136-3692776642ae\") " pod="e2e-models-e2e-worker-w4-c4006d/simulator-w4-c4006d-kserve-595455485b-4h46q" Oct 31 04:15:33.620646 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:33.620583 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nn6xd\" (UniqueName: \"kubernetes.io/projected/576e2c1f-1c80-43a9-a136-3692776642ae-kube-api-access-nn6xd\") pod \"simulator-w4-c4006d-kserve-595455485b-4h46q\" (UID: \"576e2c1f-1c80-43a9-a136-3692776642ae\") " pod="e2e-models-e2e-worker-w4-c4006d/simulator-w4-c4006d-kserve-595455485b-4h46q" Oct 31 04:15:33.721474 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:33.721444 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/576e2c1f-1c80-43a9-a136-3692776642ae-tls-certs\") pod \"simulator-w4-c4006d-kserve-595455485b-4h46q\" (UID: \"576e2c1f-1c80-43a9-a136-3692776642ae\") " pod="e2e-models-e2e-worker-w4-c4006d/simulator-w4-c4006d-kserve-595455485b-4h46q" Oct 31 04:15:33.721474 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:33.721478 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/576e2c1f-1c80-43a9-a136-3692776642ae-tmp-dir\") pod \"simulator-w4-c4006d-kserve-595455485b-4h46q\" (UID: \"576e2c1f-1c80-43a9-a136-3692776642ae\") " pod="e2e-models-e2e-worker-w4-c4006d/simulator-w4-c4006d-kserve-595455485b-4h46q" Oct 31 04:15:33.721709 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:33.721498 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/576e2c1f-1c80-43a9-a136-3692776642ae-home\") pod \"simulator-w4-c4006d-kserve-595455485b-4h46q\" (UID: \"576e2c1f-1c80-43a9-a136-3692776642ae\") " pod="e2e-models-e2e-worker-w4-c4006d/simulator-w4-c4006d-kserve-595455485b-4h46q" Oct 31 04:15:33.721709 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:33.721530 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/576e2c1f-1c80-43a9-a136-3692776642ae-dshm\") pod \"simulator-w4-c4006d-kserve-595455485b-4h46q\" (UID: \"576e2c1f-1c80-43a9-a136-3692776642ae\") " pod="e2e-models-e2e-worker-w4-c4006d/simulator-w4-c4006d-kserve-595455485b-4h46q" Oct 31 04:15:33.721709 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:33.721550 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nn6xd\" (UniqueName: \"kubernetes.io/projected/576e2c1f-1c80-43a9-a136-3692776642ae-kube-api-access-nn6xd\") pod \"simulator-w4-c4006d-kserve-595455485b-4h46q\" (UID: \"576e2c1f-1c80-43a9-a136-3692776642ae\") " pod="e2e-models-e2e-worker-w4-c4006d/simulator-w4-c4006d-kserve-595455485b-4h46q" Oct 31 04:15:33.721709 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:33.721630 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/576e2c1f-1c80-43a9-a136-3692776642ae-model-cache\") pod \"simulator-w4-c4006d-kserve-595455485b-4h46q\" (UID: \"576e2c1f-1c80-43a9-a136-3692776642ae\") " pod="e2e-models-e2e-worker-w4-c4006d/simulator-w4-c4006d-kserve-595455485b-4h46q" Oct 31 04:15:33.721920 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:33.721880 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/576e2c1f-1c80-43a9-a136-3692776642ae-tmp-dir\") pod \"simulator-w4-c4006d-kserve-595455485b-4h46q\" (UID: \"576e2c1f-1c80-43a9-a136-3692776642ae\") " pod="e2e-models-e2e-worker-w4-c4006d/simulator-w4-c4006d-kserve-595455485b-4h46q" Oct 31 04:15:33.721974 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:33.721917 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/576e2c1f-1c80-43a9-a136-3692776642ae-home\") pod \"simulator-w4-c4006d-kserve-595455485b-4h46q\" (UID: \"576e2c1f-1c80-43a9-a136-3692776642ae\") " pod="e2e-models-e2e-worker-w4-c4006d/simulator-w4-c4006d-kserve-595455485b-4h46q" Oct 31 04:15:33.722013 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:33.721993 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/576e2c1f-1c80-43a9-a136-3692776642ae-model-cache\") pod \"simulator-w4-c4006d-kserve-595455485b-4h46q\" (UID: \"576e2c1f-1c80-43a9-a136-3692776642ae\") " pod="e2e-models-e2e-worker-w4-c4006d/simulator-w4-c4006d-kserve-595455485b-4h46q" Oct 31 04:15:33.723746 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:33.723723 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/576e2c1f-1c80-43a9-a136-3692776642ae-dshm\") pod \"simulator-w4-c4006d-kserve-595455485b-4h46q\" (UID: \"576e2c1f-1c80-43a9-a136-3692776642ae\") " pod="e2e-models-e2e-worker-w4-c4006d/simulator-w4-c4006d-kserve-595455485b-4h46q" Oct 31 04:15:33.723908 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:33.723887 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/576e2c1f-1c80-43a9-a136-3692776642ae-tls-certs\") pod \"simulator-w4-c4006d-kserve-595455485b-4h46q\" (UID: \"576e2c1f-1c80-43a9-a136-3692776642ae\") " pod="e2e-models-e2e-worker-w4-c4006d/simulator-w4-c4006d-kserve-595455485b-4h46q" Oct 31 04:15:33.747180 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:33.747156 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nn6xd\" (UniqueName: \"kubernetes.io/projected/576e2c1f-1c80-43a9-a136-3692776642ae-kube-api-access-nn6xd\") pod \"simulator-w4-c4006d-kserve-595455485b-4h46q\" (UID: \"576e2c1f-1c80-43a9-a136-3692776642ae\") " pod="e2e-models-e2e-worker-w4-c4006d/simulator-w4-c4006d-kserve-595455485b-4h46q" Oct 31 04:15:33.881868 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:33.881832 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c4006d/simulator-w4-c4006d-kserve-595455485b-4h46q" Oct 31 04:15:34.180630 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:34.180604 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-c4006d/simulator-w4-c4006d-kserve-595455485b-4h46q"] Oct 31 04:15:34.180773 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:15:34.180658 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod576e2c1f_1c80_43a9_a136_3692776642ae.slice/crio-78a655ba0aafddf6f2d1f028387bbf418f903e71b3baefc4c69287f9d8fea67b WatchSource:0}: Error finding container 78a655ba0aafddf6f2d1f028387bbf418f903e71b3baefc4c69287f9d8fea67b: Status 404 returned error can't find the container with id 78a655ba0aafddf6f2d1f028387bbf418f903e71b3baefc4c69287f9d8fea67b Oct 31 04:15:34.562608 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:34.562507 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c4006d/simulator-w4-c4006d-kserve-595455485b-4h46q" event={"ID":"576e2c1f-1c80-43a9-a136-3692776642ae","Type":"ContainerStarted","Data":"91b226397f85d99ec63a941b055c3984264717467120c5a5fa7954382d6cc2fe"} Oct 31 04:15:34.562608 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:34.562551 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c4006d/simulator-w4-c4006d-kserve-595455485b-4h46q" event={"ID":"576e2c1f-1c80-43a9-a136-3692776642ae","Type":"ContainerStarted","Data":"78a655ba0aafddf6f2d1f028387bbf418f903e71b3baefc4c69287f9d8fea67b"} Oct 31 04:15:34.608346 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:34.608287 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-c4006d/simulator-w4-c4006d-kserve-595455485b-4h46q" podStartSLOduration=1.6082736560000002 podStartE2EDuration="1.608273656s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:15:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:15:34.607498408 +0000 UTC m=+682.321833025" watchObservedRunningTime="2026-10-31 04:15:34.608273656 +0000 UTC m=+682.322608270" Oct 31 04:15:34.882239 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:34.882201 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c4006d/simulator-w4-c4006d-kserve-595455485b-4h46q" Oct 31 04:15:37.953958 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:37.953914 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 04:15:38.410161 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:38.410126 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-09a20a/premium-w2-09a20a-kserve-57c45db74f-s8glf" Oct 31 04:15:38.423193 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:38.423162 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-09a20a/premium-w2-09a20a-kserve-57c45db74f-s8glf" Oct 31 04:15:38.602631 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:38.602592 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-09a20a/premium-w2-09a20a-kserve-57c45db74f-s8glf" Oct 31 04:15:40.999146 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:15:40.999106 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 04:15:43.883019 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:43.882988 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-c4006d/simulator-w4-c4006d-kserve-595455485b-4h46q" Oct 31 04:15:43.895928 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:43.895905 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-c4006d/simulator-w4-c4006d-kserve-595455485b-4h46q" Oct 31 04:15:44.629031 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:44.629001 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c4006d/simulator-w4-c4006d-kserve-595455485b-4h46q" Oct 31 04:15:50.002568 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:50.002537 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 04:15:50.427483 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:50.427452 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-zlcr5"] Oct 31 04:15:50.432470 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:50.432447 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-zlcr5" Oct 31 04:15:50.464189 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:50.464162 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-09a20a-openshift-default-dockercfg-hf9qb\"" Oct 31 04:15:50.467078 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:50.467053 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-zlcr5"] Oct 31 04:15:50.487997 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:50.487963 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2697f74a-c2a2-4d6f-ba09-adb5825a8d85-workload-socket\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-zlcr5\" (UID: \"2697f74a-c2a2-4d6f-ba09-adb5825a8d85\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-zlcr5" Oct 31 04:15:50.488129 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:50.488005 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2697f74a-c2a2-4d6f-ba09-adb5825a8d85-istiod-ca-cert\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-zlcr5\" (UID: \"2697f74a-c2a2-4d6f-ba09-adb5825a8d85\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-zlcr5" Oct 31 04:15:50.488187 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:50.488124 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2697f74a-c2a2-4d6f-ba09-adb5825a8d85-istio-envoy\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-zlcr5\" (UID: \"2697f74a-c2a2-4d6f-ba09-adb5825a8d85\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-zlcr5" Oct 31 04:15:50.488187 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:50.488150 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2697f74a-c2a2-4d6f-ba09-adb5825a8d85-istio-token\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-zlcr5\" (UID: \"2697f74a-c2a2-4d6f-ba09-adb5825a8d85\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-zlcr5" Oct 31 04:15:50.488296 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:50.488269 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2697f74a-c2a2-4d6f-ba09-adb5825a8d85-workload-certs\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-zlcr5\" (UID: \"2697f74a-c2a2-4d6f-ba09-adb5825a8d85\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-zlcr5" Oct 31 04:15:50.488353 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:50.488307 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2697f74a-c2a2-4d6f-ba09-adb5825a8d85-credential-socket\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-zlcr5\" (UID: \"2697f74a-c2a2-4d6f-ba09-adb5825a8d85\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-zlcr5" Oct 31 04:15:50.488447 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:50.488430 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vc9bv\" (UniqueName: \"kubernetes.io/projected/2697f74a-c2a2-4d6f-ba09-adb5825a8d85-kube-api-access-vc9bv\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-zlcr5\" (UID: \"2697f74a-c2a2-4d6f-ba09-adb5825a8d85\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-zlcr5" Oct 31 04:15:50.488505 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:50.488490 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2697f74a-c2a2-4d6f-ba09-adb5825a8d85-istio-data\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-zlcr5\" (UID: \"2697f74a-c2a2-4d6f-ba09-adb5825a8d85\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-zlcr5" Oct 31 04:15:50.488539 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:50.488523 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2697f74a-c2a2-4d6f-ba09-adb5825a8d85-istio-podinfo\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-zlcr5\" (UID: \"2697f74a-c2a2-4d6f-ba09-adb5825a8d85\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-zlcr5" Oct 31 04:15:50.589451 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:50.589425 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2697f74a-c2a2-4d6f-ba09-adb5825a8d85-workload-socket\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-zlcr5\" (UID: \"2697f74a-c2a2-4d6f-ba09-adb5825a8d85\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-zlcr5" Oct 31 04:15:50.589587 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:50.589458 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2697f74a-c2a2-4d6f-ba09-adb5825a8d85-istiod-ca-cert\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-zlcr5\" (UID: \"2697f74a-c2a2-4d6f-ba09-adb5825a8d85\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-zlcr5" Oct 31 04:15:50.589587 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:50.589505 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2697f74a-c2a2-4d6f-ba09-adb5825a8d85-istio-envoy\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-zlcr5\" (UID: \"2697f74a-c2a2-4d6f-ba09-adb5825a8d85\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-zlcr5" Oct 31 04:15:50.589587 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:50.589528 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2697f74a-c2a2-4d6f-ba09-adb5825a8d85-istio-token\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-zlcr5\" (UID: \"2697f74a-c2a2-4d6f-ba09-adb5825a8d85\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-zlcr5" Oct 31 04:15:50.589587 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:50.589585 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2697f74a-c2a2-4d6f-ba09-adb5825a8d85-workload-certs\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-zlcr5\" (UID: \"2697f74a-c2a2-4d6f-ba09-adb5825a8d85\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-zlcr5" Oct 31 04:15:50.589800 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:50.589616 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2697f74a-c2a2-4d6f-ba09-adb5825a8d85-credential-socket\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-zlcr5\" (UID: \"2697f74a-c2a2-4d6f-ba09-adb5825a8d85\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-zlcr5" Oct 31 04:15:50.589903 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:50.589880 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2697f74a-c2a2-4d6f-ba09-adb5825a8d85-workload-socket\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-zlcr5\" (UID: \"2697f74a-c2a2-4d6f-ba09-adb5825a8d85\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-zlcr5" Oct 31 04:15:50.590060 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:50.589925 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vc9bv\" (UniqueName: \"kubernetes.io/projected/2697f74a-c2a2-4d6f-ba09-adb5825a8d85-kube-api-access-vc9bv\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-zlcr5\" (UID: \"2697f74a-c2a2-4d6f-ba09-adb5825a8d85\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-zlcr5" Oct 31 04:15:50.590200 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:50.590163 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2697f74a-c2a2-4d6f-ba09-adb5825a8d85-credential-socket\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-zlcr5\" (UID: \"2697f74a-c2a2-4d6f-ba09-adb5825a8d85\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-zlcr5" Oct 31 04:15:50.590200 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:50.590180 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2697f74a-c2a2-4d6f-ba09-adb5825a8d85-istio-data\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-zlcr5\" (UID: \"2697f74a-c2a2-4d6f-ba09-adb5825a8d85\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-zlcr5" Oct 31 04:15:50.590353 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:50.590226 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2697f74a-c2a2-4d6f-ba09-adb5825a8d85-istio-podinfo\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-zlcr5\" (UID: \"2697f74a-c2a2-4d6f-ba09-adb5825a8d85\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-zlcr5" Oct 31 04:15:50.590353 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:50.590101 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2697f74a-c2a2-4d6f-ba09-adb5825a8d85-workload-certs\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-zlcr5\" (UID: \"2697f74a-c2a2-4d6f-ba09-adb5825a8d85\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-zlcr5" Oct 31 04:15:50.590475 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:50.590347 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2697f74a-c2a2-4d6f-ba09-adb5825a8d85-istiod-ca-cert\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-zlcr5\" (UID: \"2697f74a-c2a2-4d6f-ba09-adb5825a8d85\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-zlcr5" Oct 31 04:15:50.590475 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:50.590452 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2697f74a-c2a2-4d6f-ba09-adb5825a8d85-istio-data\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-zlcr5\" (UID: \"2697f74a-c2a2-4d6f-ba09-adb5825a8d85\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-zlcr5" Oct 31 04:15:50.592236 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:50.592212 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2697f74a-c2a2-4d6f-ba09-adb5825a8d85-istio-envoy\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-zlcr5\" (UID: \"2697f74a-c2a2-4d6f-ba09-adb5825a8d85\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-zlcr5" Oct 31 04:15:50.592729 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:50.592707 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2697f74a-c2a2-4d6f-ba09-adb5825a8d85-istio-podinfo\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-zlcr5\" (UID: \"2697f74a-c2a2-4d6f-ba09-adb5825a8d85\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-zlcr5" Oct 31 04:15:50.635732 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:50.635701 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2697f74a-c2a2-4d6f-ba09-adb5825a8d85-istio-token\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-zlcr5\" (UID: \"2697f74a-c2a2-4d6f-ba09-adb5825a8d85\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-zlcr5" Oct 31 04:15:50.643815 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:50.643758 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vc9bv\" (UniqueName: \"kubernetes.io/projected/2697f74a-c2a2-4d6f-ba09-adb5825a8d85-kube-api-access-vc9bv\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-zlcr5\" (UID: \"2697f74a-c2a2-4d6f-ba09-adb5825a8d85\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-zlcr5" Oct 31 04:15:50.749213 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:50.749120 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-zlcr5" Oct 31 04:15:51.058290 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:51.058263 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-zlcr5"] Oct 31 04:15:51.058290 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:15:51.058278 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2697f74a_c2a2_4d6f_ba09_adb5825a8d85.slice/crio-5fd870b7b77a36c598a327aa050afe4caf5f5ac6a8fbad0928fd405e997c3a42 WatchSource:0}: Error finding container 5fd870b7b77a36c598a327aa050afe4caf5f5ac6a8fbad0928fd405e997c3a42: Status 404 returned error can't find the container with id 5fd870b7b77a36c598a327aa050afe4caf5f5ac6a8fbad0928fd405e997c3a42 Oct 31 04:15:51.060659 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:51.060620 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:15:51.060732 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:51.060687 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:15:51.060732 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:51.060727 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:15:51.670041 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:51.670004 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-zlcr5" event={"ID":"2697f74a-c2a2-4d6f-ba09-adb5825a8d85","Type":"ContainerStarted","Data":"47d9a120ee2b82fc6f02a7ea9c24f974f093b8068328e505c7ecee1dd113dc9d"} Oct 31 04:15:51.670251 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:51.670061 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-zlcr5" event={"ID":"2697f74a-c2a2-4d6f-ba09-adb5825a8d85","Type":"ContainerStarted","Data":"5fd870b7b77a36c598a327aa050afe4caf5f5ac6a8fbad0928fd405e997c3a42"} Oct 31 04:15:51.741972 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:51.741910 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-zlcr5" podStartSLOduration=1.741892969 podStartE2EDuration="1.741892969s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:15:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:15:51.72784236 +0000 UTC m=+699.442176978" watchObservedRunningTime="2026-10-31 04:15:51.741892969 +0000 UTC m=+699.456227590" Oct 31 04:15:51.749989 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:51.749924 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-zlcr5" Oct 31 04:15:52.757375 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:52.752265 4376 patch_prober.go:29] interesting pod/e2e-worker-w2-09a20a-openshift-default-55d8647ff-zlcr5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Oct 31 04:15:52.757375 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:52.752319 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-zlcr5" podUID="2697f74a-c2a2-4d6f-ba09-adb5825a8d85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Oct 31 04:15:53.754362 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:53.753641 4376 patch_prober.go:29] interesting pod/e2e-worker-w2-09a20a-openshift-default-55d8647ff-zlcr5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Oct 31 04:15:53.754362 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:53.753693 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-zlcr5" podUID="2697f74a-c2a2-4d6f-ba09-adb5825a8d85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Oct 31 04:15:54.758362 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:54.757540 4376 patch_prober.go:29] interesting pod/e2e-worker-w2-09a20a-openshift-default-55d8647ff-zlcr5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Oct 31 04:15:54.758362 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:54.757617 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-zlcr5" podUID="2697f74a-c2a2-4d6f-ba09-adb5825a8d85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Oct 31 04:15:55.751717 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:55.751679 4376 patch_prober.go:29] interesting pod/e2e-worker-w2-09a20a-openshift-default-55d8647ff-zlcr5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Oct 31 04:15:55.752000 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:55.751971 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-zlcr5" podUID="2697f74a-c2a2-4d6f-ba09-adb5825a8d85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Oct 31 04:15:56.756351 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:56.752680 4376 patch_prober.go:29] interesting pod/e2e-worker-w2-09a20a-openshift-default-55d8647ff-zlcr5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Oct 31 04:15:56.756351 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:56.752755 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-zlcr5" podUID="2697f74a-c2a2-4d6f-ba09-adb5825a8d85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Oct 31 04:15:57.753448 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:57.753421 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-zlcr5" Oct 31 04:15:57.753699 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:57.753681 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-zlcr5" Oct 31 04:15:57.754311 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:15:57.754291 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-zlcr5" Oct 31 04:16:00.725151 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:16:00.725122 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-439989/premium-w1-439989-kserve-f9fc54cdf-sn2pp"] Oct 31 04:16:00.729004 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:16:00.728980 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-439989/premium-w1-439989-kserve-f9fc54cdf-sn2pp" Oct 31 04:16:00.748644 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:16:00.748608 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-439989/premium-w1-439989-kserve-f9fc54cdf-sn2pp"] Oct 31 04:16:00.757676 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:16:00.757654 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-439989\"/\"kube-root-ca.crt\"" Oct 31 04:16:00.757935 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:16:00.757917 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-439989\"/\"openshift-service-ca.crt\"" Oct 31 04:16:00.759987 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:16:00.759962 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-439989\"/\"premium-w1-439989-kserve-self-signed-certs\"" Oct 31 04:16:00.760092 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:16:00.759999 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-439989\"/\"default-dockercfg-bxjc6\"" Oct 31 04:16:00.827015 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:16:00.826990 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17b2f181-2e66-4ce2-95b0-cb213f11a0e7-model-cache\") pod \"premium-w1-439989-kserve-f9fc54cdf-sn2pp\" (UID: \"17b2f181-2e66-4ce2-95b0-cb213f11a0e7\") " pod="e2e-models-e2e-worker-w1-439989/premium-w1-439989-kserve-f9fc54cdf-sn2pp" Oct 31 04:16:00.827130 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:16:00.827098 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17b2f181-2e66-4ce2-95b0-cb213f11a0e7-dshm\") pod \"premium-w1-439989-kserve-f9fc54cdf-sn2pp\" (UID: \"17b2f181-2e66-4ce2-95b0-cb213f11a0e7\") " pod="e2e-models-e2e-worker-w1-439989/premium-w1-439989-kserve-f9fc54cdf-sn2pp" Oct 31 04:16:00.827171 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:16:00.827148 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17b2f181-2e66-4ce2-95b0-cb213f11a0e7-home\") pod \"premium-w1-439989-kserve-f9fc54cdf-sn2pp\" (UID: \"17b2f181-2e66-4ce2-95b0-cb213f11a0e7\") " pod="e2e-models-e2e-worker-w1-439989/premium-w1-439989-kserve-f9fc54cdf-sn2pp" Oct 31 04:16:00.827171 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:16:00.827165 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffw28\" (UniqueName: \"kubernetes.io/projected/17b2f181-2e66-4ce2-95b0-cb213f11a0e7-kube-api-access-ffw28\") pod \"premium-w1-439989-kserve-f9fc54cdf-sn2pp\" (UID: \"17b2f181-2e66-4ce2-95b0-cb213f11a0e7\") " pod="e2e-models-e2e-worker-w1-439989/premium-w1-439989-kserve-f9fc54cdf-sn2pp" Oct 31 04:16:00.827235 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:16:00.827213 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17b2f181-2e66-4ce2-95b0-cb213f11a0e7-tmp-dir\") pod \"premium-w1-439989-kserve-f9fc54cdf-sn2pp\" (UID: \"17b2f181-2e66-4ce2-95b0-cb213f11a0e7\") " pod="e2e-models-e2e-worker-w1-439989/premium-w1-439989-kserve-f9fc54cdf-sn2pp" Oct 31 04:16:00.827341 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:16:00.827307 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17b2f181-2e66-4ce2-95b0-cb213f11a0e7-tls-certs\") pod \"premium-w1-439989-kserve-f9fc54cdf-sn2pp\" (UID: \"17b2f181-2e66-4ce2-95b0-cb213f11a0e7\") " pod="e2e-models-e2e-worker-w1-439989/premium-w1-439989-kserve-f9fc54cdf-sn2pp" Oct 31 04:16:00.928869 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:16:00.928839 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17b2f181-2e66-4ce2-95b0-cb213f11a0e7-home\") pod \"premium-w1-439989-kserve-f9fc54cdf-sn2pp\" (UID: \"17b2f181-2e66-4ce2-95b0-cb213f11a0e7\") " pod="e2e-models-e2e-worker-w1-439989/premium-w1-439989-kserve-f9fc54cdf-sn2pp" Oct 31 04:16:00.928869 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:16:00.928870 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffw28\" (UniqueName: \"kubernetes.io/projected/17b2f181-2e66-4ce2-95b0-cb213f11a0e7-kube-api-access-ffw28\") pod \"premium-w1-439989-kserve-f9fc54cdf-sn2pp\" (UID: \"17b2f181-2e66-4ce2-95b0-cb213f11a0e7\") " pod="e2e-models-e2e-worker-w1-439989/premium-w1-439989-kserve-f9fc54cdf-sn2pp" Oct 31 04:16:00.929035 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:16:00.928919 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17b2f181-2e66-4ce2-95b0-cb213f11a0e7-tmp-dir\") pod \"premium-w1-439989-kserve-f9fc54cdf-sn2pp\" (UID: \"17b2f181-2e66-4ce2-95b0-cb213f11a0e7\") " pod="e2e-models-e2e-worker-w1-439989/premium-w1-439989-kserve-f9fc54cdf-sn2pp" Oct 31 04:16:00.929035 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:16:00.928947 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17b2f181-2e66-4ce2-95b0-cb213f11a0e7-tls-certs\") pod \"premium-w1-439989-kserve-f9fc54cdf-sn2pp\" (UID: \"17b2f181-2e66-4ce2-95b0-cb213f11a0e7\") " pod="e2e-models-e2e-worker-w1-439989/premium-w1-439989-kserve-f9fc54cdf-sn2pp" Oct 31 04:16:00.929035 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:16:00.929020 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17b2f181-2e66-4ce2-95b0-cb213f11a0e7-model-cache\") pod \"premium-w1-439989-kserve-f9fc54cdf-sn2pp\" (UID: \"17b2f181-2e66-4ce2-95b0-cb213f11a0e7\") " pod="e2e-models-e2e-worker-w1-439989/premium-w1-439989-kserve-f9fc54cdf-sn2pp" Oct 31 04:16:00.929187 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:16:00.929098 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17b2f181-2e66-4ce2-95b0-cb213f11a0e7-dshm\") pod \"premium-w1-439989-kserve-f9fc54cdf-sn2pp\" (UID: \"17b2f181-2e66-4ce2-95b0-cb213f11a0e7\") " pod="e2e-models-e2e-worker-w1-439989/premium-w1-439989-kserve-f9fc54cdf-sn2pp" Oct 31 04:16:00.929406 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:16:00.929380 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17b2f181-2e66-4ce2-95b0-cb213f11a0e7-home\") pod \"premium-w1-439989-kserve-f9fc54cdf-sn2pp\" (UID: \"17b2f181-2e66-4ce2-95b0-cb213f11a0e7\") " pod="e2e-models-e2e-worker-w1-439989/premium-w1-439989-kserve-f9fc54cdf-sn2pp" Oct 31 04:16:00.929406 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:16:00.929400 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17b2f181-2e66-4ce2-95b0-cb213f11a0e7-tmp-dir\") pod \"premium-w1-439989-kserve-f9fc54cdf-sn2pp\" (UID: \"17b2f181-2e66-4ce2-95b0-cb213f11a0e7\") " pod="e2e-models-e2e-worker-w1-439989/premium-w1-439989-kserve-f9fc54cdf-sn2pp" Oct 31 04:16:00.929531 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:16:00.929419 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17b2f181-2e66-4ce2-95b0-cb213f11a0e7-model-cache\") pod \"premium-w1-439989-kserve-f9fc54cdf-sn2pp\" (UID: \"17b2f181-2e66-4ce2-95b0-cb213f11a0e7\") " pod="e2e-models-e2e-worker-w1-439989/premium-w1-439989-kserve-f9fc54cdf-sn2pp" Oct 31 04:16:00.931273 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:16:00.931254 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17b2f181-2e66-4ce2-95b0-cb213f11a0e7-dshm\") pod \"premium-w1-439989-kserve-f9fc54cdf-sn2pp\" (UID: \"17b2f181-2e66-4ce2-95b0-cb213f11a0e7\") " pod="e2e-models-e2e-worker-w1-439989/premium-w1-439989-kserve-f9fc54cdf-sn2pp" Oct 31 04:16:00.931603 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:16:00.931585 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17b2f181-2e66-4ce2-95b0-cb213f11a0e7-tls-certs\") pod \"premium-w1-439989-kserve-f9fc54cdf-sn2pp\" (UID: \"17b2f181-2e66-4ce2-95b0-cb213f11a0e7\") " pod="e2e-models-e2e-worker-w1-439989/premium-w1-439989-kserve-f9fc54cdf-sn2pp" Oct 31 04:16:00.957547 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:16:00.957483 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffw28\" (UniqueName: \"kubernetes.io/projected/17b2f181-2e66-4ce2-95b0-cb213f11a0e7-kube-api-access-ffw28\") pod \"premium-w1-439989-kserve-f9fc54cdf-sn2pp\" (UID: \"17b2f181-2e66-4ce2-95b0-cb213f11a0e7\") " pod="e2e-models-e2e-worker-w1-439989/premium-w1-439989-kserve-f9fc54cdf-sn2pp" Oct 31 04:16:01.043607 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:16:01.043514 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-439989/premium-w1-439989-kserve-f9fc54cdf-sn2pp" Oct 31 04:16:01.338267 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:16:01.338228 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod17b2f181_2e66_4ce2_95b0_cb213f11a0e7.slice/crio-18ac60be655227c8c23cafcf62e75c9870a6bfab5ba8e5302dca2181928c9797 WatchSource:0}: Error finding container 18ac60be655227c8c23cafcf62e75c9870a6bfab5ba8e5302dca2181928c9797: Status 404 returned error can't find the container with id 18ac60be655227c8c23cafcf62e75c9870a6bfab5ba8e5302dca2181928c9797 Oct 31 04:16:01.338556 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:16:01.338537 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-439989/premium-w1-439989-kserve-f9fc54cdf-sn2pp"] Oct 31 04:16:01.721974 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:16:01.721934 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-439989/premium-w1-439989-kserve-f9fc54cdf-sn2pp" event={"ID":"17b2f181-2e66-4ce2-95b0-cb213f11a0e7","Type":"ContainerStarted","Data":"dcc1b1ea0c718cc4cc60686b4e335294d9a3a0518243b36a4576ad0ce4c0b538"} Oct 31 04:16:01.722155 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:16:01.721985 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-439989/premium-w1-439989-kserve-f9fc54cdf-sn2pp" event={"ID":"17b2f181-2e66-4ce2-95b0-cb213f11a0e7","Type":"ContainerStarted","Data":"18ac60be655227c8c23cafcf62e75c9870a6bfab5ba8e5302dca2181928c9797"} Oct 31 04:16:01.764284 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:16:01.764240 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-439989/premium-w1-439989-kserve-f9fc54cdf-sn2pp" podStartSLOduration=1.764227143 podStartE2EDuration="1.764227143s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:16:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:16:01.763765912 +0000 UTC m=+709.478100528" watchObservedRunningTime="2026-10-31 04:16:01.764227143 +0000 UTC m=+709.478561759" Oct 31 04:16:02.044257 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:16:02.044187 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-439989/premium-w1-439989-kserve-f9fc54cdf-sn2pp" Oct 31 04:16:08.639618 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:16:08.639580 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 04:16:10.997926 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:16:10.997888 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 04:16:11.044340 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:16:11.044301 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-439989/premium-w1-439989-kserve-f9fc54cdf-sn2pp" Oct 31 04:16:11.056809 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:16:11.056784 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-439989/premium-w1-439989-kserve-f9fc54cdf-sn2pp" Oct 31 04:16:11.784023 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:16:11.783996 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-439989/premium-w1-439989-kserve-f9fc54cdf-sn2pp" Oct 31 04:16:22.767641 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:16:22.767607 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 04:16:40.997411 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:16:40.997373 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 04:16:41.237342 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:16:41.237301 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 04:17:04.896639 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:04.896608 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-bf555d5c7-mjl5l"] Oct 31 04:17:04.900566 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:04.900545 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-bf555d5c7-mjl5l" Oct 31 04:17:04.917661 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:04.917640 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-bf555d5c7-mjl5l"] Oct 31 04:17:04.930148 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:04.930108 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Oct 31 04:17:04.930261 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:04.930109 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Oct 31 04:17:04.930261 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:04.930124 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Oct 31 04:17:04.930261 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:04.930174 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Oct 31 04:17:04.931801 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:04.931777 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-cdd9c78cff58\"" Oct 31 04:17:04.931801 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:04.931777 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-4f55842b7e8e\"" Oct 31 04:17:04.931907 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:04.931841 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-5qlm5\"" Oct 31 04:17:05.006386 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:04.994000 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wp2zs\" (UniqueName: \"kubernetes.io/projected/64ffa199-1f29-4a73-99ea-36b861f64b21-kube-api-access-wp2zs\") pod \"payload-processing-external-model-bf555d5c7-mjl5l\" (UID: \"64ffa199-1f29-4a73-99ea-36b861f64b21\") " pod="models-as-a-service/payload-processing-external-model-bf555d5c7-mjl5l" Oct 31 04:17:05.006706 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:05.006678 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64ffa199-1f29-4a73-99ea-36b861f64b21-plugins-config-volume\") pod \"payload-processing-external-model-bf555d5c7-mjl5l\" (UID: \"64ffa199-1f29-4a73-99ea-36b861f64b21\") " pod="models-as-a-service/payload-processing-external-model-bf555d5c7-mjl5l" Oct 31 04:17:05.006832 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:05.006717 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/64ffa199-1f29-4a73-99ea-36b861f64b21-routing-overlay-volume\") pod \"payload-processing-external-model-bf555d5c7-mjl5l\" (UID: \"64ffa199-1f29-4a73-99ea-36b861f64b21\") " pod="models-as-a-service/payload-processing-external-model-bf555d5c7-mjl5l" Oct 31 04:17:05.006832 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:05.006788 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/64ffa199-1f29-4a73-99ea-36b861f64b21-provider-credentials\") pod \"payload-processing-external-model-bf555d5c7-mjl5l\" (UID: \"64ffa199-1f29-4a73-99ea-36b861f64b21\") " pod="models-as-a-service/payload-processing-external-model-bf555d5c7-mjl5l" Oct 31 04:17:05.107938 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:05.107895 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wp2zs\" (UniqueName: \"kubernetes.io/projected/64ffa199-1f29-4a73-99ea-36b861f64b21-kube-api-access-wp2zs\") pod \"payload-processing-external-model-bf555d5c7-mjl5l\" (UID: \"64ffa199-1f29-4a73-99ea-36b861f64b21\") " pod="models-as-a-service/payload-processing-external-model-bf555d5c7-mjl5l" Oct 31 04:17:05.108153 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:05.107960 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64ffa199-1f29-4a73-99ea-36b861f64b21-plugins-config-volume\") pod \"payload-processing-external-model-bf555d5c7-mjl5l\" (UID: \"64ffa199-1f29-4a73-99ea-36b861f64b21\") " pod="models-as-a-service/payload-processing-external-model-bf555d5c7-mjl5l" Oct 31 04:17:05.108153 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:05.108007 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/64ffa199-1f29-4a73-99ea-36b861f64b21-routing-overlay-volume\") pod \"payload-processing-external-model-bf555d5c7-mjl5l\" (UID: \"64ffa199-1f29-4a73-99ea-36b861f64b21\") " pod="models-as-a-service/payload-processing-external-model-bf555d5c7-mjl5l" Oct 31 04:17:05.108153 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:05.108061 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/64ffa199-1f29-4a73-99ea-36b861f64b21-provider-credentials\") pod \"payload-processing-external-model-bf555d5c7-mjl5l\" (UID: \"64ffa199-1f29-4a73-99ea-36b861f64b21\") " pod="models-as-a-service/payload-processing-external-model-bf555d5c7-mjl5l" Oct 31 04:17:05.108766 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:05.108745 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64ffa199-1f29-4a73-99ea-36b861f64b21-plugins-config-volume\") pod \"payload-processing-external-model-bf555d5c7-mjl5l\" (UID: \"64ffa199-1f29-4a73-99ea-36b861f64b21\") " pod="models-as-a-service/payload-processing-external-model-bf555d5c7-mjl5l" Oct 31 04:17:05.108890 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:05.108814 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/64ffa199-1f29-4a73-99ea-36b861f64b21-routing-overlay-volume\") pod \"payload-processing-external-model-bf555d5c7-mjl5l\" (UID: \"64ffa199-1f29-4a73-99ea-36b861f64b21\") " pod="models-as-a-service/payload-processing-external-model-bf555d5c7-mjl5l" Oct 31 04:17:05.110560 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:05.110538 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/64ffa199-1f29-4a73-99ea-36b861f64b21-provider-credentials\") pod \"payload-processing-external-model-bf555d5c7-mjl5l\" (UID: \"64ffa199-1f29-4a73-99ea-36b861f64b21\") " pod="models-as-a-service/payload-processing-external-model-bf555d5c7-mjl5l" Oct 31 04:17:05.133438 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:05.133410 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wp2zs\" (UniqueName: \"kubernetes.io/projected/64ffa199-1f29-4a73-99ea-36b861f64b21-kube-api-access-wp2zs\") pod \"payload-processing-external-model-bf555d5c7-mjl5l\" (UID: \"64ffa199-1f29-4a73-99ea-36b861f64b21\") " pod="models-as-a-service/payload-processing-external-model-bf555d5c7-mjl5l" Oct 31 04:17:05.213474 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:05.213390 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-bf555d5c7-mjl5l" Oct 31 04:17:05.513731 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:05.513693 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-bf555d5c7-mjl5l"] Oct 31 04:17:05.514722 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:17:05.514698 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64ffa199_1f29_4a73_99ea_36b861f64b21.slice/crio-d86d6b522fd9d718fc5baf6b078dacd60a703689e8b38094dd6ba07cb1e344f3 WatchSource:0}: Error finding container d86d6b522fd9d718fc5baf6b078dacd60a703689e8b38094dd6ba07cb1e344f3: Status 404 returned error can't find the container with id d86d6b522fd9d718fc5baf6b078dacd60a703689e8b38094dd6ba07cb1e344f3 Oct 31 04:17:06.087936 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:06.087891 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-bf555d5c7-mjl5l" event={"ID":"64ffa199-1f29-4a73-99ea-36b861f64b21","Type":"ContainerStarted","Data":"72ea032d233e239311011e78df190154a9c1599bee128f884e5c6a7fdad5e0ae"} Oct 31 04:17:06.088528 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:06.087967 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-bf555d5c7-mjl5l" event={"ID":"64ffa199-1f29-4a73-99ea-36b861f64b21","Type":"ContainerStarted","Data":"d86d6b522fd9d718fc5baf6b078dacd60a703689e8b38094dd6ba07cb1e344f3"} Oct 31 04:17:06.088610 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:06.088523 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-bf555d5c7-mjl5l" Oct 31 04:17:06.135611 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:06.135560 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-bf555d5c7-mjl5l" podStartSLOduration=2.135546086 podStartE2EDuration="2.135546086s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:17:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:17:06.133525927 +0000 UTC m=+773.847860544" watchObservedRunningTime="2026-10-31 04:17:06.135546086 +0000 UTC m=+773.849880694" Oct 31 04:17:10.442451 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:10.442423 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 04:17:10.999038 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:17:10.999001 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 04:17:17.100078 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:17.100050 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-bf555d5c7-mjl5l" Oct 31 04:17:22.951861 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:22.951826 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-09a20a/e2e-conflict-a-ba60a444-kserve-78b9f5789c-kjs4g"] Oct 31 04:17:22.956970 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:22.956946 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-09a20a/e2e-conflict-a-ba60a444-kserve-78b9f5789c-kjs4g" Oct 31 04:17:22.975499 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:22.975470 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-09a20a/e2e-conflict-a-ba60a444-kserve-78b9f5789c-kjs4g"] Oct 31 04:17:22.985912 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:22.985889 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-09a20a\"/\"e2e-conflict-a-ba60a444-kserve-self-signed-certs\"" Oct 31 04:17:23.129451 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:23.129422 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/79160b8c-f9b5-4cfa-8d3f-31da9a42dbe5-home\") pod \"e2e-conflict-a-ba60a444-kserve-78b9f5789c-kjs4g\" (UID: \"79160b8c-f9b5-4cfa-8d3f-31da9a42dbe5\") " pod="e2e-models-e2e-worker-w2-09a20a/e2e-conflict-a-ba60a444-kserve-78b9f5789c-kjs4g" Oct 31 04:17:23.129621 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:23.129485 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/79160b8c-f9b5-4cfa-8d3f-31da9a42dbe5-dshm\") pod \"e2e-conflict-a-ba60a444-kserve-78b9f5789c-kjs4g\" (UID: \"79160b8c-f9b5-4cfa-8d3f-31da9a42dbe5\") " pod="e2e-models-e2e-worker-w2-09a20a/e2e-conflict-a-ba60a444-kserve-78b9f5789c-kjs4g" Oct 31 04:17:23.129621 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:23.129576 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79160b8c-f9b5-4cfa-8d3f-31da9a42dbe5-tmp-dir\") pod \"e2e-conflict-a-ba60a444-kserve-78b9f5789c-kjs4g\" (UID: \"79160b8c-f9b5-4cfa-8d3f-31da9a42dbe5\") " pod="e2e-models-e2e-worker-w2-09a20a/e2e-conflict-a-ba60a444-kserve-78b9f5789c-kjs4g" Oct 31 04:17:23.129724 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:23.129634 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnnsr\" (UniqueName: \"kubernetes.io/projected/79160b8c-f9b5-4cfa-8d3f-31da9a42dbe5-kube-api-access-tnnsr\") pod \"e2e-conflict-a-ba60a444-kserve-78b9f5789c-kjs4g\" (UID: \"79160b8c-f9b5-4cfa-8d3f-31da9a42dbe5\") " pod="e2e-models-e2e-worker-w2-09a20a/e2e-conflict-a-ba60a444-kserve-78b9f5789c-kjs4g" Oct 31 04:17:23.129724 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:23.129705 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79160b8c-f9b5-4cfa-8d3f-31da9a42dbe5-model-cache\") pod \"e2e-conflict-a-ba60a444-kserve-78b9f5789c-kjs4g\" (UID: \"79160b8c-f9b5-4cfa-8d3f-31da9a42dbe5\") " pod="e2e-models-e2e-worker-w2-09a20a/e2e-conflict-a-ba60a444-kserve-78b9f5789c-kjs4g" Oct 31 04:17:23.129806 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:23.129796 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79160b8c-f9b5-4cfa-8d3f-31da9a42dbe5-tls-certs\") pod \"e2e-conflict-a-ba60a444-kserve-78b9f5789c-kjs4g\" (UID: \"79160b8c-f9b5-4cfa-8d3f-31da9a42dbe5\") " pod="e2e-models-e2e-worker-w2-09a20a/e2e-conflict-a-ba60a444-kserve-78b9f5789c-kjs4g" Oct 31 04:17:23.212674 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:23.212570 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-d7frf"] Oct 31 04:17:23.219523 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:23.219480 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-d7frf"] Oct 31 04:17:23.219677 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:23.219576 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-d7frf" Oct 31 04:17:23.230731 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:23.230705 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79160b8c-f9b5-4cfa-8d3f-31da9a42dbe5-tls-certs\") pod \"e2e-conflict-a-ba60a444-kserve-78b9f5789c-kjs4g\" (UID: \"79160b8c-f9b5-4cfa-8d3f-31da9a42dbe5\") " pod="e2e-models-e2e-worker-w2-09a20a/e2e-conflict-a-ba60a444-kserve-78b9f5789c-kjs4g" Oct 31 04:17:23.230866 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:23.230746 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/79160b8c-f9b5-4cfa-8d3f-31da9a42dbe5-home\") pod \"e2e-conflict-a-ba60a444-kserve-78b9f5789c-kjs4g\" (UID: \"79160b8c-f9b5-4cfa-8d3f-31da9a42dbe5\") " pod="e2e-models-e2e-worker-w2-09a20a/e2e-conflict-a-ba60a444-kserve-78b9f5789c-kjs4g" Oct 31 04:17:23.230866 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:23.230797 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/79160b8c-f9b5-4cfa-8d3f-31da9a42dbe5-dshm\") pod \"e2e-conflict-a-ba60a444-kserve-78b9f5789c-kjs4g\" (UID: \"79160b8c-f9b5-4cfa-8d3f-31da9a42dbe5\") " pod="e2e-models-e2e-worker-w2-09a20a/e2e-conflict-a-ba60a444-kserve-78b9f5789c-kjs4g" Oct 31 04:17:23.230866 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:23.230829 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79160b8c-f9b5-4cfa-8d3f-31da9a42dbe5-tmp-dir\") pod \"e2e-conflict-a-ba60a444-kserve-78b9f5789c-kjs4g\" (UID: \"79160b8c-f9b5-4cfa-8d3f-31da9a42dbe5\") " pod="e2e-models-e2e-worker-w2-09a20a/e2e-conflict-a-ba60a444-kserve-78b9f5789c-kjs4g" Oct 31 04:17:23.231046 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:23.230873 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnnsr\" (UniqueName: \"kubernetes.io/projected/79160b8c-f9b5-4cfa-8d3f-31da9a42dbe5-kube-api-access-tnnsr\") pod \"e2e-conflict-a-ba60a444-kserve-78b9f5789c-kjs4g\" (UID: \"79160b8c-f9b5-4cfa-8d3f-31da9a42dbe5\") " pod="e2e-models-e2e-worker-w2-09a20a/e2e-conflict-a-ba60a444-kserve-78b9f5789c-kjs4g" Oct 31 04:17:23.231046 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:23.230890 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79160b8c-f9b5-4cfa-8d3f-31da9a42dbe5-model-cache\") pod \"e2e-conflict-a-ba60a444-kserve-78b9f5789c-kjs4g\" (UID: \"79160b8c-f9b5-4cfa-8d3f-31da9a42dbe5\") " pod="e2e-models-e2e-worker-w2-09a20a/e2e-conflict-a-ba60a444-kserve-78b9f5789c-kjs4g" Oct 31 04:17:23.231319 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:23.231295 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79160b8c-f9b5-4cfa-8d3f-31da9a42dbe5-tmp-dir\") pod \"e2e-conflict-a-ba60a444-kserve-78b9f5789c-kjs4g\" (UID: \"79160b8c-f9b5-4cfa-8d3f-31da9a42dbe5\") " pod="e2e-models-e2e-worker-w2-09a20a/e2e-conflict-a-ba60a444-kserve-78b9f5789c-kjs4g" Oct 31 04:17:23.231440 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:23.231324 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/79160b8c-f9b5-4cfa-8d3f-31da9a42dbe5-home\") pod \"e2e-conflict-a-ba60a444-kserve-78b9f5789c-kjs4g\" (UID: \"79160b8c-f9b5-4cfa-8d3f-31da9a42dbe5\") " pod="e2e-models-e2e-worker-w2-09a20a/e2e-conflict-a-ba60a444-kserve-78b9f5789c-kjs4g" Oct 31 04:17:23.231440 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:23.231387 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79160b8c-f9b5-4cfa-8d3f-31da9a42dbe5-model-cache\") pod \"e2e-conflict-a-ba60a444-kserve-78b9f5789c-kjs4g\" (UID: \"79160b8c-f9b5-4cfa-8d3f-31da9a42dbe5\") " pod="e2e-models-e2e-worker-w2-09a20a/e2e-conflict-a-ba60a444-kserve-78b9f5789c-kjs4g" Oct 31 04:17:23.233172 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:23.233153 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79160b8c-f9b5-4cfa-8d3f-31da9a42dbe5-tls-certs\") pod \"e2e-conflict-a-ba60a444-kserve-78b9f5789c-kjs4g\" (UID: \"79160b8c-f9b5-4cfa-8d3f-31da9a42dbe5\") " pod="e2e-models-e2e-worker-w2-09a20a/e2e-conflict-a-ba60a444-kserve-78b9f5789c-kjs4g" Oct 31 04:17:23.233640 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:23.233625 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/79160b8c-f9b5-4cfa-8d3f-31da9a42dbe5-dshm\") pod \"e2e-conflict-a-ba60a444-kserve-78b9f5789c-kjs4g\" (UID: \"79160b8c-f9b5-4cfa-8d3f-31da9a42dbe5\") " pod="e2e-models-e2e-worker-w2-09a20a/e2e-conflict-a-ba60a444-kserve-78b9f5789c-kjs4g" Oct 31 04:17:23.238057 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:23.238017 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-4c2387-openshift-default-dockercfg-x4j6w\"" Oct 31 04:17:23.256266 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:23.256247 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnnsr\" (UniqueName: \"kubernetes.io/projected/79160b8c-f9b5-4cfa-8d3f-31da9a42dbe5-kube-api-access-tnnsr\") pod \"e2e-conflict-a-ba60a444-kserve-78b9f5789c-kjs4g\" (UID: \"79160b8c-f9b5-4cfa-8d3f-31da9a42dbe5\") " pod="e2e-models-e2e-worker-w2-09a20a/e2e-conflict-a-ba60a444-kserve-78b9f5789c-kjs4g" Oct 31 04:17:23.273057 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:23.273033 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-09a20a/e2e-conflict-a-ba60a444-kserve-78b9f5789c-kjs4g" Oct 31 04:17:23.333046 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:23.333014 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfcd34ac-2ed4-4f1e-bd06-9cca74916956-istiod-ca-cert\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-d7frf\" (UID: \"dfcd34ac-2ed4-4f1e-bd06-9cca74916956\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-d7frf" Oct 31 04:17:23.333187 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:23.333064 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ms878\" (UniqueName: \"kubernetes.io/projected/dfcd34ac-2ed4-4f1e-bd06-9cca74916956-kube-api-access-ms878\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-d7frf\" (UID: \"dfcd34ac-2ed4-4f1e-bd06-9cca74916956\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-d7frf" Oct 31 04:17:23.333187 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:23.333153 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfcd34ac-2ed4-4f1e-bd06-9cca74916956-credential-socket\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-d7frf\" (UID: \"dfcd34ac-2ed4-4f1e-bd06-9cca74916956\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-d7frf" Oct 31 04:17:23.333429 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:23.333271 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfcd34ac-2ed4-4f1e-bd06-9cca74916956-istio-data\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-d7frf\" (UID: \"dfcd34ac-2ed4-4f1e-bd06-9cca74916956\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-d7frf" Oct 31 04:17:23.333429 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:23.333353 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfcd34ac-2ed4-4f1e-bd06-9cca74916956-istio-podinfo\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-d7frf\" (UID: \"dfcd34ac-2ed4-4f1e-bd06-9cca74916956\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-d7frf" Oct 31 04:17:23.333545 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:23.333440 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfcd34ac-2ed4-4f1e-bd06-9cca74916956-istio-token\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-d7frf\" (UID: \"dfcd34ac-2ed4-4f1e-bd06-9cca74916956\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-d7frf" Oct 31 04:17:23.333545 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:23.333498 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfcd34ac-2ed4-4f1e-bd06-9cca74916956-workload-certs\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-d7frf\" (UID: \"dfcd34ac-2ed4-4f1e-bd06-9cca74916956\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-d7frf" Oct 31 04:17:23.333545 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:23.333538 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfcd34ac-2ed4-4f1e-bd06-9cca74916956-istio-envoy\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-d7frf\" (UID: \"dfcd34ac-2ed4-4f1e-bd06-9cca74916956\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-d7frf" Oct 31 04:17:23.333713 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:23.333570 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfcd34ac-2ed4-4f1e-bd06-9cca74916956-workload-socket\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-d7frf\" (UID: \"dfcd34ac-2ed4-4f1e-bd06-9cca74916956\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-d7frf" Oct 31 04:17:23.435065 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:23.435027 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfcd34ac-2ed4-4f1e-bd06-9cca74916956-credential-socket\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-d7frf\" (UID: \"dfcd34ac-2ed4-4f1e-bd06-9cca74916956\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-d7frf" Oct 31 04:17:23.435241 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:23.435123 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfcd34ac-2ed4-4f1e-bd06-9cca74916956-istio-data\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-d7frf\" (UID: \"dfcd34ac-2ed4-4f1e-bd06-9cca74916956\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-d7frf" Oct 31 04:17:23.435241 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:23.435151 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfcd34ac-2ed4-4f1e-bd06-9cca74916956-istio-podinfo\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-d7frf\" (UID: \"dfcd34ac-2ed4-4f1e-bd06-9cca74916956\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-d7frf" Oct 31 04:17:23.435241 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:23.435216 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfcd34ac-2ed4-4f1e-bd06-9cca74916956-istio-token\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-d7frf\" (UID: \"dfcd34ac-2ed4-4f1e-bd06-9cca74916956\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-d7frf" Oct 31 04:17:23.435448 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:23.435269 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfcd34ac-2ed4-4f1e-bd06-9cca74916956-workload-certs\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-d7frf\" (UID: \"dfcd34ac-2ed4-4f1e-bd06-9cca74916956\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-d7frf" Oct 31 04:17:23.435448 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:23.435298 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfcd34ac-2ed4-4f1e-bd06-9cca74916956-istio-envoy\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-d7frf\" (UID: \"dfcd34ac-2ed4-4f1e-bd06-9cca74916956\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-d7frf" Oct 31 04:17:23.435448 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:23.435408 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfcd34ac-2ed4-4f1e-bd06-9cca74916956-workload-socket\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-d7frf\" (UID: \"dfcd34ac-2ed4-4f1e-bd06-9cca74916956\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-d7frf" Oct 31 04:17:23.435611 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:23.435496 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfcd34ac-2ed4-4f1e-bd06-9cca74916956-istiod-ca-cert\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-d7frf\" (UID: \"dfcd34ac-2ed4-4f1e-bd06-9cca74916956\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-d7frf" Oct 31 04:17:23.435611 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:23.435531 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ms878\" (UniqueName: \"kubernetes.io/projected/dfcd34ac-2ed4-4f1e-bd06-9cca74916956-kube-api-access-ms878\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-d7frf\" (UID: \"dfcd34ac-2ed4-4f1e-bd06-9cca74916956\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-d7frf" Oct 31 04:17:23.435611 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:23.435540 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfcd34ac-2ed4-4f1e-bd06-9cca74916956-credential-socket\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-d7frf\" (UID: \"dfcd34ac-2ed4-4f1e-bd06-9cca74916956\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-d7frf" Oct 31 04:17:23.435611 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:23.435575 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfcd34ac-2ed4-4f1e-bd06-9cca74916956-istio-data\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-d7frf\" (UID: \"dfcd34ac-2ed4-4f1e-bd06-9cca74916956\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-d7frf" Oct 31 04:17:23.435780 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:23.435690 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfcd34ac-2ed4-4f1e-bd06-9cca74916956-workload-certs\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-d7frf\" (UID: \"dfcd34ac-2ed4-4f1e-bd06-9cca74916956\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-d7frf" Oct 31 04:17:23.435858 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:23.435836 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfcd34ac-2ed4-4f1e-bd06-9cca74916956-workload-socket\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-d7frf\" (UID: \"dfcd34ac-2ed4-4f1e-bd06-9cca74916956\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-d7frf" Oct 31 04:17:23.436445 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:23.436422 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfcd34ac-2ed4-4f1e-bd06-9cca74916956-istiod-ca-cert\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-d7frf\" (UID: \"dfcd34ac-2ed4-4f1e-bd06-9cca74916956\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-d7frf" Oct 31 04:17:23.438153 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:23.438133 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfcd34ac-2ed4-4f1e-bd06-9cca74916956-istio-envoy\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-d7frf\" (UID: \"dfcd34ac-2ed4-4f1e-bd06-9cca74916956\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-d7frf" Oct 31 04:17:23.438411 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:23.438392 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfcd34ac-2ed4-4f1e-bd06-9cca74916956-istio-podinfo\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-d7frf\" (UID: \"dfcd34ac-2ed4-4f1e-bd06-9cca74916956\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-d7frf" Oct 31 04:17:23.460760 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:23.460732 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfcd34ac-2ed4-4f1e-bd06-9cca74916956-istio-token\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-d7frf\" (UID: \"dfcd34ac-2ed4-4f1e-bd06-9cca74916956\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-d7frf" Oct 31 04:17:23.460931 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:23.460913 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ms878\" (UniqueName: \"kubernetes.io/projected/dfcd34ac-2ed4-4f1e-bd06-9cca74916956-kube-api-access-ms878\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-d7frf\" (UID: \"dfcd34ac-2ed4-4f1e-bd06-9cca74916956\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-d7frf" Oct 31 04:17:23.532893 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:23.532840 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-d7frf" Oct 31 04:17:23.577957 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:17:23.577931 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod79160b8c_f9b5_4cfa_8d3f_31da9a42dbe5.slice/crio-e36a28e29189480055d87135423c3e786f1fbde01d8f1317735d5ed1feabb180 WatchSource:0}: Error finding container e36a28e29189480055d87135423c3e786f1fbde01d8f1317735d5ed1feabb180: Status 404 returned error can't find the container with id e36a28e29189480055d87135423c3e786f1fbde01d8f1317735d5ed1feabb180 Oct 31 04:17:23.579242 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:23.579217 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-09a20a/e2e-conflict-a-ba60a444-kserve-78b9f5789c-kjs4g"] Oct 31 04:17:23.846475 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:17:23.846448 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfcd34ac_2ed4_4f1e_bd06_9cca74916956.slice/crio-b0e21e852e20c9bf939dbb22b350e8b0cb978f75b9f23ee67bdc6d46fa04becb WatchSource:0}: Error finding container b0e21e852e20c9bf939dbb22b350e8b0cb978f75b9f23ee67bdc6d46fa04becb: Status 404 returned error can't find the container with id b0e21e852e20c9bf939dbb22b350e8b0cb978f75b9f23ee67bdc6d46fa04becb Oct 31 04:17:23.846975 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:23.846949 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-d7frf"] Oct 31 04:17:23.848846 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:23.848816 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:17:23.848931 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:23.848875 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:17:23.848931 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:23.848918 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:17:24.250348 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:24.250199 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-d7frf" event={"ID":"dfcd34ac-2ed4-4f1e-bd06-9cca74916956","Type":"ContainerStarted","Data":"3ca2c47646dda985888ec2e91896aceb1af3f7d1b0c94def3e80def15de0862a"} Oct 31 04:17:24.250348 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:24.250269 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-d7frf" event={"ID":"dfcd34ac-2ed4-4f1e-bd06-9cca74916956","Type":"ContainerStarted","Data":"b0e21e852e20c9bf939dbb22b350e8b0cb978f75b9f23ee67bdc6d46fa04becb"} Oct 31 04:17:24.256348 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:24.253137 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-09a20a/e2e-conflict-a-ba60a444-kserve-78b9f5789c-kjs4g" event={"ID":"79160b8c-f9b5-4cfa-8d3f-31da9a42dbe5","Type":"ContainerStarted","Data":"f34d596fa8ad272f83fbdfc9b50b50bcaa1be1cf2ce8973d78d5bd9c5abe57e9"} Oct 31 04:17:24.256348 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:24.253183 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-09a20a/e2e-conflict-a-ba60a444-kserve-78b9f5789c-kjs4g" event={"ID":"79160b8c-f9b5-4cfa-8d3f-31da9a42dbe5","Type":"ContainerStarted","Data":"e36a28e29189480055d87135423c3e786f1fbde01d8f1317735d5ed1feabb180"} Oct 31 04:17:24.283009 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:24.277442 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-09a20a/e2e-conflict-a-ba60a444-kserve-78b9f5789c-kjs4g" Oct 31 04:17:24.359413 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:24.358974 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-d7frf" podStartSLOduration=1.358954493 podStartE2EDuration="1.358954493s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:17:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:17:24.319633107 +0000 UTC m=+792.033967752" watchObservedRunningTime="2026-10-31 04:17:24.358954493 +0000 UTC m=+792.073289126" Oct 31 04:17:24.362166 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:24.362134 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wmkf5"] Oct 31 04:17:24.366190 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:24.366167 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wmkf5" Oct 31 04:17:24.371765 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:24.371708 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-09a20a/e2e-conflict-a-ba60a444-kserve-78b9f5789c-kjs4g" podStartSLOduration=2.371691521 podStartE2EDuration="2.371691521s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:17:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:17:24.36996065 +0000 UTC m=+792.084295290" watchObservedRunningTime="2026-10-31 04:17:24.371691521 +0000 UTC m=+792.086026138" Oct 31 04:17:24.396353 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:24.393401 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wmkf5"] Oct 31 04:17:24.470355 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:24.460954 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d135a318-172d-4408-8e38-ca0e26946381-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wmkf5\" (UID: \"d135a318-172d-4408-8e38-ca0e26946381\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wmkf5" Oct 31 04:17:24.470355 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:24.461065 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d135a318-172d-4408-8e38-ca0e26946381-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wmkf5\" (UID: \"d135a318-172d-4408-8e38-ca0e26946381\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wmkf5" Oct 31 04:17:24.470355 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:24.461119 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d135a318-172d-4408-8e38-ca0e26946381-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wmkf5\" (UID: \"d135a318-172d-4408-8e38-ca0e26946381\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wmkf5" Oct 31 04:17:24.470355 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:24.461183 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d135a318-172d-4408-8e38-ca0e26946381-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wmkf5\" (UID: \"d135a318-172d-4408-8e38-ca0e26946381\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wmkf5" Oct 31 04:17:24.470355 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:24.461222 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d135a318-172d-4408-8e38-ca0e26946381-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wmkf5\" (UID: \"d135a318-172d-4408-8e38-ca0e26946381\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wmkf5" Oct 31 04:17:24.470355 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:24.461258 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmdx4\" (UniqueName: \"kubernetes.io/projected/d135a318-172d-4408-8e38-ca0e26946381-kube-api-access-hmdx4\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wmkf5\" (UID: \"d135a318-172d-4408-8e38-ca0e26946381\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wmkf5" Oct 31 04:17:24.470355 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:24.461307 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d135a318-172d-4408-8e38-ca0e26946381-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wmkf5\" (UID: \"d135a318-172d-4408-8e38-ca0e26946381\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wmkf5" Oct 31 04:17:24.470355 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:24.461349 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d135a318-172d-4408-8e38-ca0e26946381-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wmkf5\" (UID: \"d135a318-172d-4408-8e38-ca0e26946381\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wmkf5" Oct 31 04:17:24.470355 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:24.461387 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d135a318-172d-4408-8e38-ca0e26946381-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wmkf5\" (UID: \"d135a318-172d-4408-8e38-ca0e26946381\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wmkf5" Oct 31 04:17:24.533095 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:24.533022 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-d7frf" Oct 31 04:17:24.534345 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:24.534304 4376 patch_prober.go:29] interesting pod/e2e-worker-w3-4c2387-openshift-default-5b68d44445-d7frf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Oct 31 04:17:24.534497 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:24.534367 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-d7frf" podUID="dfcd34ac-2ed4-4f1e-bd06-9cca74916956" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Oct 31 04:17:24.572984 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:24.572942 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d135a318-172d-4408-8e38-ca0e26946381-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wmkf5\" (UID: \"d135a318-172d-4408-8e38-ca0e26946381\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wmkf5" Oct 31 04:17:24.573187 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:24.573004 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d135a318-172d-4408-8e38-ca0e26946381-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wmkf5\" (UID: \"d135a318-172d-4408-8e38-ca0e26946381\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wmkf5" Oct 31 04:17:24.573187 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:24.573039 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmdx4\" (UniqueName: \"kubernetes.io/projected/d135a318-172d-4408-8e38-ca0e26946381-kube-api-access-hmdx4\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wmkf5\" (UID: \"d135a318-172d-4408-8e38-ca0e26946381\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wmkf5" Oct 31 04:17:24.573187 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:24.573084 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d135a318-172d-4408-8e38-ca0e26946381-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wmkf5\" (UID: \"d135a318-172d-4408-8e38-ca0e26946381\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wmkf5" Oct 31 04:17:24.573187 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:24.573109 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d135a318-172d-4408-8e38-ca0e26946381-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wmkf5\" (UID: \"d135a318-172d-4408-8e38-ca0e26946381\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wmkf5" Oct 31 04:17:24.573187 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:24.573143 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d135a318-172d-4408-8e38-ca0e26946381-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wmkf5\" (UID: \"d135a318-172d-4408-8e38-ca0e26946381\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wmkf5" Oct 31 04:17:24.573604 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:24.573195 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d135a318-172d-4408-8e38-ca0e26946381-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wmkf5\" (UID: \"d135a318-172d-4408-8e38-ca0e26946381\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wmkf5" Oct 31 04:17:24.575370 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:24.575319 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d135a318-172d-4408-8e38-ca0e26946381-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wmkf5\" (UID: \"d135a318-172d-4408-8e38-ca0e26946381\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wmkf5" Oct 31 04:17:24.576042 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:24.576017 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d135a318-172d-4408-8e38-ca0e26946381-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wmkf5\" (UID: \"d135a318-172d-4408-8e38-ca0e26946381\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wmkf5" Oct 31 04:17:24.576670 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:24.576632 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d135a318-172d-4408-8e38-ca0e26946381-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wmkf5\" (UID: \"d135a318-172d-4408-8e38-ca0e26946381\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wmkf5" Oct 31 04:17:24.576833 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:24.576742 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d135a318-172d-4408-8e38-ca0e26946381-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wmkf5\" (UID: \"d135a318-172d-4408-8e38-ca0e26946381\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wmkf5" Oct 31 04:17:24.576833 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:24.576817 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d135a318-172d-4408-8e38-ca0e26946381-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wmkf5\" (UID: \"d135a318-172d-4408-8e38-ca0e26946381\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wmkf5" Oct 31 04:17:24.577011 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:24.576739 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d135a318-172d-4408-8e38-ca0e26946381-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wmkf5\" (UID: \"d135a318-172d-4408-8e38-ca0e26946381\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wmkf5" Oct 31 04:17:24.577439 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:24.577416 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d135a318-172d-4408-8e38-ca0e26946381-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wmkf5\" (UID: \"d135a318-172d-4408-8e38-ca0e26946381\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wmkf5" Oct 31 04:17:24.580150 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:24.580126 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d135a318-172d-4408-8e38-ca0e26946381-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wmkf5\" (UID: \"d135a318-172d-4408-8e38-ca0e26946381\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wmkf5" Oct 31 04:17:24.582828 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:24.582805 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d135a318-172d-4408-8e38-ca0e26946381-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wmkf5\" (UID: \"d135a318-172d-4408-8e38-ca0e26946381\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wmkf5" Oct 31 04:17:24.599768 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:24.599746 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmdx4\" (UniqueName: \"kubernetes.io/projected/d135a318-172d-4408-8e38-ca0e26946381-kube-api-access-hmdx4\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wmkf5\" (UID: \"d135a318-172d-4408-8e38-ca0e26946381\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wmkf5" Oct 31 04:17:24.601664 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:24.601633 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d135a318-172d-4408-8e38-ca0e26946381-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wmkf5\" (UID: \"d135a318-172d-4408-8e38-ca0e26946381\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wmkf5" Oct 31 04:17:24.682372 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:24.682337 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wmkf5" Oct 31 04:17:25.016749 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:17:25.016717 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd135a318_172d_4408_8e38_ca0e26946381.slice/crio-55f4f034380f1d2019eb2d1a4c1d75db3aec6a69bbf3c7093c8498e0c71bf8c1 WatchSource:0}: Error finding container 55f4f034380f1d2019eb2d1a4c1d75db3aec6a69bbf3c7093c8498e0c71bf8c1: Status 404 returned error can't find the container with id 55f4f034380f1d2019eb2d1a4c1d75db3aec6a69bbf3c7093c8498e0c71bf8c1 Oct 31 04:17:25.016994 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:25.016976 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wmkf5"] Oct 31 04:17:25.019039 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:25.019008 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:17:25.019122 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:25.019071 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:17:25.019122 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:25.019101 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:17:25.270267 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:25.270167 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wmkf5" event={"ID":"d135a318-172d-4408-8e38-ca0e26946381","Type":"ContainerStarted","Data":"74946a5a9a9631dd0fafccc2fb98e2ade83a6269ccfcbd538e533d04741f061b"} Oct 31 04:17:25.270267 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:25.270228 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wmkf5" event={"ID":"d135a318-172d-4408-8e38-ca0e26946381","Type":"ContainerStarted","Data":"55f4f034380f1d2019eb2d1a4c1d75db3aec6a69bbf3c7093c8498e0c71bf8c1"} Oct 31 04:17:25.325770 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:25.325715 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wmkf5" podStartSLOduration=1.325703009 podStartE2EDuration="1.325703009s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:17:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:17:25.318699975 +0000 UTC m=+793.033034592" watchObservedRunningTime="2026-10-31 04:17:25.325703009 +0000 UTC m=+793.040037626" Oct 31 04:17:25.533600 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:25.533526 4376 patch_prober.go:29] interesting pod/e2e-worker-w3-4c2387-openshift-default-5b68d44445-d7frf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Oct 31 04:17:25.533600 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:25.533579 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-d7frf" podUID="dfcd34ac-2ed4-4f1e-bd06-9cca74916956" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Oct 31 04:17:25.683514 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:25.683480 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wmkf5" Oct 31 04:17:26.533787 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:26.533757 4376 patch_prober.go:29] interesting pod/e2e-worker-w3-4c2387-openshift-default-5b68d44445-d7frf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Oct 31 04:17:26.534784 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:26.534748 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-d7frf" podUID="dfcd34ac-2ed4-4f1e-bd06-9cca74916956" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Oct 31 04:17:26.688557 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:26.688513 4376 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-wmkf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Oct 31 04:17:26.688741 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:26.688582 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wmkf5" podUID="d135a318-172d-4408-8e38-ca0e26946381" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Oct 31 04:17:27.534087 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:27.534050 4376 patch_prober.go:29] interesting pod/e2e-worker-w3-4c2387-openshift-default-5b68d44445-d7frf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Oct 31 04:17:27.534499 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:27.534097 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-d7frf" podUID="dfcd34ac-2ed4-4f1e-bd06-9cca74916956" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Oct 31 04:17:27.686390 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:27.685684 4376 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-wmkf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Oct 31 04:17:27.686390 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:27.685758 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wmkf5" podUID="d135a318-172d-4408-8e38-ca0e26946381" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Oct 31 04:17:28.534014 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:28.533982 4376 patch_prober.go:29] interesting pod/e2e-worker-w3-4c2387-openshift-default-5b68d44445-d7frf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Oct 31 04:17:28.534187 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:28.534028 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-d7frf" podUID="dfcd34ac-2ed4-4f1e-bd06-9cca74916956" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Oct 31 04:17:28.688352 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:28.687740 4376 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-wmkf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Oct 31 04:17:28.688352 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:28.687805 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wmkf5" podUID="d135a318-172d-4408-8e38-ca0e26946381" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Oct 31 04:17:29.533944 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:29.533907 4376 patch_prober.go:29] interesting pod/e2e-worker-w3-4c2387-openshift-default-5b68d44445-d7frf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Oct 31 04:17:29.534095 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:29.533959 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-d7frf" podUID="dfcd34ac-2ed4-4f1e-bd06-9cca74916956" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Oct 31 04:17:29.687352 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:29.686765 4376 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-wmkf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Oct 31 04:17:29.687352 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:29.686830 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wmkf5" podUID="d135a318-172d-4408-8e38-ca0e26946381" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Oct 31 04:17:30.683057 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:30.683018 4376 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-wmkf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Oct 31 04:17:30.683345 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:30.683092 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wmkf5" podUID="d135a318-172d-4408-8e38-ca0e26946381" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Oct 31 04:17:30.723147 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:30.723121 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-d7frf" Oct 31 04:17:30.723800 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:30.723773 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-d7frf" Oct 31 04:17:30.724588 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:30.724567 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-d7frf" Oct 31 04:17:31.683613 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:31.683578 4376 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-wmkf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Oct 31 04:17:31.683791 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:31.683639 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wmkf5" podUID="d135a318-172d-4408-8e38-ca0e26946381" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Oct 31 04:17:31.750720 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:31.750686 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 04:17:32.687674 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:32.687636 4376 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-wmkf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Oct 31 04:17:32.687968 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:32.687934 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wmkf5" podUID="d135a318-172d-4408-8e38-ca0e26946381" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Oct 31 04:17:33.276361 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:33.275406 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-09a20a/e2e-conflict-a-ba60a444-kserve-78b9f5789c-kjs4g" Oct 31 04:17:33.300384 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:33.300090 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-09a20a/e2e-conflict-a-ba60a444-kserve-78b9f5789c-kjs4g" Oct 31 04:17:33.362802 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:33.362771 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-09a20a/e2e-conflict-a-ba60a444-kserve-78b9f5789c-kjs4g" Oct 31 04:17:33.889351 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:33.888777 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wmkf5" Oct 31 04:17:33.896592 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:33.890080 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wmkf5" Oct 31 04:17:33.896764 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:33.896671 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wmkf5" Oct 31 04:17:35.734575 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:35.734546 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-pwfbr"] Oct 31 04:17:35.738449 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:35.738425 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-pwfbr" Oct 31 04:17:35.763089 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:35.763058 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-pwfbr"] Oct 31 04:17:35.851268 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:35.851227 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91eecd3e-fc45-4cf9-8515-a0f723aa9968-istio-podinfo\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-pwfbr\" (UID: \"91eecd3e-fc45-4cf9-8515-a0f723aa9968\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-pwfbr" Oct 31 04:17:35.851469 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:35.851357 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91eecd3e-fc45-4cf9-8515-a0f723aa9968-credential-socket\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-pwfbr\" (UID: \"91eecd3e-fc45-4cf9-8515-a0f723aa9968\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-pwfbr" Oct 31 04:17:35.851469 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:35.851388 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91eecd3e-fc45-4cf9-8515-a0f723aa9968-istio-data\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-pwfbr\" (UID: \"91eecd3e-fc45-4cf9-8515-a0f723aa9968\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-pwfbr" Oct 31 04:17:35.851619 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:35.851546 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91eecd3e-fc45-4cf9-8515-a0f723aa9968-workload-certs\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-pwfbr\" (UID: \"91eecd3e-fc45-4cf9-8515-a0f723aa9968\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-pwfbr" Oct 31 04:17:35.851619 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:35.851577 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91eecd3e-fc45-4cf9-8515-a0f723aa9968-istiod-ca-cert\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-pwfbr\" (UID: \"91eecd3e-fc45-4cf9-8515-a0f723aa9968\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-pwfbr" Oct 31 04:17:35.851619 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:35.851600 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91eecd3e-fc45-4cf9-8515-a0f723aa9968-istio-envoy\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-pwfbr\" (UID: \"91eecd3e-fc45-4cf9-8515-a0f723aa9968\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-pwfbr" Oct 31 04:17:35.851797 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:35.851674 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91eecd3e-fc45-4cf9-8515-a0f723aa9968-istio-token\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-pwfbr\" (UID: \"91eecd3e-fc45-4cf9-8515-a0f723aa9968\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-pwfbr" Oct 31 04:17:35.851797 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:35.851747 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqsjj\" (UniqueName: \"kubernetes.io/projected/91eecd3e-fc45-4cf9-8515-a0f723aa9968-kube-api-access-wqsjj\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-pwfbr\" (UID: \"91eecd3e-fc45-4cf9-8515-a0f723aa9968\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-pwfbr" Oct 31 04:17:35.851797 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:35.851782 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91eecd3e-fc45-4cf9-8515-a0f723aa9968-workload-socket\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-pwfbr\" (UID: \"91eecd3e-fc45-4cf9-8515-a0f723aa9968\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-pwfbr" Oct 31 04:17:35.953256 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:35.953211 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91eecd3e-fc45-4cf9-8515-a0f723aa9968-istio-token\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-pwfbr\" (UID: \"91eecd3e-fc45-4cf9-8515-a0f723aa9968\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-pwfbr" Oct 31 04:17:35.953449 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:35.953284 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqsjj\" (UniqueName: \"kubernetes.io/projected/91eecd3e-fc45-4cf9-8515-a0f723aa9968-kube-api-access-wqsjj\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-pwfbr\" (UID: \"91eecd3e-fc45-4cf9-8515-a0f723aa9968\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-pwfbr" Oct 31 04:17:35.953449 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:35.953347 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91eecd3e-fc45-4cf9-8515-a0f723aa9968-workload-socket\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-pwfbr\" (UID: \"91eecd3e-fc45-4cf9-8515-a0f723aa9968\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-pwfbr" Oct 31 04:17:35.953449 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:35.953416 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91eecd3e-fc45-4cf9-8515-a0f723aa9968-istio-podinfo\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-pwfbr\" (UID: \"91eecd3e-fc45-4cf9-8515-a0f723aa9968\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-pwfbr" Oct 31 04:17:35.953642 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:35.953465 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91eecd3e-fc45-4cf9-8515-a0f723aa9968-credential-socket\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-pwfbr\" (UID: \"91eecd3e-fc45-4cf9-8515-a0f723aa9968\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-pwfbr" Oct 31 04:17:35.953642 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:35.953494 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91eecd3e-fc45-4cf9-8515-a0f723aa9968-istio-data\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-pwfbr\" (UID: \"91eecd3e-fc45-4cf9-8515-a0f723aa9968\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-pwfbr" Oct 31 04:17:35.953642 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:35.953589 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91eecd3e-fc45-4cf9-8515-a0f723aa9968-workload-certs\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-pwfbr\" (UID: \"91eecd3e-fc45-4cf9-8515-a0f723aa9968\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-pwfbr" Oct 31 04:17:35.953642 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:35.953627 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91eecd3e-fc45-4cf9-8515-a0f723aa9968-istiod-ca-cert\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-pwfbr\" (UID: \"91eecd3e-fc45-4cf9-8515-a0f723aa9968\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-pwfbr" Oct 31 04:17:35.953857 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:35.953656 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91eecd3e-fc45-4cf9-8515-a0f723aa9968-istio-envoy\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-pwfbr\" (UID: \"91eecd3e-fc45-4cf9-8515-a0f723aa9968\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-pwfbr" Oct 31 04:17:35.953922 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:35.953883 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91eecd3e-fc45-4cf9-8515-a0f723aa9968-workload-socket\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-pwfbr\" (UID: \"91eecd3e-fc45-4cf9-8515-a0f723aa9968\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-pwfbr" Oct 31 04:17:35.954191 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:35.954134 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91eecd3e-fc45-4cf9-8515-a0f723aa9968-workload-certs\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-pwfbr\" (UID: \"91eecd3e-fc45-4cf9-8515-a0f723aa9968\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-pwfbr" Oct 31 04:17:35.954191 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:35.954178 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91eecd3e-fc45-4cf9-8515-a0f723aa9968-istio-data\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-pwfbr\" (UID: \"91eecd3e-fc45-4cf9-8515-a0f723aa9968\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-pwfbr" Oct 31 04:17:35.954468 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:35.954437 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91eecd3e-fc45-4cf9-8515-a0f723aa9968-credential-socket\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-pwfbr\" (UID: \"91eecd3e-fc45-4cf9-8515-a0f723aa9968\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-pwfbr" Oct 31 04:17:35.954593 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:35.954542 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91eecd3e-fc45-4cf9-8515-a0f723aa9968-istiod-ca-cert\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-pwfbr\" (UID: \"91eecd3e-fc45-4cf9-8515-a0f723aa9968\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-pwfbr" Oct 31 04:17:35.956481 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:35.956454 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91eecd3e-fc45-4cf9-8515-a0f723aa9968-istio-envoy\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-pwfbr\" (UID: \"91eecd3e-fc45-4cf9-8515-a0f723aa9968\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-pwfbr" Oct 31 04:17:35.956773 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:35.956750 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91eecd3e-fc45-4cf9-8515-a0f723aa9968-istio-podinfo\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-pwfbr\" (UID: \"91eecd3e-fc45-4cf9-8515-a0f723aa9968\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-pwfbr" Oct 31 04:17:35.979501 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:35.979479 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91eecd3e-fc45-4cf9-8515-a0f723aa9968-istio-token\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-pwfbr\" (UID: \"91eecd3e-fc45-4cf9-8515-a0f723aa9968\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-pwfbr" Oct 31 04:17:35.979821 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:35.979799 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqsjj\" (UniqueName: \"kubernetes.io/projected/91eecd3e-fc45-4cf9-8515-a0f723aa9968-kube-api-access-wqsjj\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-pwfbr\" (UID: \"91eecd3e-fc45-4cf9-8515-a0f723aa9968\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-pwfbr" Oct 31 04:17:36.059510 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:36.059412 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-pwfbr" Oct 31 04:17:36.392143 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:36.392106 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-pwfbr"] Oct 31 04:17:36.412951 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:36.412914 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:17:36.413088 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:36.412982 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:17:36.413088 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:36.413016 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:17:37.222964 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:37.222925 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-bf555d5c7-mjl5l"] Oct 31 04:17:37.385410 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:37.385369 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-pwfbr" event={"ID":"91eecd3e-fc45-4cf9-8515-a0f723aa9968","Type":"ContainerStarted","Data":"88f96685a490995d2b8220307831114d1f9093d62957f8abb16c30127d85ba80"} Oct 31 04:17:37.385610 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:37.385425 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-pwfbr" event={"ID":"91eecd3e-fc45-4cf9-8515-a0f723aa9968","Type":"ContainerStarted","Data":"2874989d93930c463e4c261d5fb43d77818c3ab82b7fd26df628847e05e7adc3"} Oct 31 04:17:37.451353 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:37.437490 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-pwfbr" podStartSLOduration=2.4374716149999998 podStartE2EDuration="2.437471615s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:17:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:17:37.435170528 +0000 UTC m=+805.149505163" watchObservedRunningTime="2026-10-31 04:17:37.437471615 +0000 UTC m=+805.151806235" Oct 31 04:17:38.062587 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:38.062543 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-pwfbr" Oct 31 04:17:38.064348 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:38.064296 4376 patch_prober.go:29] interesting pod/e2e-worker-w2-09a20a-openshift-default-55d8647ff-pwfbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 31 04:17:38.064499 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:38.064370 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-pwfbr" podUID="91eecd3e-fc45-4cf9-8515-a0f723aa9968" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 31 04:17:39.062742 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:39.062705 4376 patch_prober.go:29] interesting pod/e2e-worker-w2-09a20a-openshift-default-55d8647ff-pwfbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 31 04:17:39.063150 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:39.062759 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-pwfbr" podUID="91eecd3e-fc45-4cf9-8515-a0f723aa9968" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 31 04:17:40.062773 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:40.062738 4376 patch_prober.go:29] interesting pod/e2e-worker-w2-09a20a-openshift-default-55d8647ff-pwfbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 31 04:17:40.063168 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:40.062793 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-pwfbr" podUID="91eecd3e-fc45-4cf9-8515-a0f723aa9968" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 31 04:17:40.999641 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:17:40.999601 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 04:17:41.062354 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:41.061725 4376 patch_prober.go:29] interesting pod/e2e-worker-w2-09a20a-openshift-default-55d8647ff-pwfbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 31 04:17:41.062354 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:41.061791 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-pwfbr" podUID="91eecd3e-fc45-4cf9-8515-a0f723aa9968" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 31 04:17:42.063900 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:42.063685 4376 patch_prober.go:29] interesting pod/e2e-worker-w2-09a20a-openshift-default-55d8647ff-pwfbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 31 04:17:42.063900 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:42.063756 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-pwfbr" podUID="91eecd3e-fc45-4cf9-8515-a0f723aa9968" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 31 04:17:42.223841 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:42.223796 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-bf555d5c7-mjl5l" podUID="64ffa199-1f29-4a73-99ea-36b861f64b21" containerName="payload-processing" containerID="cri-o://72ea032d233e239311011e78df190154a9c1599bee128f884e5c6a7fdad5e0ae" gracePeriod=25 Oct 31 04:17:42.439839 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:42.439502 4376 generic.go:356] "Generic (PLEG): container finished" podID="64ffa199-1f29-4a73-99ea-36b861f64b21" containerID="72ea032d233e239311011e78df190154a9c1599bee128f884e5c6a7fdad5e0ae" exitCode=0 Oct 31 04:17:42.439839 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:42.439678 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-bf555d5c7-mjl5l" event={"ID":"64ffa199-1f29-4a73-99ea-36b861f64b21","Type":"ContainerDied","Data":"72ea032d233e239311011e78df190154a9c1599bee128f884e5c6a7fdad5e0ae"} Oct 31 04:17:42.568639 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:42.568613 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-bf555d5c7-mjl5l" Oct 31 04:17:42.607338 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:42.607301 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64ffa199-1f29-4a73-99ea-36b861f64b21-provider-credentials\" (UniqueName: \"kubernetes.io/projected/64ffa199-1f29-4a73-99ea-36b861f64b21-provider-credentials\") pod \"64ffa199-1f29-4a73-99ea-36b861f64b21\" (UID: \"64ffa199-1f29-4a73-99ea-36b861f64b21\") " Oct 31 04:17:42.607489 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:42.607355 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64ffa199-1f29-4a73-99ea-36b861f64b21-kube-api-access-wp2zs\" (UniqueName: \"kubernetes.io/projected/64ffa199-1f29-4a73-99ea-36b861f64b21-kube-api-access-wp2zs\") pod \"64ffa199-1f29-4a73-99ea-36b861f64b21\" (UID: \"64ffa199-1f29-4a73-99ea-36b861f64b21\") " Oct 31 04:17:42.607489 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:42.607425 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64ffa199-1f29-4a73-99ea-36b861f64b21-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/64ffa199-1f29-4a73-99ea-36b861f64b21-routing-overlay-volume\") pod \"64ffa199-1f29-4a73-99ea-36b861f64b21\" (UID: \"64ffa199-1f29-4a73-99ea-36b861f64b21\") " Oct 31 04:17:42.607489 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:42.607459 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64ffa199-1f29-4a73-99ea-36b861f64b21-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64ffa199-1f29-4a73-99ea-36b861f64b21-plugins-config-volume\") pod \"64ffa199-1f29-4a73-99ea-36b861f64b21\" (UID: \"64ffa199-1f29-4a73-99ea-36b861f64b21\") " Oct 31 04:17:42.607855 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:42.607825 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64ffa199-1f29-4a73-99ea-36b861f64b21-routing-overlay-volume" pod "64ffa199-1f29-4a73-99ea-36b861f64b21" (UID: "64ffa199-1f29-4a73-99ea-36b861f64b21"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:17:42.607968 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:42.607869 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64ffa199-1f29-4a73-99ea-36b861f64b21-plugins-config-volume" pod "64ffa199-1f29-4a73-99ea-36b861f64b21" (UID: "64ffa199-1f29-4a73-99ea-36b861f64b21"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:17:42.609594 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:42.609569 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64ffa199-1f29-4a73-99ea-36b861f64b21-provider-credentials" pod "64ffa199-1f29-4a73-99ea-36b861f64b21" (UID: "64ffa199-1f29-4a73-99ea-36b861f64b21"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:17:42.609658 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:42.609624 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64ffa199-1f29-4a73-99ea-36b861f64b21-kube-api-access-wp2zs" pod "64ffa199-1f29-4a73-99ea-36b861f64b21" (UID: "64ffa199-1f29-4a73-99ea-36b861f64b21"). InnerVolumeSpecName "kube-api-access-wp2zs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:17:42.708371 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:42.708324 4376 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/64ffa199-1f29-4a73-99ea-36b861f64b21-provider-credentials\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:17:42.708371 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:42.708367 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wp2zs\" (UniqueName: \"kubernetes.io/projected/64ffa199-1f29-4a73-99ea-36b861f64b21-kube-api-access-wp2zs\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:17:42.708371 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:42.708377 4376 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/64ffa199-1f29-4a73-99ea-36b861f64b21-routing-overlay-volume\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:17:42.708664 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:42.708387 4376 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64ffa199-1f29-4a73-99ea-36b861f64b21-plugins-config-volume\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:17:43.060257 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:43.060148 4376 patch_prober.go:29] interesting pod/e2e-worker-w2-09a20a-openshift-default-55d8647ff-pwfbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 31 04:17:43.060257 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:43.060231 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-pwfbr" podUID="91eecd3e-fc45-4cf9-8515-a0f723aa9968" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 31 04:17:43.449281 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:43.449191 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-bf555d5c7-mjl5l" Oct 31 04:17:43.449281 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:43.449241 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-bf555d5c7-mjl5l" event={"ID":"64ffa199-1f29-4a73-99ea-36b861f64b21","Type":"ContainerDied","Data":"d86d6b522fd9d718fc5baf6b078dacd60a703689e8b38094dd6ba07cb1e344f3"} Oct 31 04:17:43.449803 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:43.449313 4376 scope.go:122] "RemoveContainer" containerID="72ea032d233e239311011e78df190154a9c1599bee128f884e5c6a7fdad5e0ae" Oct 31 04:17:43.507243 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:43.507219 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-bf555d5c7-mjl5l"] Oct 31 04:17:43.523368 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:43.523342 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-bf555d5c7-mjl5l"] Oct 31 04:17:44.063927 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:44.063895 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-pwfbr" Oct 31 04:17:44.064212 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:44.064192 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-pwfbr" Oct 31 04:17:44.064920 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:44.064900 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-pwfbr" Oct 31 04:17:44.478968 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:44.478931 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64ffa199-1f29-4a73-99ea-36b861f64b21" path="/var/lib/kubelet/pods/64ffa199-1f29-4a73-99ea-36b861f64b21/volumes" Oct 31 04:17:48.250315 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:48.250284 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 04:17:51.335515 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:51.335476 4376 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-crjjl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 04:17:51.335902 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:17:51.335554 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-crjjl" podUID="63486e50-4059-425f-b04a-4adea538d427" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 04:18:07.006482 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:07.006446 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-f445668d4-gmmrd"] Oct 31 04:18:07.007211 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:07.007173 4376 cpu_manager.go:414] "containerMap: removing container" podUID="64ffa199-1f29-4a73-99ea-36b861f64b21" containerName="payload-processing" Oct 31 04:18:07.007211 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:07.007201 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64ffa199-1f29-4a73-99ea-36b861f64b21" containerName="payload-processing" Oct 31 04:18:07.007411 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:07.007315 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="64ffa199-1f29-4a73-99ea-36b861f64b21" containerName="payload-processing" Oct 31 04:18:07.010780 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:07.010756 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-f445668d4-gmmrd" Oct 31 04:18:07.026884 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:07.026858 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-f445668d4-gmmrd"] Oct 31 04:18:07.038677 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:07.038653 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Oct 31 04:18:07.038677 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:07.038650 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-5e88718f0543\"" Oct 31 04:18:07.038840 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:07.038705 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-6c645c79ce32\"" Oct 31 04:18:07.043111 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:07.043093 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Oct 31 04:18:07.043209 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:07.043140 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-cdd9c78cff58\"" Oct 31 04:18:07.043209 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:07.043160 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-5qlm5\"" Oct 31 04:18:07.043209 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:07.043181 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Oct 31 04:18:07.043343 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:07.043223 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Oct 31 04:18:07.140707 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:07.140673 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d306e0df-7e92-48f8-a089-8b130186db1d-plugins-config-volume\") pod \"payload-processing-external-model-f445668d4-gmmrd\" (UID: \"d306e0df-7e92-48f8-a089-8b130186db1d\") " pod="models-as-a-service/payload-processing-external-model-f445668d4-gmmrd" Oct 31 04:18:07.140877 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:07.140715 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqmrn\" (UniqueName: \"kubernetes.io/projected/d306e0df-7e92-48f8-a089-8b130186db1d-kube-api-access-zqmrn\") pod \"payload-processing-external-model-f445668d4-gmmrd\" (UID: \"d306e0df-7e92-48f8-a089-8b130186db1d\") " pod="models-as-a-service/payload-processing-external-model-f445668d4-gmmrd" Oct 31 04:18:07.140877 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:07.140860 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d306e0df-7e92-48f8-a089-8b130186db1d-provider-credentials\") pod \"payload-processing-external-model-f445668d4-gmmrd\" (UID: \"d306e0df-7e92-48f8-a089-8b130186db1d\") " pod="models-as-a-service/payload-processing-external-model-f445668d4-gmmrd" Oct 31 04:18:07.140968 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:07.140881 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d306e0df-7e92-48f8-a089-8b130186db1d-routing-overlay-volume\") pod \"payload-processing-external-model-f445668d4-gmmrd\" (UID: \"d306e0df-7e92-48f8-a089-8b130186db1d\") " pod="models-as-a-service/payload-processing-external-model-f445668d4-gmmrd" Oct 31 04:18:07.241743 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:07.241706 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d306e0df-7e92-48f8-a089-8b130186db1d-plugins-config-volume\") pod \"payload-processing-external-model-f445668d4-gmmrd\" (UID: \"d306e0df-7e92-48f8-a089-8b130186db1d\") " pod="models-as-a-service/payload-processing-external-model-f445668d4-gmmrd" Oct 31 04:18:07.241743 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:07.241746 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqmrn\" (UniqueName: \"kubernetes.io/projected/d306e0df-7e92-48f8-a089-8b130186db1d-kube-api-access-zqmrn\") pod \"payload-processing-external-model-f445668d4-gmmrd\" (UID: \"d306e0df-7e92-48f8-a089-8b130186db1d\") " pod="models-as-a-service/payload-processing-external-model-f445668d4-gmmrd" Oct 31 04:18:07.242015 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:07.241827 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d306e0df-7e92-48f8-a089-8b130186db1d-provider-credentials\") pod \"payload-processing-external-model-f445668d4-gmmrd\" (UID: \"d306e0df-7e92-48f8-a089-8b130186db1d\") " pod="models-as-a-service/payload-processing-external-model-f445668d4-gmmrd" Oct 31 04:18:07.242015 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:07.241847 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d306e0df-7e92-48f8-a089-8b130186db1d-routing-overlay-volume\") pod \"payload-processing-external-model-f445668d4-gmmrd\" (UID: \"d306e0df-7e92-48f8-a089-8b130186db1d\") " pod="models-as-a-service/payload-processing-external-model-f445668d4-gmmrd" Oct 31 04:18:07.242493 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:07.242466 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d306e0df-7e92-48f8-a089-8b130186db1d-routing-overlay-volume\") pod \"payload-processing-external-model-f445668d4-gmmrd\" (UID: \"d306e0df-7e92-48f8-a089-8b130186db1d\") " pod="models-as-a-service/payload-processing-external-model-f445668d4-gmmrd" Oct 31 04:18:07.242628 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:07.242528 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d306e0df-7e92-48f8-a089-8b130186db1d-plugins-config-volume\") pod \"payload-processing-external-model-f445668d4-gmmrd\" (UID: \"d306e0df-7e92-48f8-a089-8b130186db1d\") " pod="models-as-a-service/payload-processing-external-model-f445668d4-gmmrd" Oct 31 04:18:07.244440 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:07.244413 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d306e0df-7e92-48f8-a089-8b130186db1d-provider-credentials\") pod \"payload-processing-external-model-f445668d4-gmmrd\" (UID: \"d306e0df-7e92-48f8-a089-8b130186db1d\") " pod="models-as-a-service/payload-processing-external-model-f445668d4-gmmrd" Oct 31 04:18:07.267080 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:07.267021 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqmrn\" (UniqueName: \"kubernetes.io/projected/d306e0df-7e92-48f8-a089-8b130186db1d-kube-api-access-zqmrn\") pod \"payload-processing-external-model-f445668d4-gmmrd\" (UID: \"d306e0df-7e92-48f8-a089-8b130186db1d\") " pod="models-as-a-service/payload-processing-external-model-f445668d4-gmmrd" Oct 31 04:18:07.329352 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:07.328554 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-f445668d4-gmmrd" Oct 31 04:18:07.646736 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:18:07.646702 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd306e0df_7e92_48f8_a089_8b130186db1d.slice/crio-dbe53d3b2158307e586ee90e8804c0716f467d83bdc6e358e5c19563a18a7c53 WatchSource:0}: Error finding container dbe53d3b2158307e586ee90e8804c0716f467d83bdc6e358e5c19563a18a7c53: Status 404 returned error can't find the container with id dbe53d3b2158307e586ee90e8804c0716f467d83bdc6e358e5c19563a18a7c53 Oct 31 04:18:07.647296 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:07.647275 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-f445668d4-gmmrd"] Oct 31 04:18:08.622828 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:08.622787 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-f445668d4-gmmrd" event={"ID":"d306e0df-7e92-48f8-a089-8b130186db1d","Type":"ContainerStarted","Data":"756286597bfccae50ba40d32b968c6b4803c2cd3834d1b0c1f3023e67e8292e2"} Oct 31 04:18:08.623238 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:08.622849 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-f445668d4-gmmrd" Oct 31 04:18:08.623238 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:08.622872 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-f445668d4-gmmrd" event={"ID":"d306e0df-7e92-48f8-a089-8b130186db1d","Type":"ContainerStarted","Data":"dbe53d3b2158307e586ee90e8804c0716f467d83bdc6e358e5c19563a18a7c53"} Oct 31 04:18:08.672295 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:08.672236 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-f445668d4-gmmrd" podStartSLOduration=2.672218152 podStartE2EDuration="2.672218152s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:18:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:18:08.668193162 +0000 UTC m=+836.382527780" watchObservedRunningTime="2026-10-31 04:18:08.672218152 +0000 UTC m=+836.386552770" Oct 31 04:18:10.999261 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:18:10.999213 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 04:18:12.846877 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:12.846841 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 04:18:19.632548 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:19.632517 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-f445668d4-gmmrd" Oct 31 04:18:20.880386 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:20.880345 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-5shj7"] Oct 31 04:18:20.885374 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:20.885355 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-5shj7" Oct 31 04:18:20.910771 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:20.910735 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-5shj7"] Oct 31 04:18:21.098627 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:21.098594 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/982c902b-4689-4eb3-93fb-8930a3136465-istio-envoy\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-5shj7\" (UID: \"982c902b-4689-4eb3-93fb-8930a3136465\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-5shj7" Oct 31 04:18:21.098803 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:21.098655 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/982c902b-4689-4eb3-93fb-8930a3136465-credential-socket\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-5shj7\" (UID: \"982c902b-4689-4eb3-93fb-8930a3136465\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-5shj7" Oct 31 04:18:21.098803 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:21.098752 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/982c902b-4689-4eb3-93fb-8930a3136465-workload-certs\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-5shj7\" (UID: \"982c902b-4689-4eb3-93fb-8930a3136465\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-5shj7" Oct 31 04:18:21.098922 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:21.098816 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glhxj\" (UniqueName: \"kubernetes.io/projected/982c902b-4689-4eb3-93fb-8930a3136465-kube-api-access-glhxj\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-5shj7\" (UID: \"982c902b-4689-4eb3-93fb-8930a3136465\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-5shj7" Oct 31 04:18:21.098922 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:21.098874 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/982c902b-4689-4eb3-93fb-8930a3136465-istiod-ca-cert\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-5shj7\" (UID: \"982c902b-4689-4eb3-93fb-8930a3136465\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-5shj7" Oct 31 04:18:21.099009 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:21.098985 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/982c902b-4689-4eb3-93fb-8930a3136465-istio-podinfo\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-5shj7\" (UID: \"982c902b-4689-4eb3-93fb-8930a3136465\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-5shj7" Oct 31 04:18:21.099077 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:21.099064 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/982c902b-4689-4eb3-93fb-8930a3136465-istio-token\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-5shj7\" (UID: \"982c902b-4689-4eb3-93fb-8930a3136465\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-5shj7" Oct 31 04:18:21.099114 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:21.099097 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/982c902b-4689-4eb3-93fb-8930a3136465-workload-socket\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-5shj7\" (UID: \"982c902b-4689-4eb3-93fb-8930a3136465\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-5shj7" Oct 31 04:18:21.099161 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:21.099124 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/982c902b-4689-4eb3-93fb-8930a3136465-istio-data\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-5shj7\" (UID: \"982c902b-4689-4eb3-93fb-8930a3136465\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-5shj7" Oct 31 04:18:21.199998 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:21.199924 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/982c902b-4689-4eb3-93fb-8930a3136465-workload-socket\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-5shj7\" (UID: \"982c902b-4689-4eb3-93fb-8930a3136465\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-5shj7" Oct 31 04:18:21.199998 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:21.199957 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/982c902b-4689-4eb3-93fb-8930a3136465-istio-data\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-5shj7\" (UID: \"982c902b-4689-4eb3-93fb-8930a3136465\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-5shj7" Oct 31 04:18:21.200197 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:21.200010 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/982c902b-4689-4eb3-93fb-8930a3136465-istio-envoy\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-5shj7\" (UID: \"982c902b-4689-4eb3-93fb-8930a3136465\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-5shj7" Oct 31 04:18:21.200197 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:21.200058 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/982c902b-4689-4eb3-93fb-8930a3136465-credential-socket\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-5shj7\" (UID: \"982c902b-4689-4eb3-93fb-8930a3136465\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-5shj7" Oct 31 04:18:21.200197 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:21.200096 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/982c902b-4689-4eb3-93fb-8930a3136465-workload-certs\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-5shj7\" (UID: \"982c902b-4689-4eb3-93fb-8930a3136465\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-5shj7" Oct 31 04:18:21.200197 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:21.200126 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glhxj\" (UniqueName: \"kubernetes.io/projected/982c902b-4689-4eb3-93fb-8930a3136465-kube-api-access-glhxj\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-5shj7\" (UID: \"982c902b-4689-4eb3-93fb-8930a3136465\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-5shj7" Oct 31 04:18:21.200197 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:21.200176 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/982c902b-4689-4eb3-93fb-8930a3136465-istiod-ca-cert\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-5shj7\" (UID: \"982c902b-4689-4eb3-93fb-8930a3136465\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-5shj7" Oct 31 04:18:21.200501 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:21.200247 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/982c902b-4689-4eb3-93fb-8930a3136465-istio-podinfo\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-5shj7\" (UID: \"982c902b-4689-4eb3-93fb-8930a3136465\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-5shj7" Oct 31 04:18:21.200501 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:21.200307 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/982c902b-4689-4eb3-93fb-8930a3136465-istio-token\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-5shj7\" (UID: \"982c902b-4689-4eb3-93fb-8930a3136465\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-5shj7" Oct 31 04:18:21.200501 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:21.200311 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/982c902b-4689-4eb3-93fb-8930a3136465-workload-socket\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-5shj7\" (UID: \"982c902b-4689-4eb3-93fb-8930a3136465\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-5shj7" Oct 31 04:18:21.200501 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:21.200406 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/982c902b-4689-4eb3-93fb-8930a3136465-istio-data\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-5shj7\" (UID: \"982c902b-4689-4eb3-93fb-8930a3136465\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-5shj7" Oct 31 04:18:21.200974 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:21.200941 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/982c902b-4689-4eb3-93fb-8930a3136465-credential-socket\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-5shj7\" (UID: \"982c902b-4689-4eb3-93fb-8930a3136465\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-5shj7" Oct 31 04:18:21.200974 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:21.200960 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/982c902b-4689-4eb3-93fb-8930a3136465-istiod-ca-cert\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-5shj7\" (UID: \"982c902b-4689-4eb3-93fb-8930a3136465\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-5shj7" Oct 31 04:18:21.201119 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:21.200990 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/982c902b-4689-4eb3-93fb-8930a3136465-workload-certs\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-5shj7\" (UID: \"982c902b-4689-4eb3-93fb-8930a3136465\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-5shj7" Oct 31 04:18:21.202718 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:21.202692 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/982c902b-4689-4eb3-93fb-8930a3136465-istio-envoy\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-5shj7\" (UID: \"982c902b-4689-4eb3-93fb-8930a3136465\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-5shj7" Oct 31 04:18:21.202810 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:21.202698 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/982c902b-4689-4eb3-93fb-8930a3136465-istio-podinfo\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-5shj7\" (UID: \"982c902b-4689-4eb3-93fb-8930a3136465\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-5shj7" Oct 31 04:18:21.225236 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:21.225207 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/982c902b-4689-4eb3-93fb-8930a3136465-istio-token\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-5shj7\" (UID: \"982c902b-4689-4eb3-93fb-8930a3136465\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-5shj7" Oct 31 04:18:21.225408 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:21.225392 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-glhxj\" (UniqueName: \"kubernetes.io/projected/982c902b-4689-4eb3-93fb-8930a3136465-kube-api-access-glhxj\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-5shj7\" (UID: \"982c902b-4689-4eb3-93fb-8930a3136465\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-5shj7" Oct 31 04:18:21.225547 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:21.225533 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-5shj7" Oct 31 04:18:21.537490 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:18:21.537456 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod982c902b_4689_4eb3_93fb_8930a3136465.slice/crio-e73841b8a77074b8e5b28ac1630b8b6effee89b66fb20ed0db393ed9c99c8b04 WatchSource:0}: Error finding container e73841b8a77074b8e5b28ac1630b8b6effee89b66fb20ed0db393ed9c99c8b04: Status 404 returned error can't find the container with id e73841b8a77074b8e5b28ac1630b8b6effee89b66fb20ed0db393ed9c99c8b04 Oct 31 04:18:21.538535 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:21.538511 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-5shj7"] Oct 31 04:18:21.539887 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:21.539852 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:18:21.539964 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:21.539936 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:18:21.540004 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:21.539966 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:18:21.761678 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:21.761639 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-5shj7" event={"ID":"982c902b-4689-4eb3-93fb-8930a3136465","Type":"ContainerStarted","Data":"20a52217f980f305428621e7a04e253804041300b42a6a00ed24f737f1b6a78d"} Oct 31 04:18:21.761849 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:21.761706 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-5shj7" event={"ID":"982c902b-4689-4eb3-93fb-8930a3136465","Type":"ContainerStarted","Data":"e73841b8a77074b8e5b28ac1630b8b6effee89b66fb20ed0db393ed9c99c8b04"} Oct 31 04:18:21.812023 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:21.811926 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-5shj7" podStartSLOduration=1.811913959 podStartE2EDuration="1.811913959s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:18:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:18:21.810708247 +0000 UTC m=+849.525042881" watchObservedRunningTime="2026-10-31 04:18:21.811913959 +0000 UTC m=+849.526248576" Oct 31 04:18:22.226092 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:22.226054 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-5shj7" Oct 31 04:18:22.227632 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:22.227610 4376 patch_prober.go:29] interesting pod/e2e-worker-w2-09a20a-openshift-default-55d8647ff-5shj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 31 04:18:22.227691 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:22.227656 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-5shj7" podUID="982c902b-4689-4eb3-93fb-8930a3136465" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 31 04:18:23.226249 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:23.226209 4376 patch_prober.go:29] interesting pod/e2e-worker-w2-09a20a-openshift-default-55d8647ff-5shj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 31 04:18:23.226654 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:23.226281 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-5shj7" podUID="982c902b-4689-4eb3-93fb-8930a3136465" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 31 04:18:24.226520 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:24.226480 4376 patch_prober.go:29] interesting pod/e2e-worker-w2-09a20a-openshift-default-55d8647ff-5shj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 31 04:18:24.226926 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:24.226557 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-5shj7" podUID="982c902b-4689-4eb3-93fb-8930a3136465" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 31 04:18:25.226554 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:25.226479 4376 patch_prober.go:29] interesting pod/e2e-worker-w2-09a20a-openshift-default-55d8647ff-5shj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 31 04:18:25.226554 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:25.226539 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-5shj7" podUID="982c902b-4689-4eb3-93fb-8930a3136465" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 31 04:18:26.226858 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:26.226821 4376 patch_prober.go:29] interesting pod/e2e-worker-w2-09a20a-openshift-default-55d8647ff-5shj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 31 04:18:26.227243 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:26.226874 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-5shj7" podUID="982c902b-4689-4eb3-93fb-8930a3136465" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 31 04:18:27.226785 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:27.226750 4376 patch_prober.go:29] interesting pod/e2e-worker-w2-09a20a-openshift-default-55d8647ff-5shj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 31 04:18:27.226965 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:27.226818 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-5shj7" podUID="982c902b-4689-4eb3-93fb-8930a3136465" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 31 04:18:28.226296 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:28.226254 4376 patch_prober.go:29] interesting pod/e2e-worker-w2-09a20a-openshift-default-55d8647ff-5shj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 31 04:18:28.226516 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:28.226354 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-5shj7" podUID="982c902b-4689-4eb3-93fb-8930a3136465" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 31 04:18:29.557353 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:29.552407 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-d7npm" podUID="174171e7-100c-4b84-becd-0176672f1805" containerName="main" probeResult="failure" output="Get \"https://10.133.0.34:8000/ready\": context deadline exceeded" Oct 31 04:18:29.654609 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:29.653645 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-5shj7" Oct 31 04:18:29.654609 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:29.654549 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-5shj7" Oct 31 04:18:29.665697 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:29.665609 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-5shj7" Oct 31 04:18:32.482642 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:32.482606 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-09a20a/e2e-conflict-a-ba60a444-kserve-78b9f5789c-kjs4g"] Oct 31 04:18:35.930233 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:35.930198 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 04:18:41.000029 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:18:40.999988 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 04:18:44.701045 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:44.701006 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-6c6d95cc8-gk8xd"] Oct 31 04:18:44.719752 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:44.719719 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6c6d95cc8-gk8xd"] Oct 31 04:18:44.720318 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:44.720266 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6c6d95cc8-gk8xd" Oct 31 04:18:44.859264 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:44.856111 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/25cb40cf-a748-4af8-8aab-a2c9c75b53dc-routing-overlay-volume\") pod \"payload-processing-external-model-6c6d95cc8-gk8xd\" (UID: \"25cb40cf-a748-4af8-8aab-a2c9c75b53dc\") " pod="models-as-a-service/payload-processing-external-model-6c6d95cc8-gk8xd" Oct 31 04:18:44.861819 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:44.861785 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/25cb40cf-a748-4af8-8aab-a2c9c75b53dc-provider-credentials\") pod \"payload-processing-external-model-6c6d95cc8-gk8xd\" (UID: \"25cb40cf-a748-4af8-8aab-a2c9c75b53dc\") " pod="models-as-a-service/payload-processing-external-model-6c6d95cc8-gk8xd" Oct 31 04:18:44.862104 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:44.862083 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25cb40cf-a748-4af8-8aab-a2c9c75b53dc-plugins-config-volume\") pod \"payload-processing-external-model-6c6d95cc8-gk8xd\" (UID: \"25cb40cf-a748-4af8-8aab-a2c9c75b53dc\") " pod="models-as-a-service/payload-processing-external-model-6c6d95cc8-gk8xd" Oct 31 04:18:44.862259 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:44.862241 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d88z7\" (UniqueName: \"kubernetes.io/projected/25cb40cf-a748-4af8-8aab-a2c9c75b53dc-kube-api-access-d88z7\") pod \"payload-processing-external-model-6c6d95cc8-gk8xd\" (UID: \"25cb40cf-a748-4af8-8aab-a2c9c75b53dc\") " pod="models-as-a-service/payload-processing-external-model-6c6d95cc8-gk8xd" Oct 31 04:18:44.965069 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:44.964967 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/25cb40cf-a748-4af8-8aab-a2c9c75b53dc-provider-credentials\") pod \"payload-processing-external-model-6c6d95cc8-gk8xd\" (UID: \"25cb40cf-a748-4af8-8aab-a2c9c75b53dc\") " pod="models-as-a-service/payload-processing-external-model-6c6d95cc8-gk8xd" Oct 31 04:18:44.965069 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:44.965052 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25cb40cf-a748-4af8-8aab-a2c9c75b53dc-plugins-config-volume\") pod \"payload-processing-external-model-6c6d95cc8-gk8xd\" (UID: \"25cb40cf-a748-4af8-8aab-a2c9c75b53dc\") " pod="models-as-a-service/payload-processing-external-model-6c6d95cc8-gk8xd" Oct 31 04:18:44.965279 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:44.965092 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d88z7\" (UniqueName: \"kubernetes.io/projected/25cb40cf-a748-4af8-8aab-a2c9c75b53dc-kube-api-access-d88z7\") pod \"payload-processing-external-model-6c6d95cc8-gk8xd\" (UID: \"25cb40cf-a748-4af8-8aab-a2c9c75b53dc\") " pod="models-as-a-service/payload-processing-external-model-6c6d95cc8-gk8xd" Oct 31 04:18:44.965279 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:44.965262 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/25cb40cf-a748-4af8-8aab-a2c9c75b53dc-routing-overlay-volume\") pod \"payload-processing-external-model-6c6d95cc8-gk8xd\" (UID: \"25cb40cf-a748-4af8-8aab-a2c9c75b53dc\") " pod="models-as-a-service/payload-processing-external-model-6c6d95cc8-gk8xd" Oct 31 04:18:44.966215 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:44.966182 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/25cb40cf-a748-4af8-8aab-a2c9c75b53dc-routing-overlay-volume\") pod \"payload-processing-external-model-6c6d95cc8-gk8xd\" (UID: \"25cb40cf-a748-4af8-8aab-a2c9c75b53dc\") " pod="models-as-a-service/payload-processing-external-model-6c6d95cc8-gk8xd" Oct 31 04:18:44.966525 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:44.966499 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25cb40cf-a748-4af8-8aab-a2c9c75b53dc-plugins-config-volume\") pod \"payload-processing-external-model-6c6d95cc8-gk8xd\" (UID: \"25cb40cf-a748-4af8-8aab-a2c9c75b53dc\") " pod="models-as-a-service/payload-processing-external-model-6c6d95cc8-gk8xd" Oct 31 04:18:44.968993 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:44.968968 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/25cb40cf-a748-4af8-8aab-a2c9c75b53dc-provider-credentials\") pod \"payload-processing-external-model-6c6d95cc8-gk8xd\" (UID: \"25cb40cf-a748-4af8-8aab-a2c9c75b53dc\") " pod="models-as-a-service/payload-processing-external-model-6c6d95cc8-gk8xd" Oct 31 04:18:44.994119 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:44.994087 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d88z7\" (UniqueName: \"kubernetes.io/projected/25cb40cf-a748-4af8-8aab-a2c9c75b53dc-kube-api-access-d88z7\") pod \"payload-processing-external-model-6c6d95cc8-gk8xd\" (UID: \"25cb40cf-a748-4af8-8aab-a2c9c75b53dc\") " pod="models-as-a-service/payload-processing-external-model-6c6d95cc8-gk8xd" Oct 31 04:18:45.103605 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:45.103565 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6c6d95cc8-gk8xd" Oct 31 04:18:45.614944 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:18:45.614909 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25cb40cf_a748_4af8_8aab_a2c9c75b53dc.slice/crio-e4bee872ab473c36032c783d57183cb0d591d4f3fa5cdca1dcdf303c2d91f821 WatchSource:0}: Error finding container e4bee872ab473c36032c783d57183cb0d591d4f3fa5cdca1dcdf303c2d91f821: Status 404 returned error can't find the container with id e4bee872ab473c36032c783d57183cb0d591d4f3fa5cdca1dcdf303c2d91f821 Oct 31 04:18:45.617499 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:45.616147 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6c6d95cc8-gk8xd"] Oct 31 04:18:46.013299 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:46.013265 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6c6d95cc8-gk8xd" event={"ID":"25cb40cf-a748-4af8-8aab-a2c9c75b53dc","Type":"ContainerStarted","Data":"59ac2c541fd32cec62991e72540f21d4271809ae11cda06cd59c40a1ccd7fd6d"} Oct 31 04:18:46.013729 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:46.013315 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6c6d95cc8-gk8xd" event={"ID":"25cb40cf-a748-4af8-8aab-a2c9c75b53dc","Type":"ContainerStarted","Data":"e4bee872ab473c36032c783d57183cb0d591d4f3fa5cdca1dcdf303c2d91f821"} Oct 31 04:18:46.013729 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:46.013402 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-6c6d95cc8-gk8xd" Oct 31 04:18:46.058964 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:46.058907 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-6c6d95cc8-gk8xd" podStartSLOduration=2.058894829 podStartE2EDuration="2.058894829s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:18:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:18:46.058281929 +0000 UTC m=+873.772616546" watchObservedRunningTime="2026-10-31 04:18:46.058894829 +0000 UTC m=+873.773229446" Oct 31 04:18:47.511758 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:47.511719 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-09a20a/e2e-conflict-a-ba60a444-kserve-78b9f5789c-kjs4g" podUID="79160b8c-f9b5-4cfa-8d3f-31da9a42dbe5" containerName="main" containerID="cri-o://f34d596fa8ad272f83fbdfc9b50b50bcaa1be1cf2ce8973d78d5bd9c5abe57e9" gracePeriod=45 Oct 31 04:18:47.786132 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:47.786109 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-09a20a/e2e-conflict-a-ba60a444-kserve-78b9f5789c-kjs4g" Oct 31 04:18:47.938147 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:47.938113 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79160b8c-f9b5-4cfa-8d3f-31da9a42dbe5-home\" (UniqueName: \"kubernetes.io/empty-dir/79160b8c-f9b5-4cfa-8d3f-31da9a42dbe5-home\") pod \"79160b8c-f9b5-4cfa-8d3f-31da9a42dbe5\" (UID: \"79160b8c-f9b5-4cfa-8d3f-31da9a42dbe5\") " Oct 31 04:18:47.938369 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:47.938182 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79160b8c-f9b5-4cfa-8d3f-31da9a42dbe5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/79160b8c-f9b5-4cfa-8d3f-31da9a42dbe5-dshm\") pod \"79160b8c-f9b5-4cfa-8d3f-31da9a42dbe5\" (UID: \"79160b8c-f9b5-4cfa-8d3f-31da9a42dbe5\") " Oct 31 04:18:47.938369 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:47.938270 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79160b8c-f9b5-4cfa-8d3f-31da9a42dbe5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79160b8c-f9b5-4cfa-8d3f-31da9a42dbe5-model-cache\") pod \"79160b8c-f9b5-4cfa-8d3f-31da9a42dbe5\" (UID: \"79160b8c-f9b5-4cfa-8d3f-31da9a42dbe5\") " Oct 31 04:18:47.938369 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:47.938317 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79160b8c-f9b5-4cfa-8d3f-31da9a42dbe5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79160b8c-f9b5-4cfa-8d3f-31da9a42dbe5-tmp-dir\") pod \"79160b8c-f9b5-4cfa-8d3f-31da9a42dbe5\" (UID: \"79160b8c-f9b5-4cfa-8d3f-31da9a42dbe5\") " Oct 31 04:18:47.938369 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:47.938362 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79160b8c-f9b5-4cfa-8d3f-31da9a42dbe5-kube-api-access-tnnsr\" (UniqueName: \"kubernetes.io/projected/79160b8c-f9b5-4cfa-8d3f-31da9a42dbe5-kube-api-access-tnnsr\") pod \"79160b8c-f9b5-4cfa-8d3f-31da9a42dbe5\" (UID: \"79160b8c-f9b5-4cfa-8d3f-31da9a42dbe5\") " Oct 31 04:18:47.938597 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:47.938446 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/79160b8c-f9b5-4cfa-8d3f-31da9a42dbe5-tls-certs\" (UniqueName: \"kubernetes.io/secret/79160b8c-f9b5-4cfa-8d3f-31da9a42dbe5-tls-certs\") pod \"79160b8c-f9b5-4cfa-8d3f-31da9a42dbe5\" (UID: \"79160b8c-f9b5-4cfa-8d3f-31da9a42dbe5\") " Oct 31 04:18:47.938597 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:47.938461 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79160b8c-f9b5-4cfa-8d3f-31da9a42dbe5-home" pod "79160b8c-f9b5-4cfa-8d3f-31da9a42dbe5" (UID: "79160b8c-f9b5-4cfa-8d3f-31da9a42dbe5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:18:47.938695 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:47.938608 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79160b8c-f9b5-4cfa-8d3f-31da9a42dbe5-tmp-dir" pod "79160b8c-f9b5-4cfa-8d3f-31da9a42dbe5" (UID: "79160b8c-f9b5-4cfa-8d3f-31da9a42dbe5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:18:47.938695 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:47.938622 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79160b8c-f9b5-4cfa-8d3f-31da9a42dbe5-model-cache" pod "79160b8c-f9b5-4cfa-8d3f-31da9a42dbe5" (UID: "79160b8c-f9b5-4cfa-8d3f-31da9a42dbe5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:18:47.938850 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:47.938831 4376 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79160b8c-f9b5-4cfa-8d3f-31da9a42dbe5-model-cache\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:18:47.938850 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:47.938849 4376 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79160b8c-f9b5-4cfa-8d3f-31da9a42dbe5-tmp-dir\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:18:47.938945 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:47.938857 4376 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/79160b8c-f9b5-4cfa-8d3f-31da9a42dbe5-home\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:18:47.940542 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:47.940523 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79160b8c-f9b5-4cfa-8d3f-31da9a42dbe5-dshm" pod "79160b8c-f9b5-4cfa-8d3f-31da9a42dbe5" (UID: "79160b8c-f9b5-4cfa-8d3f-31da9a42dbe5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:18:47.940636 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:47.940613 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79160b8c-f9b5-4cfa-8d3f-31da9a42dbe5-kube-api-access-tnnsr" pod "79160b8c-f9b5-4cfa-8d3f-31da9a42dbe5" (UID: "79160b8c-f9b5-4cfa-8d3f-31da9a42dbe5"). InnerVolumeSpecName "kube-api-access-tnnsr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:18:47.940674 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:47.940629 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79160b8c-f9b5-4cfa-8d3f-31da9a42dbe5-tls-certs" pod "79160b8c-f9b5-4cfa-8d3f-31da9a42dbe5" (UID: "79160b8c-f9b5-4cfa-8d3f-31da9a42dbe5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:18:48.026191 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:48.026092 4376 generic.go:356] "Generic (PLEG): container finished" podID="79160b8c-f9b5-4cfa-8d3f-31da9a42dbe5" containerID="f34d596fa8ad272f83fbdfc9b50b50bcaa1be1cf2ce8973d78d5bd9c5abe57e9" exitCode=0 Oct 31 04:18:48.026191 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:48.026144 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-09a20a/e2e-conflict-a-ba60a444-kserve-78b9f5789c-kjs4g" Oct 31 04:18:48.026191 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:48.026166 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-09a20a/e2e-conflict-a-ba60a444-kserve-78b9f5789c-kjs4g" event={"ID":"79160b8c-f9b5-4cfa-8d3f-31da9a42dbe5","Type":"ContainerDied","Data":"f34d596fa8ad272f83fbdfc9b50b50bcaa1be1cf2ce8973d78d5bd9c5abe57e9"} Oct 31 04:18:48.026497 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:48.026226 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-09a20a/e2e-conflict-a-ba60a444-kserve-78b9f5789c-kjs4g" event={"ID":"79160b8c-f9b5-4cfa-8d3f-31da9a42dbe5","Type":"ContainerDied","Data":"e36a28e29189480055d87135423c3e786f1fbde01d8f1317735d5ed1feabb180"} Oct 31 04:18:48.026497 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:48.026247 4376 scope.go:122] "RemoveContainer" containerID="f34d596fa8ad272f83fbdfc9b50b50bcaa1be1cf2ce8973d78d5bd9c5abe57e9" Oct 31 04:18:48.036371 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:48.036353 4376 scope.go:122] "RemoveContainer" containerID="f34d596fa8ad272f83fbdfc9b50b50bcaa1be1cf2ce8973d78d5bd9c5abe57e9" Oct 31 04:18:48.036637 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:18:48.036610 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f34d596fa8ad272f83fbdfc9b50b50bcaa1be1cf2ce8973d78d5bd9c5abe57e9\": container with ID starting with f34d596fa8ad272f83fbdfc9b50b50bcaa1be1cf2ce8973d78d5bd9c5abe57e9 not found: ID does not exist" containerID="f34d596fa8ad272f83fbdfc9b50b50bcaa1be1cf2ce8973d78d5bd9c5abe57e9" Oct 31 04:18:48.036746 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:48.036642 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f34d596fa8ad272f83fbdfc9b50b50bcaa1be1cf2ce8973d78d5bd9c5abe57e9"} err="failed to get container status \"f34d596fa8ad272f83fbdfc9b50b50bcaa1be1cf2ce8973d78d5bd9c5abe57e9\": rpc error: code = NotFound desc = could not find container \"f34d596fa8ad272f83fbdfc9b50b50bcaa1be1cf2ce8973d78d5bd9c5abe57e9\": container with ID starting with f34d596fa8ad272f83fbdfc9b50b50bcaa1be1cf2ce8973d78d5bd9c5abe57e9 not found: ID does not exist" Oct 31 04:18:48.040355 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:48.040311 4376 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/79160b8c-f9b5-4cfa-8d3f-31da9a42dbe5-dshm\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:18:48.040456 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:48.040363 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tnnsr\" (UniqueName: \"kubernetes.io/projected/79160b8c-f9b5-4cfa-8d3f-31da9a42dbe5-kube-api-access-tnnsr\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:18:48.040456 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:48.040374 4376 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79160b8c-f9b5-4cfa-8d3f-31da9a42dbe5-tls-certs\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:18:48.056173 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:48.056143 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-c4006d/distinct-2-w4-c4006d-kserve-776b5b899-kpl6z"] Oct 31 04:18:48.056913 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:48.056894 4376 cpu_manager.go:414] "containerMap: removing container" podUID="79160b8c-f9b5-4cfa-8d3f-31da9a42dbe5" containerName="main" Oct 31 04:18:48.056985 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:48.056918 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79160b8c-f9b5-4cfa-8d3f-31da9a42dbe5" containerName="main" Oct 31 04:18:48.057108 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:48.057095 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="79160b8c-f9b5-4cfa-8d3f-31da9a42dbe5" containerName="main" Oct 31 04:18:48.062316 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:48.062294 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c4006d/distinct-2-w4-c4006d-kserve-776b5b899-kpl6z" Oct 31 04:18:48.077718 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:48.077695 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-c4006d/distinct-2-w4-c4006d-kserve-776b5b899-kpl6z"] Oct 31 04:18:48.079535 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:48.079512 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-c4006d\"/\"distinct-2-w4-c4006d-kserve-self-signed-certs\"" Oct 31 04:18:48.089590 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:48.089563 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-09a20a/e2e-conflict-a-ba60a444-kserve-78b9f5789c-kjs4g"] Oct 31 04:18:48.091825 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:48.091807 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-09a20a/e2e-conflict-a-ba60a444-kserve-78b9f5789c-kjs4g"] Oct 31 04:18:48.244626 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:48.244445 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc53083d-eadd-4cb8-8633-35e02d400b9c-tmp-dir\") pod \"distinct-2-w4-c4006d-kserve-776b5b899-kpl6z\" (UID: \"dc53083d-eadd-4cb8-8633-35e02d400b9c\") " pod="e2e-models-e2e-worker-w4-c4006d/distinct-2-w4-c4006d-kserve-776b5b899-kpl6z" Oct 31 04:18:48.244626 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:48.244529 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc53083d-eadd-4cb8-8633-35e02d400b9c-home\") pod \"distinct-2-w4-c4006d-kserve-776b5b899-kpl6z\" (UID: \"dc53083d-eadd-4cb8-8633-35e02d400b9c\") " pod="e2e-models-e2e-worker-w4-c4006d/distinct-2-w4-c4006d-kserve-776b5b899-kpl6z" Oct 31 04:18:48.244626 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:48.244559 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc53083d-eadd-4cb8-8633-35e02d400b9c-tls-certs\") pod \"distinct-2-w4-c4006d-kserve-776b5b899-kpl6z\" (UID: \"dc53083d-eadd-4cb8-8633-35e02d400b9c\") " pod="e2e-models-e2e-worker-w4-c4006d/distinct-2-w4-c4006d-kserve-776b5b899-kpl6z" Oct 31 04:18:48.245648 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:48.244659 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc53083d-eadd-4cb8-8633-35e02d400b9c-model-cache\") pod \"distinct-2-w4-c4006d-kserve-776b5b899-kpl6z\" (UID: \"dc53083d-eadd-4cb8-8633-35e02d400b9c\") " pod="e2e-models-e2e-worker-w4-c4006d/distinct-2-w4-c4006d-kserve-776b5b899-kpl6z" Oct 31 04:18:48.245648 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:48.244758 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc53083d-eadd-4cb8-8633-35e02d400b9c-dshm\") pod \"distinct-2-w4-c4006d-kserve-776b5b899-kpl6z\" (UID: \"dc53083d-eadd-4cb8-8633-35e02d400b9c\") " pod="e2e-models-e2e-worker-w4-c4006d/distinct-2-w4-c4006d-kserve-776b5b899-kpl6z" Oct 31 04:18:48.245648 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:48.244933 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mn7kq\" (UniqueName: \"kubernetes.io/projected/dc53083d-eadd-4cb8-8633-35e02d400b9c-kube-api-access-mn7kq\") pod \"distinct-2-w4-c4006d-kserve-776b5b899-kpl6z\" (UID: \"dc53083d-eadd-4cb8-8633-35e02d400b9c\") " pod="e2e-models-e2e-worker-w4-c4006d/distinct-2-w4-c4006d-kserve-776b5b899-kpl6z" Oct 31 04:18:48.346233 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:48.346151 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mn7kq\" (UniqueName: \"kubernetes.io/projected/dc53083d-eadd-4cb8-8633-35e02d400b9c-kube-api-access-mn7kq\") pod \"distinct-2-w4-c4006d-kserve-776b5b899-kpl6z\" (UID: \"dc53083d-eadd-4cb8-8633-35e02d400b9c\") " pod="e2e-models-e2e-worker-w4-c4006d/distinct-2-w4-c4006d-kserve-776b5b899-kpl6z" Oct 31 04:18:48.346233 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:48.346227 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc53083d-eadd-4cb8-8633-35e02d400b9c-tmp-dir\") pod \"distinct-2-w4-c4006d-kserve-776b5b899-kpl6z\" (UID: \"dc53083d-eadd-4cb8-8633-35e02d400b9c\") " pod="e2e-models-e2e-worker-w4-c4006d/distinct-2-w4-c4006d-kserve-776b5b899-kpl6z" Oct 31 04:18:48.346498 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:48.346248 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc53083d-eadd-4cb8-8633-35e02d400b9c-home\") pod \"distinct-2-w4-c4006d-kserve-776b5b899-kpl6z\" (UID: \"dc53083d-eadd-4cb8-8633-35e02d400b9c\") " pod="e2e-models-e2e-worker-w4-c4006d/distinct-2-w4-c4006d-kserve-776b5b899-kpl6z" Oct 31 04:18:48.346498 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:48.346265 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc53083d-eadd-4cb8-8633-35e02d400b9c-tls-certs\") pod \"distinct-2-w4-c4006d-kserve-776b5b899-kpl6z\" (UID: \"dc53083d-eadd-4cb8-8633-35e02d400b9c\") " pod="e2e-models-e2e-worker-w4-c4006d/distinct-2-w4-c4006d-kserve-776b5b899-kpl6z" Oct 31 04:18:48.346498 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:48.346284 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc53083d-eadd-4cb8-8633-35e02d400b9c-model-cache\") pod \"distinct-2-w4-c4006d-kserve-776b5b899-kpl6z\" (UID: \"dc53083d-eadd-4cb8-8633-35e02d400b9c\") " pod="e2e-models-e2e-worker-w4-c4006d/distinct-2-w4-c4006d-kserve-776b5b899-kpl6z" Oct 31 04:18:48.346498 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:48.346353 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc53083d-eadd-4cb8-8633-35e02d400b9c-dshm\") pod \"distinct-2-w4-c4006d-kserve-776b5b899-kpl6z\" (UID: \"dc53083d-eadd-4cb8-8633-35e02d400b9c\") " pod="e2e-models-e2e-worker-w4-c4006d/distinct-2-w4-c4006d-kserve-776b5b899-kpl6z" Oct 31 04:18:48.346733 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:48.346706 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc53083d-eadd-4cb8-8633-35e02d400b9c-home\") pod \"distinct-2-w4-c4006d-kserve-776b5b899-kpl6z\" (UID: \"dc53083d-eadd-4cb8-8633-35e02d400b9c\") " pod="e2e-models-e2e-worker-w4-c4006d/distinct-2-w4-c4006d-kserve-776b5b899-kpl6z" Oct 31 04:18:48.346793 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:48.346764 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc53083d-eadd-4cb8-8633-35e02d400b9c-tmp-dir\") pod \"distinct-2-w4-c4006d-kserve-776b5b899-kpl6z\" (UID: \"dc53083d-eadd-4cb8-8633-35e02d400b9c\") " pod="e2e-models-e2e-worker-w4-c4006d/distinct-2-w4-c4006d-kserve-776b5b899-kpl6z" Oct 31 04:18:48.346849 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:48.346832 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc53083d-eadd-4cb8-8633-35e02d400b9c-model-cache\") pod \"distinct-2-w4-c4006d-kserve-776b5b899-kpl6z\" (UID: \"dc53083d-eadd-4cb8-8633-35e02d400b9c\") " pod="e2e-models-e2e-worker-w4-c4006d/distinct-2-w4-c4006d-kserve-776b5b899-kpl6z" Oct 31 04:18:48.349419 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:48.349395 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc53083d-eadd-4cb8-8633-35e02d400b9c-tls-certs\") pod \"distinct-2-w4-c4006d-kserve-776b5b899-kpl6z\" (UID: \"dc53083d-eadd-4cb8-8633-35e02d400b9c\") " pod="e2e-models-e2e-worker-w4-c4006d/distinct-2-w4-c4006d-kserve-776b5b899-kpl6z" Oct 31 04:18:48.349588 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:48.349571 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc53083d-eadd-4cb8-8633-35e02d400b9c-dshm\") pod \"distinct-2-w4-c4006d-kserve-776b5b899-kpl6z\" (UID: \"dc53083d-eadd-4cb8-8633-35e02d400b9c\") " pod="e2e-models-e2e-worker-w4-c4006d/distinct-2-w4-c4006d-kserve-776b5b899-kpl6z" Oct 31 04:18:48.371339 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:48.371300 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mn7kq\" (UniqueName: \"kubernetes.io/projected/dc53083d-eadd-4cb8-8633-35e02d400b9c-kube-api-access-mn7kq\") pod \"distinct-2-w4-c4006d-kserve-776b5b899-kpl6z\" (UID: \"dc53083d-eadd-4cb8-8633-35e02d400b9c\") " pod="e2e-models-e2e-worker-w4-c4006d/distinct-2-w4-c4006d-kserve-776b5b899-kpl6z" Oct 31 04:18:48.376974 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:48.376948 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c4006d/distinct-2-w4-c4006d-kserve-776b5b899-kpl6z" Oct 31 04:18:48.448486 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:48.448451 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79160b8c-f9b5-4cfa-8d3f-31da9a42dbe5" path="/var/lib/kubelet/pods/79160b8c-f9b5-4cfa-8d3f-31da9a42dbe5/volumes" Oct 31 04:18:48.682619 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:18:48.682584 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddc53083d_eadd_4cb8_8633_35e02d400b9c.slice/crio-d94caeb527da1f56f2dca6f903744bf860fea59a34a0ab66b8615737df500ea9 WatchSource:0}: Error finding container d94caeb527da1f56f2dca6f903744bf860fea59a34a0ab66b8615737df500ea9: Status 404 returned error can't find the container with id d94caeb527da1f56f2dca6f903744bf860fea59a34a0ab66b8615737df500ea9 Oct 31 04:18:48.683585 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:48.683563 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-c4006d/distinct-2-w4-c4006d-kserve-776b5b899-kpl6z"] Oct 31 04:18:49.040842 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:49.040748 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c4006d/distinct-2-w4-c4006d-kserve-776b5b899-kpl6z" event={"ID":"dc53083d-eadd-4cb8-8633-35e02d400b9c","Type":"ContainerStarted","Data":"6c20446894934dcfa9b40993be9fa2a23769d15a81224a240f6cbcc76caba357"} Oct 31 04:18:49.040842 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:49.040790 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c4006d/distinct-2-w4-c4006d-kserve-776b5b899-kpl6z" event={"ID":"dc53083d-eadd-4cb8-8633-35e02d400b9c","Type":"ContainerStarted","Data":"d94caeb527da1f56f2dca6f903744bf860fea59a34a0ab66b8615737df500ea9"} Oct 31 04:18:49.086082 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:49.086028 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-c4006d/distinct-2-w4-c4006d-kserve-776b5b899-kpl6z" podStartSLOduration=1.086015668 podStartE2EDuration="1.086015668s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:18:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:18:49.085541456 +0000 UTC m=+876.799876097" watchObservedRunningTime="2026-10-31 04:18:49.086015668 +0000 UTC m=+876.800350284" Oct 31 04:18:49.377963 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:49.377929 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c4006d/distinct-2-w4-c4006d-kserve-776b5b899-kpl6z" Oct 31 04:18:54.477474 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:54.477430 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-09a20a/distinct-w2-09a20a-kserve-76557f6448-l8d69"] Oct 31 04:18:54.482548 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:54.482522 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-09a20a/distinct-w2-09a20a-kserve-76557f6448-l8d69" Oct 31 04:18:54.499845 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:54.499812 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-09a20a/distinct-w2-09a20a-kserve-76557f6448-l8d69"] Oct 31 04:18:54.511593 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:54.511557 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-09a20a\"/\"distinct-w2-09a20a-kserve-self-signed-certs\"" Oct 31 04:18:54.523130 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:54.523099 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d298000-b65b-45fd-a1ee-092e47a7eb24-model-cache\") pod \"distinct-w2-09a20a-kserve-76557f6448-l8d69\" (UID: \"7d298000-b65b-45fd-a1ee-092e47a7eb24\") " pod="e2e-models-e2e-worker-w2-09a20a/distinct-w2-09a20a-kserve-76557f6448-l8d69" Oct 31 04:18:54.523299 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:54.523234 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d298000-b65b-45fd-a1ee-092e47a7eb24-home\") pod \"distinct-w2-09a20a-kserve-76557f6448-l8d69\" (UID: \"7d298000-b65b-45fd-a1ee-092e47a7eb24\") " pod="e2e-models-e2e-worker-w2-09a20a/distinct-w2-09a20a-kserve-76557f6448-l8d69" Oct 31 04:18:54.523398 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:54.523313 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d298000-b65b-45fd-a1ee-092e47a7eb24-tmp-dir\") pod \"distinct-w2-09a20a-kserve-76557f6448-l8d69\" (UID: \"7d298000-b65b-45fd-a1ee-092e47a7eb24\") " pod="e2e-models-e2e-worker-w2-09a20a/distinct-w2-09a20a-kserve-76557f6448-l8d69" Oct 31 04:18:54.523398 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:54.523368 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d298000-b65b-45fd-a1ee-092e47a7eb24-tls-certs\") pod \"distinct-w2-09a20a-kserve-76557f6448-l8d69\" (UID: \"7d298000-b65b-45fd-a1ee-092e47a7eb24\") " pod="e2e-models-e2e-worker-w2-09a20a/distinct-w2-09a20a-kserve-76557f6448-l8d69" Oct 31 04:18:54.523591 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:54.523567 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bsxlc\" (UniqueName: \"kubernetes.io/projected/7d298000-b65b-45fd-a1ee-092e47a7eb24-kube-api-access-bsxlc\") pod \"distinct-w2-09a20a-kserve-76557f6448-l8d69\" (UID: \"7d298000-b65b-45fd-a1ee-092e47a7eb24\") " pod="e2e-models-e2e-worker-w2-09a20a/distinct-w2-09a20a-kserve-76557f6448-l8d69" Oct 31 04:18:54.523722 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:54.523703 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d298000-b65b-45fd-a1ee-092e47a7eb24-dshm\") pod \"distinct-w2-09a20a-kserve-76557f6448-l8d69\" (UID: \"7d298000-b65b-45fd-a1ee-092e47a7eb24\") " pod="e2e-models-e2e-worker-w2-09a20a/distinct-w2-09a20a-kserve-76557f6448-l8d69" Oct 31 04:18:54.624849 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:54.624814 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d298000-b65b-45fd-a1ee-092e47a7eb24-model-cache\") pod \"distinct-w2-09a20a-kserve-76557f6448-l8d69\" (UID: \"7d298000-b65b-45fd-a1ee-092e47a7eb24\") " pod="e2e-models-e2e-worker-w2-09a20a/distinct-w2-09a20a-kserve-76557f6448-l8d69" Oct 31 04:18:54.625086 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:54.624868 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d298000-b65b-45fd-a1ee-092e47a7eb24-home\") pod \"distinct-w2-09a20a-kserve-76557f6448-l8d69\" (UID: \"7d298000-b65b-45fd-a1ee-092e47a7eb24\") " pod="e2e-models-e2e-worker-w2-09a20a/distinct-w2-09a20a-kserve-76557f6448-l8d69" Oct 31 04:18:54.625086 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:54.624897 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d298000-b65b-45fd-a1ee-092e47a7eb24-tmp-dir\") pod \"distinct-w2-09a20a-kserve-76557f6448-l8d69\" (UID: \"7d298000-b65b-45fd-a1ee-092e47a7eb24\") " pod="e2e-models-e2e-worker-w2-09a20a/distinct-w2-09a20a-kserve-76557f6448-l8d69" Oct 31 04:18:54.625086 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:54.624918 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d298000-b65b-45fd-a1ee-092e47a7eb24-tls-certs\") pod \"distinct-w2-09a20a-kserve-76557f6448-l8d69\" (UID: \"7d298000-b65b-45fd-a1ee-092e47a7eb24\") " pod="e2e-models-e2e-worker-w2-09a20a/distinct-w2-09a20a-kserve-76557f6448-l8d69" Oct 31 04:18:54.625086 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:54.624965 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsxlc\" (UniqueName: \"kubernetes.io/projected/7d298000-b65b-45fd-a1ee-092e47a7eb24-kube-api-access-bsxlc\") pod \"distinct-w2-09a20a-kserve-76557f6448-l8d69\" (UID: \"7d298000-b65b-45fd-a1ee-092e47a7eb24\") " pod="e2e-models-e2e-worker-w2-09a20a/distinct-w2-09a20a-kserve-76557f6448-l8d69" Oct 31 04:18:54.625086 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:54.625023 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d298000-b65b-45fd-a1ee-092e47a7eb24-dshm\") pod \"distinct-w2-09a20a-kserve-76557f6448-l8d69\" (UID: \"7d298000-b65b-45fd-a1ee-092e47a7eb24\") " pod="e2e-models-e2e-worker-w2-09a20a/distinct-w2-09a20a-kserve-76557f6448-l8d69" Oct 31 04:18:54.625446 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:54.625293 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d298000-b65b-45fd-a1ee-092e47a7eb24-model-cache\") pod \"distinct-w2-09a20a-kserve-76557f6448-l8d69\" (UID: \"7d298000-b65b-45fd-a1ee-092e47a7eb24\") " pod="e2e-models-e2e-worker-w2-09a20a/distinct-w2-09a20a-kserve-76557f6448-l8d69" Oct 31 04:18:54.625446 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:54.625316 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d298000-b65b-45fd-a1ee-092e47a7eb24-tmp-dir\") pod \"distinct-w2-09a20a-kserve-76557f6448-l8d69\" (UID: \"7d298000-b65b-45fd-a1ee-092e47a7eb24\") " pod="e2e-models-e2e-worker-w2-09a20a/distinct-w2-09a20a-kserve-76557f6448-l8d69" Oct 31 04:18:54.625446 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:54.625400 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d298000-b65b-45fd-a1ee-092e47a7eb24-home\") pod \"distinct-w2-09a20a-kserve-76557f6448-l8d69\" (UID: \"7d298000-b65b-45fd-a1ee-092e47a7eb24\") " pod="e2e-models-e2e-worker-w2-09a20a/distinct-w2-09a20a-kserve-76557f6448-l8d69" Oct 31 04:18:54.627305 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:54.627283 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d298000-b65b-45fd-a1ee-092e47a7eb24-dshm\") pod \"distinct-w2-09a20a-kserve-76557f6448-l8d69\" (UID: \"7d298000-b65b-45fd-a1ee-092e47a7eb24\") " pod="e2e-models-e2e-worker-w2-09a20a/distinct-w2-09a20a-kserve-76557f6448-l8d69" Oct 31 04:18:54.627511 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:54.627495 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d298000-b65b-45fd-a1ee-092e47a7eb24-tls-certs\") pod \"distinct-w2-09a20a-kserve-76557f6448-l8d69\" (UID: \"7d298000-b65b-45fd-a1ee-092e47a7eb24\") " pod="e2e-models-e2e-worker-w2-09a20a/distinct-w2-09a20a-kserve-76557f6448-l8d69" Oct 31 04:18:54.651027 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:54.650999 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsxlc\" (UniqueName: \"kubernetes.io/projected/7d298000-b65b-45fd-a1ee-092e47a7eb24-kube-api-access-bsxlc\") pod \"distinct-w2-09a20a-kserve-76557f6448-l8d69\" (UID: \"7d298000-b65b-45fd-a1ee-092e47a7eb24\") " pod="e2e-models-e2e-worker-w2-09a20a/distinct-w2-09a20a-kserve-76557f6448-l8d69" Oct 31 04:18:54.798813 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:54.798722 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-09a20a/distinct-w2-09a20a-kserve-76557f6448-l8d69" Oct 31 04:18:55.100313 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:18:55.100284 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7d298000_b65b_45fd_a1ee_092e47a7eb24.slice/crio-3dac9adf2b9b205a6c67b610fbd8b11348dd2b08dd8632743c7a1d138db8a9ac WatchSource:0}: Error finding container 3dac9adf2b9b205a6c67b610fbd8b11348dd2b08dd8632743c7a1d138db8a9ac: Status 404 returned error can't find the container with id 3dac9adf2b9b205a6c67b610fbd8b11348dd2b08dd8632743c7a1d138db8a9ac Oct 31 04:18:55.100568 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:55.100545 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-09a20a/distinct-w2-09a20a-kserve-76557f6448-l8d69"] Oct 31 04:18:55.639309 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:55.639271 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 04:18:56.082919 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:56.082832 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-09a20a/distinct-w2-09a20a-kserve-76557f6448-l8d69" event={"ID":"7d298000-b65b-45fd-a1ee-092e47a7eb24","Type":"ContainerStarted","Data":"c6f1417f936622a50af2a4c91f291b31122a0958bfaa10f70676fdbb15c3e4b2"} Oct 31 04:18:56.082919 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:56.082874 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-09a20a/distinct-w2-09a20a-kserve-76557f6448-l8d69" event={"ID":"7d298000-b65b-45fd-a1ee-092e47a7eb24","Type":"ContainerStarted","Data":"3dac9adf2b9b205a6c67b610fbd8b11348dd2b08dd8632743c7a1d138db8a9ac"} Oct 31 04:18:56.131950 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:56.131894 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-09a20a/distinct-w2-09a20a-kserve-76557f6448-l8d69" podStartSLOduration=2.131878679 podStartE2EDuration="2.131878679s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:18:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:18:56.130411482 +0000 UTC m=+883.844746100" watchObservedRunningTime="2026-10-31 04:18:56.131878679 +0000 UTC m=+883.846213296" Oct 31 04:18:56.799231 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:56.799192 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-09a20a/distinct-w2-09a20a-kserve-76557f6448-l8d69" Oct 31 04:18:57.057407 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:57.056535 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-6c6d95cc8-gk8xd" Oct 31 04:18:57.145406 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:57.145372 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-f445668d4-gmmrd"] Oct 31 04:18:57.242359 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:18:57.218528 4376 configmap.go:193] Couldn't get configMap models-as-a-service/payload-processing-external-model-plugins: configmap "payload-processing-external-model-plugins" not found Oct 31 04:18:57.242359 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:18:57.218588 4376 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-cdd9c78cff58: secret "agc-cred-cdd9c78cff58" not found Oct 31 04:18:57.242359 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:18:57.218608 4376 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-6c6d95cc8-gk8xd: secret "agc-cred-cdd9c78cff58" not found Oct 31 04:18:57.242359 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:18:57.218631 4376 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/25cb40cf-a748-4af8-8aab-a2c9c75b53dc-plugins-config-volume podName:25cb40cf-a748-4af8-8aab-a2c9c75b53dc nodeName:}" failed. No retries permitted until 2026-10-31 04:18:57.718607806 +0000 UTC m=+885.432942418 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/25cb40cf-a748-4af8-8aab-a2c9c75b53dc-plugins-config-volume") pod "payload-processing-external-model-6c6d95cc8-gk8xd" (UID: "25cb40cf-a748-4af8-8aab-a2c9c75b53dc") : configmap "payload-processing-external-model-plugins" not found Oct 31 04:18:57.242359 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:18:57.218669 4376 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/25cb40cf-a748-4af8-8aab-a2c9c75b53dc-provider-credentials podName:25cb40cf-a748-4af8-8aab-a2c9c75b53dc nodeName:}" failed. No retries permitted until 2026-10-31 04:18:57.718652752 +0000 UTC m=+885.432987352 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/25cb40cf-a748-4af8-8aab-a2c9c75b53dc-provider-credentials") pod "payload-processing-external-model-6c6d95cc8-gk8xd" (UID: "25cb40cf-a748-4af8-8aab-a2c9c75b53dc") : secret "agc-cred-cdd9c78cff58" not found Oct 31 04:18:57.242858 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:57.242827 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6c6d95cc8-gk8xd"] Oct 31 04:18:57.739375 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:18:57.732318 4376 configmap.go:193] Couldn't get configMap models-as-a-service/payload-processing-external-model-plugins: configmap "payload-processing-external-model-plugins" not found Oct 31 04:18:57.739375 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:18:57.732439 4376 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/25cb40cf-a748-4af8-8aab-a2c9c75b53dc-plugins-config-volume podName:25cb40cf-a748-4af8-8aab-a2c9c75b53dc nodeName:}" failed. No retries permitted until 2026-10-31 04:18:58.732414121 +0000 UTC m=+886.446748738 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/25cb40cf-a748-4af8-8aab-a2c9c75b53dc-plugins-config-volume") pod "payload-processing-external-model-6c6d95cc8-gk8xd" (UID: "25cb40cf-a748-4af8-8aab-a2c9c75b53dc") : configmap "payload-processing-external-model-plugins" not found Oct 31 04:18:57.739375 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:18:57.732904 4376 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-cdd9c78cff58: secret "agc-cred-cdd9c78cff58" not found Oct 31 04:18:57.739375 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:18:57.732920 4376 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-6c6d95cc8-gk8xd: secret "agc-cred-cdd9c78cff58" not found Oct 31 04:18:57.739375 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:18:57.732966 4376 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/25cb40cf-a748-4af8-8aab-a2c9c75b53dc-provider-credentials podName:25cb40cf-a748-4af8-8aab-a2c9c75b53dc nodeName:}" failed. No retries permitted until 2026-10-31 04:18:58.732950398 +0000 UTC m=+886.447285007 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/25cb40cf-a748-4af8-8aab-a2c9c75b53dc-provider-credentials") pod "payload-processing-external-model-6c6d95cc8-gk8xd" (UID: "25cb40cf-a748-4af8-8aab-a2c9c75b53dc") : secret "agc-cred-cdd9c78cff58" not found Oct 31 04:18:58.377313 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:58.377279 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-c4006d/distinct-2-w4-c4006d-kserve-776b5b899-kpl6z" Oct 31 04:18:58.391677 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:58.391639 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-c4006d/distinct-2-w4-c4006d-kserve-776b5b899-kpl6z" Oct 31 04:18:58.759087 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:18:58.759001 4376 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-cdd9c78cff58: secret "agc-cred-cdd9c78cff58" not found Oct 31 04:18:58.759087 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:18:58.759025 4376 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-6c6d95cc8-gk8xd: secret "agc-cred-cdd9c78cff58" not found Oct 31 04:18:58.759087 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:18:58.759028 4376 configmap.go:193] Couldn't get configMap models-as-a-service/payload-processing-external-model-plugins: configmap "payload-processing-external-model-plugins" not found Oct 31 04:18:58.759087 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:18:58.759081 4376 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/25cb40cf-a748-4af8-8aab-a2c9c75b53dc-provider-credentials podName:25cb40cf-a748-4af8-8aab-a2c9c75b53dc nodeName:}" failed. No retries permitted until 2026-10-31 04:19:00.759064421 +0000 UTC m=+888.473399016 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/25cb40cf-a748-4af8-8aab-a2c9c75b53dc-provider-credentials") pod "payload-processing-external-model-6c6d95cc8-gk8xd" (UID: "25cb40cf-a748-4af8-8aab-a2c9c75b53dc") : secret "agc-cred-cdd9c78cff58" not found Oct 31 04:18:58.759365 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:18:58.759095 4376 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/25cb40cf-a748-4af8-8aab-a2c9c75b53dc-plugins-config-volume podName:25cb40cf-a748-4af8-8aab-a2c9c75b53dc nodeName:}" failed. No retries permitted until 2026-10-31 04:19:00.759089299 +0000 UTC m=+888.473423894 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/25cb40cf-a748-4af8-8aab-a2c9c75b53dc-plugins-config-volume") pod "payload-processing-external-model-6c6d95cc8-gk8xd" (UID: "25cb40cf-a748-4af8-8aab-a2c9c75b53dc") : configmap "payload-processing-external-model-plugins" not found Oct 31 04:18:59.136299 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:18:59.136271 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c4006d/distinct-2-w4-c4006d-kserve-776b5b899-kpl6z" Oct 31 04:19:00.785925 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:19:00.785892 4376 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-cdd9c78cff58: secret "agc-cred-cdd9c78cff58" not found Oct 31 04:19:00.785925 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:19:00.785916 4376 configmap.go:193] Couldn't get configMap models-as-a-service/payload-processing-external-model-plugins: configmap "payload-processing-external-model-plugins" not found Oct 31 04:19:00.786384 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:19:00.785982 4376 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/25cb40cf-a748-4af8-8aab-a2c9c75b53dc-plugins-config-volume podName:25cb40cf-a748-4af8-8aab-a2c9c75b53dc nodeName:}" failed. No retries permitted until 2026-10-31 04:19:04.785966484 +0000 UTC m=+892.500301079 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/25cb40cf-a748-4af8-8aab-a2c9c75b53dc-plugins-config-volume") pod "payload-processing-external-model-6c6d95cc8-gk8xd" (UID: "25cb40cf-a748-4af8-8aab-a2c9c75b53dc") : configmap "payload-processing-external-model-plugins" not found Oct 31 04:19:00.786384 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:19:00.785920 4376 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-6c6d95cc8-gk8xd: secret "agc-cred-cdd9c78cff58" not found Oct 31 04:19:00.786384 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:19:00.786059 4376 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/25cb40cf-a748-4af8-8aab-a2c9c75b53dc-provider-credentials podName:25cb40cf-a748-4af8-8aab-a2c9c75b53dc nodeName:}" failed. No retries permitted until 2026-10-31 04:19:04.78604526 +0000 UTC m=+892.500379868 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/25cb40cf-a748-4af8-8aab-a2c9c75b53dc-provider-credentials") pod "payload-processing-external-model-6c6d95cc8-gk8xd" (UID: "25cb40cf-a748-4af8-8aab-a2c9c75b53dc") : secret "agc-cred-cdd9c78cff58" not found Oct 31 04:19:02.146102 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:02.146064 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-f445668d4-gmmrd" podUID="d306e0df-7e92-48f8-a089-8b130186db1d" containerName="payload-processing" containerID="cri-o://756286597bfccae50ba40d32b968c6b4803c2cd3834d1b0c1f3023e67e8292e2" gracePeriod=25 Oct 31 04:19:02.288553 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:02.288511 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-6c6d95cc8-gk8xd" podUID="25cb40cf-a748-4af8-8aab-a2c9c75b53dc" containerName="payload-processing" containerID="cri-o://59ac2c541fd32cec62991e72540f21d4271809ae11cda06cd59c40a1ccd7fd6d" gracePeriod=25 Oct 31 04:19:02.419741 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:02.419712 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-f445668d4-gmmrd" Oct 31 04:19:02.508538 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:02.506778 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d306e0df-7e92-48f8-a089-8b130186db1d-kube-api-access-zqmrn\" (UniqueName: \"kubernetes.io/projected/d306e0df-7e92-48f8-a089-8b130186db1d-kube-api-access-zqmrn\") pod \"d306e0df-7e92-48f8-a089-8b130186db1d\" (UID: \"d306e0df-7e92-48f8-a089-8b130186db1d\") " Oct 31 04:19:02.508538 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:02.506904 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d306e0df-7e92-48f8-a089-8b130186db1d-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d306e0df-7e92-48f8-a089-8b130186db1d-routing-overlay-volume\") pod \"d306e0df-7e92-48f8-a089-8b130186db1d\" (UID: \"d306e0df-7e92-48f8-a089-8b130186db1d\") " Oct 31 04:19:02.508538 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:02.506950 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d306e0df-7e92-48f8-a089-8b130186db1d-provider-credentials\" (UniqueName: \"kubernetes.io/projected/d306e0df-7e92-48f8-a089-8b130186db1d-provider-credentials\") pod \"d306e0df-7e92-48f8-a089-8b130186db1d\" (UID: \"d306e0df-7e92-48f8-a089-8b130186db1d\") " Oct 31 04:19:02.508538 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:02.507024 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d306e0df-7e92-48f8-a089-8b130186db1d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d306e0df-7e92-48f8-a089-8b130186db1d-plugins-config-volume\") pod \"d306e0df-7e92-48f8-a089-8b130186db1d\" (UID: \"d306e0df-7e92-48f8-a089-8b130186db1d\") " Oct 31 04:19:02.510358 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:02.510263 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d306e0df-7e92-48f8-a089-8b130186db1d-plugins-config-volume" pod "d306e0df-7e92-48f8-a089-8b130186db1d" (UID: "d306e0df-7e92-48f8-a089-8b130186db1d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:19:02.514448 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:02.514420 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d306e0df-7e92-48f8-a089-8b130186db1d-provider-credentials" pod "d306e0df-7e92-48f8-a089-8b130186db1d" (UID: "d306e0df-7e92-48f8-a089-8b130186db1d"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:19:02.514951 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:02.514911 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d306e0df-7e92-48f8-a089-8b130186db1d-routing-overlay-volume" pod "d306e0df-7e92-48f8-a089-8b130186db1d" (UID: "d306e0df-7e92-48f8-a089-8b130186db1d"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:19:02.531351 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:02.530902 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d306e0df-7e92-48f8-a089-8b130186db1d-kube-api-access-zqmrn" pod "d306e0df-7e92-48f8-a089-8b130186db1d" (UID: "d306e0df-7e92-48f8-a089-8b130186db1d"). InnerVolumeSpecName "kube-api-access-zqmrn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:19:02.610136 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:02.610114 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6c6d95cc8-gk8xd" Oct 31 04:19:02.610673 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:02.610647 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqmrn\" (UniqueName: \"kubernetes.io/projected/d306e0df-7e92-48f8-a089-8b130186db1d-kube-api-access-zqmrn\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:19:02.610773 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:02.610676 4376 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d306e0df-7e92-48f8-a089-8b130186db1d-routing-overlay-volume\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:19:02.610773 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:02.610695 4376 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d306e0df-7e92-48f8-a089-8b130186db1d-provider-credentials\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:19:02.610773 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:02.610710 4376 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d306e0df-7e92-48f8-a089-8b130186db1d-plugins-config-volume\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:19:02.711290 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:02.711207 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25cb40cf-a748-4af8-8aab-a2c9c75b53dc-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/25cb40cf-a748-4af8-8aab-a2c9c75b53dc-routing-overlay-volume\") pod \"25cb40cf-a748-4af8-8aab-a2c9c75b53dc\" (UID: \"25cb40cf-a748-4af8-8aab-a2c9c75b53dc\") " Oct 31 04:19:02.711290 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:02.711252 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25cb40cf-a748-4af8-8aab-a2c9c75b53dc-provider-credentials\" (UniqueName: \"kubernetes.io/projected/25cb40cf-a748-4af8-8aab-a2c9c75b53dc-provider-credentials\") pod \"25cb40cf-a748-4af8-8aab-a2c9c75b53dc\" (UID: \"25cb40cf-a748-4af8-8aab-a2c9c75b53dc\") " Oct 31 04:19:02.711290 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:02.711277 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25cb40cf-a748-4af8-8aab-a2c9c75b53dc-kube-api-access-d88z7\" (UniqueName: \"kubernetes.io/projected/25cb40cf-a748-4af8-8aab-a2c9c75b53dc-kube-api-access-d88z7\") pod \"25cb40cf-a748-4af8-8aab-a2c9c75b53dc\" (UID: \"25cb40cf-a748-4af8-8aab-a2c9c75b53dc\") " Oct 31 04:19:02.711585 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:02.711453 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25cb40cf-a748-4af8-8aab-a2c9c75b53dc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25cb40cf-a748-4af8-8aab-a2c9c75b53dc-plugins-config-volume\") pod \"25cb40cf-a748-4af8-8aab-a2c9c75b53dc\" (UID: \"25cb40cf-a748-4af8-8aab-a2c9c75b53dc\") " Oct 31 04:19:02.711735 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:02.711704 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25cb40cf-a748-4af8-8aab-a2c9c75b53dc-routing-overlay-volume" pod "25cb40cf-a748-4af8-8aab-a2c9c75b53dc" (UID: "25cb40cf-a748-4af8-8aab-a2c9c75b53dc"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:19:02.711864 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:02.711842 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25cb40cf-a748-4af8-8aab-a2c9c75b53dc-plugins-config-volume" pod "25cb40cf-a748-4af8-8aab-a2c9c75b53dc" (UID: "25cb40cf-a748-4af8-8aab-a2c9c75b53dc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:19:02.711917 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:02.711868 4376 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/25cb40cf-a748-4af8-8aab-a2c9c75b53dc-routing-overlay-volume\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:19:02.714033 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:02.714014 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25cb40cf-a748-4af8-8aab-a2c9c75b53dc-provider-credentials" pod "25cb40cf-a748-4af8-8aab-a2c9c75b53dc" (UID: "25cb40cf-a748-4af8-8aab-a2c9c75b53dc"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:19:02.714115 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:02.714046 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25cb40cf-a748-4af8-8aab-a2c9c75b53dc-kube-api-access-d88z7" pod "25cb40cf-a748-4af8-8aab-a2c9c75b53dc" (UID: "25cb40cf-a748-4af8-8aab-a2c9c75b53dc"). InnerVolumeSpecName "kube-api-access-d88z7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:19:02.812951 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:02.812917 4376 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/25cb40cf-a748-4af8-8aab-a2c9c75b53dc-provider-credentials\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:19:02.812951 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:02.812943 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d88z7\" (UniqueName: \"kubernetes.io/projected/25cb40cf-a748-4af8-8aab-a2c9c75b53dc-kube-api-access-d88z7\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:19:02.812951 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:02.812954 4376 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25cb40cf-a748-4af8-8aab-a2c9c75b53dc-plugins-config-volume\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:19:03.149212 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:03.149180 4376 generic.go:356] "Generic (PLEG): container finished" podID="d306e0df-7e92-48f8-a089-8b130186db1d" containerID="756286597bfccae50ba40d32b968c6b4803c2cd3834d1b0c1f3023e67e8292e2" exitCode=0 Oct 31 04:19:03.149792 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:03.149224 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-f445668d4-gmmrd" Oct 31 04:19:03.149792 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:03.149268 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-f445668d4-gmmrd" event={"ID":"d306e0df-7e92-48f8-a089-8b130186db1d","Type":"ContainerDied","Data":"756286597bfccae50ba40d32b968c6b4803c2cd3834d1b0c1f3023e67e8292e2"} Oct 31 04:19:03.149792 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:03.149343 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-f445668d4-gmmrd" event={"ID":"d306e0df-7e92-48f8-a089-8b130186db1d","Type":"ContainerDied","Data":"dbe53d3b2158307e586ee90e8804c0716f467d83bdc6e358e5c19563a18a7c53"} Oct 31 04:19:03.149792 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:03.149378 4376 scope.go:122] "RemoveContainer" containerID="756286597bfccae50ba40d32b968c6b4803c2cd3834d1b0c1f3023e67e8292e2" Oct 31 04:19:03.150803 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:03.150779 4376 generic.go:356] "Generic (PLEG): container finished" podID="25cb40cf-a748-4af8-8aab-a2c9c75b53dc" containerID="59ac2c541fd32cec62991e72540f21d4271809ae11cda06cd59c40a1ccd7fd6d" exitCode=0 Oct 31 04:19:03.150940 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:03.150862 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6c6d95cc8-gk8xd" Oct 31 04:19:03.150940 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:03.150881 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6c6d95cc8-gk8xd" event={"ID":"25cb40cf-a748-4af8-8aab-a2c9c75b53dc","Type":"ContainerDied","Data":"59ac2c541fd32cec62991e72540f21d4271809ae11cda06cd59c40a1ccd7fd6d"} Oct 31 04:19:03.150940 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:03.150913 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6c6d95cc8-gk8xd" event={"ID":"25cb40cf-a748-4af8-8aab-a2c9c75b53dc","Type":"ContainerDied","Data":"e4bee872ab473c36032c783d57183cb0d591d4f3fa5cdca1dcdf303c2d91f821"} Oct 31 04:19:03.159789 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:03.159769 4376 scope.go:122] "RemoveContainer" containerID="756286597bfccae50ba40d32b968c6b4803c2cd3834d1b0c1f3023e67e8292e2" Oct 31 04:19:03.160059 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:19:03.160029 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"756286597bfccae50ba40d32b968c6b4803c2cd3834d1b0c1f3023e67e8292e2\": container with ID starting with 756286597bfccae50ba40d32b968c6b4803c2cd3834d1b0c1f3023e67e8292e2 not found: ID does not exist" containerID="756286597bfccae50ba40d32b968c6b4803c2cd3834d1b0c1f3023e67e8292e2" Oct 31 04:19:03.160150 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:03.160058 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"756286597bfccae50ba40d32b968c6b4803c2cd3834d1b0c1f3023e67e8292e2"} err="failed to get container status \"756286597bfccae50ba40d32b968c6b4803c2cd3834d1b0c1f3023e67e8292e2\": rpc error: code = NotFound desc = could not find container \"756286597bfccae50ba40d32b968c6b4803c2cd3834d1b0c1f3023e67e8292e2\": container with ID starting with 756286597bfccae50ba40d32b968c6b4803c2cd3834d1b0c1f3023e67e8292e2 not found: ID does not exist" Oct 31 04:19:03.160150 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:03.160078 4376 scope.go:122] "RemoveContainer" containerID="59ac2c541fd32cec62991e72540f21d4271809ae11cda06cd59c40a1ccd7fd6d" Oct 31 04:19:03.180545 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:03.180527 4376 scope.go:122] "RemoveContainer" containerID="59ac2c541fd32cec62991e72540f21d4271809ae11cda06cd59c40a1ccd7fd6d" Oct 31 04:19:03.180815 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:19:03.180798 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59ac2c541fd32cec62991e72540f21d4271809ae11cda06cd59c40a1ccd7fd6d\": container with ID starting with 59ac2c541fd32cec62991e72540f21d4271809ae11cda06cd59c40a1ccd7fd6d not found: ID does not exist" containerID="59ac2c541fd32cec62991e72540f21d4271809ae11cda06cd59c40a1ccd7fd6d" Oct 31 04:19:03.180860 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:03.180822 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59ac2c541fd32cec62991e72540f21d4271809ae11cda06cd59c40a1ccd7fd6d"} err="failed to get container status \"59ac2c541fd32cec62991e72540f21d4271809ae11cda06cd59c40a1ccd7fd6d\": rpc error: code = NotFound desc = could not find container \"59ac2c541fd32cec62991e72540f21d4271809ae11cda06cd59c40a1ccd7fd6d\": container with ID starting with 59ac2c541fd32cec62991e72540f21d4271809ae11cda06cd59c40a1ccd7fd6d not found: ID does not exist" Oct 31 04:19:03.225936 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:03.225903 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6c6d95cc8-gk8xd"] Oct 31 04:19:03.240564 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:03.240541 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-6c6d95cc8-gk8xd"] Oct 31 04:19:03.269224 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:03.269200 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-f445668d4-gmmrd"] Oct 31 04:19:03.273083 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:03.273064 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-f445668d4-gmmrd"] Oct 31 04:19:04.442817 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:04.442784 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25cb40cf-a748-4af8-8aab-a2c9c75b53dc" path="/var/lib/kubelet/pods/25cb40cf-a748-4af8-8aab-a2c9c75b53dc/volumes" Oct 31 04:19:04.443200 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:04.443192 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d306e0df-7e92-48f8-a089-8b130186db1d" path="/var/lib/kubelet/pods/d306e0df-7e92-48f8-a089-8b130186db1d/volumes" Oct 31 04:19:04.800043 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:04.799947 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-09a20a/distinct-w2-09a20a-kserve-76557f6448-l8d69" Oct 31 04:19:04.812752 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:04.812722 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-09a20a/distinct-w2-09a20a-kserve-76557f6448-l8d69" Oct 31 04:19:05.176229 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:05.176202 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-09a20a/distinct-w2-09a20a-kserve-76557f6448-l8d69" Oct 31 04:19:11.000292 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:19:11.000246 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 04:19:13.012366 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:13.010386 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-4c2387/distinct-w3-4c2387-kserve-768bdd84c7-6jmtl"] Oct 31 04:19:13.013698 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:13.013671 4376 cpu_manager.go:414] "containerMap: removing container" podUID="25cb40cf-a748-4af8-8aab-a2c9c75b53dc" containerName="payload-processing" Oct 31 04:19:13.013854 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:13.013837 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25cb40cf-a748-4af8-8aab-a2c9c75b53dc" containerName="payload-processing" Oct 31 04:19:13.013971 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:13.013957 4376 cpu_manager.go:414] "containerMap: removing container" podUID="d306e0df-7e92-48f8-a089-8b130186db1d" containerName="payload-processing" Oct 31 04:19:13.014064 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:13.014052 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d306e0df-7e92-48f8-a089-8b130186db1d" containerName="payload-processing" Oct 31 04:19:13.014321 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:13.014306 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="25cb40cf-a748-4af8-8aab-a2c9c75b53dc" containerName="payload-processing" Oct 31 04:19:13.014460 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:13.014446 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="d306e0df-7e92-48f8-a089-8b130186db1d" containerName="payload-processing" Oct 31 04:19:13.030320 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:13.030293 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-4c2387/distinct-w3-4c2387-kserve-768bdd84c7-6jmtl" Oct 31 04:19:13.093352 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:13.092562 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2c75691-4cf4-484b-96e2-3c2503b8d78c-tls-certs\") pod \"distinct-w3-4c2387-kserve-768bdd84c7-6jmtl\" (UID: \"b2c75691-4cf4-484b-96e2-3c2503b8d78c\") " pod="e2e-models-e2e-worker-w3-4c2387/distinct-w3-4c2387-kserve-768bdd84c7-6jmtl" Oct 31 04:19:13.093352 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:13.092624 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2c75691-4cf4-484b-96e2-3c2503b8d78c-model-cache\") pod \"distinct-w3-4c2387-kserve-768bdd84c7-6jmtl\" (UID: \"b2c75691-4cf4-484b-96e2-3c2503b8d78c\") " pod="e2e-models-e2e-worker-w3-4c2387/distinct-w3-4c2387-kserve-768bdd84c7-6jmtl" Oct 31 04:19:13.093352 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:13.092652 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2c75691-4cf4-484b-96e2-3c2503b8d78c-home\") pod \"distinct-w3-4c2387-kserve-768bdd84c7-6jmtl\" (UID: \"b2c75691-4cf4-484b-96e2-3c2503b8d78c\") " pod="e2e-models-e2e-worker-w3-4c2387/distinct-w3-4c2387-kserve-768bdd84c7-6jmtl" Oct 31 04:19:13.093352 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:13.092736 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2c75691-4cf4-484b-96e2-3c2503b8d78c-tmp-dir\") pod \"distinct-w3-4c2387-kserve-768bdd84c7-6jmtl\" (UID: \"b2c75691-4cf4-484b-96e2-3c2503b8d78c\") " pod="e2e-models-e2e-worker-w3-4c2387/distinct-w3-4c2387-kserve-768bdd84c7-6jmtl" Oct 31 04:19:13.093352 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:13.092783 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2c75691-4cf4-484b-96e2-3c2503b8d78c-dshm\") pod \"distinct-w3-4c2387-kserve-768bdd84c7-6jmtl\" (UID: \"b2c75691-4cf4-484b-96e2-3c2503b8d78c\") " pod="e2e-models-e2e-worker-w3-4c2387/distinct-w3-4c2387-kserve-768bdd84c7-6jmtl" Oct 31 04:19:13.093352 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:13.092838 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2bdk\" (UniqueName: \"kubernetes.io/projected/b2c75691-4cf4-484b-96e2-3c2503b8d78c-kube-api-access-k2bdk\") pod \"distinct-w3-4c2387-kserve-768bdd84c7-6jmtl\" (UID: \"b2c75691-4cf4-484b-96e2-3c2503b8d78c\") " pod="e2e-models-e2e-worker-w3-4c2387/distinct-w3-4c2387-kserve-768bdd84c7-6jmtl" Oct 31 04:19:13.126628 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:13.120664 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-4c2387\"/\"distinct-w3-4c2387-kserve-self-signed-certs\"" Oct 31 04:19:13.126628 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:13.120869 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-4c2387\"/\"openshift-service-ca.crt\"" Oct 31 04:19:13.126628 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:13.121486 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-4c2387\"/\"kube-root-ca.crt\"" Oct 31 04:19:13.126628 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:13.121560 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-4c2387\"/\"default-dockercfg-zfb5b\"" Oct 31 04:19:13.140381 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:13.138830 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-4c2387/distinct-w3-4c2387-kserve-768bdd84c7-6jmtl"] Oct 31 04:19:13.389353 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:13.381738 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2c75691-4cf4-484b-96e2-3c2503b8d78c-dshm\") pod \"distinct-w3-4c2387-kserve-768bdd84c7-6jmtl\" (UID: \"b2c75691-4cf4-484b-96e2-3c2503b8d78c\") " pod="e2e-models-e2e-worker-w3-4c2387/distinct-w3-4c2387-kserve-768bdd84c7-6jmtl" Oct 31 04:19:13.451546 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:13.443512 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2bdk\" (UniqueName: \"kubernetes.io/projected/b2c75691-4cf4-484b-96e2-3c2503b8d78c-kube-api-access-k2bdk\") pod \"distinct-w3-4c2387-kserve-768bdd84c7-6jmtl\" (UID: \"b2c75691-4cf4-484b-96e2-3c2503b8d78c\") " pod="e2e-models-e2e-worker-w3-4c2387/distinct-w3-4c2387-kserve-768bdd84c7-6jmtl" Oct 31 04:19:13.451546 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:13.443746 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2c75691-4cf4-484b-96e2-3c2503b8d78c-tls-certs\") pod \"distinct-w3-4c2387-kserve-768bdd84c7-6jmtl\" (UID: \"b2c75691-4cf4-484b-96e2-3c2503b8d78c\") " pod="e2e-models-e2e-worker-w3-4c2387/distinct-w3-4c2387-kserve-768bdd84c7-6jmtl" Oct 31 04:19:13.451546 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:13.443807 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2c75691-4cf4-484b-96e2-3c2503b8d78c-model-cache\") pod \"distinct-w3-4c2387-kserve-768bdd84c7-6jmtl\" (UID: \"b2c75691-4cf4-484b-96e2-3c2503b8d78c\") " pod="e2e-models-e2e-worker-w3-4c2387/distinct-w3-4c2387-kserve-768bdd84c7-6jmtl" Oct 31 04:19:13.451546 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:13.443840 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2c75691-4cf4-484b-96e2-3c2503b8d78c-home\") pod \"distinct-w3-4c2387-kserve-768bdd84c7-6jmtl\" (UID: \"b2c75691-4cf4-484b-96e2-3c2503b8d78c\") " pod="e2e-models-e2e-worker-w3-4c2387/distinct-w3-4c2387-kserve-768bdd84c7-6jmtl" Oct 31 04:19:13.451546 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:13.443944 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2c75691-4cf4-484b-96e2-3c2503b8d78c-tmp-dir\") pod \"distinct-w3-4c2387-kserve-768bdd84c7-6jmtl\" (UID: \"b2c75691-4cf4-484b-96e2-3c2503b8d78c\") " pod="e2e-models-e2e-worker-w3-4c2387/distinct-w3-4c2387-kserve-768bdd84c7-6jmtl" Oct 31 04:19:13.451546 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:13.444454 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2c75691-4cf4-484b-96e2-3c2503b8d78c-tmp-dir\") pod \"distinct-w3-4c2387-kserve-768bdd84c7-6jmtl\" (UID: \"b2c75691-4cf4-484b-96e2-3c2503b8d78c\") " pod="e2e-models-e2e-worker-w3-4c2387/distinct-w3-4c2387-kserve-768bdd84c7-6jmtl" Oct 31 04:19:13.451546 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:13.445573 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2c75691-4cf4-484b-96e2-3c2503b8d78c-model-cache\") pod \"distinct-w3-4c2387-kserve-768bdd84c7-6jmtl\" (UID: \"b2c75691-4cf4-484b-96e2-3c2503b8d78c\") " pod="e2e-models-e2e-worker-w3-4c2387/distinct-w3-4c2387-kserve-768bdd84c7-6jmtl" Oct 31 04:19:13.451546 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:13.445845 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2c75691-4cf4-484b-96e2-3c2503b8d78c-home\") pod \"distinct-w3-4c2387-kserve-768bdd84c7-6jmtl\" (UID: \"b2c75691-4cf4-484b-96e2-3c2503b8d78c\") " pod="e2e-models-e2e-worker-w3-4c2387/distinct-w3-4c2387-kserve-768bdd84c7-6jmtl" Oct 31 04:19:13.555726 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:13.555653 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2c75691-4cf4-484b-96e2-3c2503b8d78c-dshm\") pod \"distinct-w3-4c2387-kserve-768bdd84c7-6jmtl\" (UID: \"b2c75691-4cf4-484b-96e2-3c2503b8d78c\") " pod="e2e-models-e2e-worker-w3-4c2387/distinct-w3-4c2387-kserve-768bdd84c7-6jmtl" Oct 31 04:19:13.561178 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:13.561148 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2bdk\" (UniqueName: \"kubernetes.io/projected/b2c75691-4cf4-484b-96e2-3c2503b8d78c-kube-api-access-k2bdk\") pod \"distinct-w3-4c2387-kserve-768bdd84c7-6jmtl\" (UID: \"b2c75691-4cf4-484b-96e2-3c2503b8d78c\") " pod="e2e-models-e2e-worker-w3-4c2387/distinct-w3-4c2387-kserve-768bdd84c7-6jmtl" Oct 31 04:19:13.596355 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:13.580036 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2c75691-4cf4-484b-96e2-3c2503b8d78c-tls-certs\") pod \"distinct-w3-4c2387-kserve-768bdd84c7-6jmtl\" (UID: \"b2c75691-4cf4-484b-96e2-3c2503b8d78c\") " pod="e2e-models-e2e-worker-w3-4c2387/distinct-w3-4c2387-kserve-768bdd84c7-6jmtl" Oct 31 04:19:13.895964 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:13.895931 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-4c2387/distinct-w3-4c2387-kserve-768bdd84c7-6jmtl" Oct 31 04:19:13.952368 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:13.927457 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-bpgxq"] Oct 31 04:19:14.025645 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:14.025595 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c4006d/simulator-w4-c4006d-kserve-595455485b-4h46q" podUID="576e2c1f-1c80-43a9-a136-3692776642ae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:19:14.028762 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:14.028726 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-d7npm" podUID="174171e7-100c-4b84-becd-0176672f1805" containerName="main" probeResult="failure" output="Get \"https://10.133.0.34:8000/ready\": context deadline exceeded" Oct 31 04:19:14.076484 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:14.076440 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-09a20a/premium-w2-09a20a-kserve-57c45db74f-s8glf" podUID="a3f9d0a0-a106-48bb-b6f2-a980c3aff4e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Oct 31 04:19:14.079057 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:14.079023 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-439989/premium-w1-439989-kserve-f9fc54cdf-sn2pp" podUID="17b2f181-2e66-4ce2-95b0-cb213f11a0e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Oct 31 04:19:14.128570 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:14.128532 4376 patch_prober.go:29] interesting pod/e2e-worker-w2-09a20a-openshift-default-55d8647ff-zlcr5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 04:19:14.128855 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:14.128822 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-zlcr5" podUID="2697f74a-c2a2-4d6f-ba09-adb5825a8d85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 04:19:14.183111 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:14.183028 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-bpgxq"] Oct 31 04:19:14.183570 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:14.183399 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-bpgxq" Oct 31 04:19:14.254353 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:14.238706 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-09a20a/distinct-w2-09a20a-kserve-76557f6448-l8d69" podUID="7d298000-b65b-45fd-a1ee-092e47a7eb24" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:19:14.323906 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:14.323199 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/880c7780-4eac-4a67-b948-3d52a354c554-istio-podinfo\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-bpgxq\" (UID: \"880c7780-4eac-4a67-b948-3d52a354c554\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-bpgxq" Oct 31 04:19:14.323906 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:14.323258 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j49nq\" (UniqueName: \"kubernetes.io/projected/880c7780-4eac-4a67-b948-3d52a354c554-kube-api-access-j49nq\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-bpgxq\" (UID: \"880c7780-4eac-4a67-b948-3d52a354c554\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-bpgxq" Oct 31 04:19:14.323906 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:14.323298 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/880c7780-4eac-4a67-b948-3d52a354c554-workload-certs\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-bpgxq\" (UID: \"880c7780-4eac-4a67-b948-3d52a354c554\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-bpgxq" Oct 31 04:19:14.323906 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:14.323342 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/880c7780-4eac-4a67-b948-3d52a354c554-istio-data\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-bpgxq\" (UID: \"880c7780-4eac-4a67-b948-3d52a354c554\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-bpgxq" Oct 31 04:19:14.323906 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:14.323384 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/880c7780-4eac-4a67-b948-3d52a354c554-istiod-ca-cert\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-bpgxq\" (UID: \"880c7780-4eac-4a67-b948-3d52a354c554\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-bpgxq" Oct 31 04:19:14.323906 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:14.323414 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/880c7780-4eac-4a67-b948-3d52a354c554-istio-token\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-bpgxq\" (UID: \"880c7780-4eac-4a67-b948-3d52a354c554\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-bpgxq" Oct 31 04:19:14.323906 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:14.323458 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/880c7780-4eac-4a67-b948-3d52a354c554-istio-envoy\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-bpgxq\" (UID: \"880c7780-4eac-4a67-b948-3d52a354c554\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-bpgxq" Oct 31 04:19:14.323906 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:14.323555 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/880c7780-4eac-4a67-b948-3d52a354c554-workload-socket\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-bpgxq\" (UID: \"880c7780-4eac-4a67-b948-3d52a354c554\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-bpgxq" Oct 31 04:19:14.323906 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:14.323686 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/880c7780-4eac-4a67-b948-3d52a354c554-credential-socket\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-bpgxq\" (UID: \"880c7780-4eac-4a67-b948-3d52a354c554\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-bpgxq" Oct 31 04:19:14.442658 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:14.427372 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/880c7780-4eac-4a67-b948-3d52a354c554-credential-socket\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-bpgxq\" (UID: \"880c7780-4eac-4a67-b948-3d52a354c554\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-bpgxq" Oct 31 04:19:14.442658 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:14.427469 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/880c7780-4eac-4a67-b948-3d52a354c554-istio-podinfo\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-bpgxq\" (UID: \"880c7780-4eac-4a67-b948-3d52a354c554\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-bpgxq" Oct 31 04:19:14.442658 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:14.427504 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j49nq\" (UniqueName: \"kubernetes.io/projected/880c7780-4eac-4a67-b948-3d52a354c554-kube-api-access-j49nq\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-bpgxq\" (UID: \"880c7780-4eac-4a67-b948-3d52a354c554\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-bpgxq" Oct 31 04:19:14.442658 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:14.427542 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/880c7780-4eac-4a67-b948-3d52a354c554-workload-certs\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-bpgxq\" (UID: \"880c7780-4eac-4a67-b948-3d52a354c554\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-bpgxq" Oct 31 04:19:14.442658 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:14.427568 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/880c7780-4eac-4a67-b948-3d52a354c554-istio-data\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-bpgxq\" (UID: \"880c7780-4eac-4a67-b948-3d52a354c554\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-bpgxq" Oct 31 04:19:14.442658 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:14.427606 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/880c7780-4eac-4a67-b948-3d52a354c554-istiod-ca-cert\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-bpgxq\" (UID: \"880c7780-4eac-4a67-b948-3d52a354c554\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-bpgxq" Oct 31 04:19:14.442658 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:14.427637 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/880c7780-4eac-4a67-b948-3d52a354c554-istio-token\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-bpgxq\" (UID: \"880c7780-4eac-4a67-b948-3d52a354c554\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-bpgxq" Oct 31 04:19:14.442658 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:14.427682 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/880c7780-4eac-4a67-b948-3d52a354c554-istio-envoy\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-bpgxq\" (UID: \"880c7780-4eac-4a67-b948-3d52a354c554\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-bpgxq" Oct 31 04:19:14.442658 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:14.427782 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/880c7780-4eac-4a67-b948-3d52a354c554-workload-socket\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-bpgxq\" (UID: \"880c7780-4eac-4a67-b948-3d52a354c554\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-bpgxq" Oct 31 04:19:14.442658 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:14.428300 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/880c7780-4eac-4a67-b948-3d52a354c554-workload-socket\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-bpgxq\" (UID: \"880c7780-4eac-4a67-b948-3d52a354c554\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-bpgxq" Oct 31 04:19:14.442658 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:14.428613 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/880c7780-4eac-4a67-b948-3d52a354c554-credential-socket\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-bpgxq\" (UID: \"880c7780-4eac-4a67-b948-3d52a354c554\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-bpgxq" Oct 31 04:19:14.442658 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:14.440732 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/880c7780-4eac-4a67-b948-3d52a354c554-istio-data\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-bpgxq\" (UID: \"880c7780-4eac-4a67-b948-3d52a354c554\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-bpgxq" Oct 31 04:19:14.442658 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:14.441373 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/880c7780-4eac-4a67-b948-3d52a354c554-workload-certs\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-bpgxq\" (UID: \"880c7780-4eac-4a67-b948-3d52a354c554\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-bpgxq" Oct 31 04:19:14.442658 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:14.442264 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/880c7780-4eac-4a67-b948-3d52a354c554-istiod-ca-cert\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-bpgxq\" (UID: \"880c7780-4eac-4a67-b948-3d52a354c554\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-bpgxq" Oct 31 04:19:14.740432 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:14.736943 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/880c7780-4eac-4a67-b948-3d52a354c554-istio-podinfo\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-bpgxq\" (UID: \"880c7780-4eac-4a67-b948-3d52a354c554\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-bpgxq" Oct 31 04:19:14.746203 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:14.746118 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/880c7780-4eac-4a67-b948-3d52a354c554-istio-token\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-bpgxq\" (UID: \"880c7780-4eac-4a67-b948-3d52a354c554\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-bpgxq" Oct 31 04:19:14.832001 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:14.805141 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j49nq\" (UniqueName: \"kubernetes.io/projected/880c7780-4eac-4a67-b948-3d52a354c554-kube-api-access-j49nq\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-bpgxq\" (UID: \"880c7780-4eac-4a67-b948-3d52a354c554\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-bpgxq" Oct 31 04:19:14.832001 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:14.812138 4376 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-d7npm" podUID="174171e7-100c-4b84-becd-0176672f1805" containerName="main" probeResult="failure" output="Get \"https://10.133.0.34:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:19:14.913673 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:14.905813 4376 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-c4006d/simulator-w4-c4006d-kserve-595455485b-4h46q" podUID="576e2c1f-1c80-43a9-a136-3692776642ae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:19:14.934888 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:14.934228 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/880c7780-4eac-4a67-b948-3d52a354c554-istio-envoy\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-bpgxq\" (UID: \"880c7780-4eac-4a67-b948-3d52a354c554\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-bpgxq" Oct 31 04:19:15.073472 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:19:15.073438 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb2c75691_4cf4_484b_96e2_3c2503b8d78c.slice/crio-bae544aa0168a3a9fe9bbb08132d74cf32cec489df228faf668e547ac3e15844 WatchSource:0}: Error finding container bae544aa0168a3a9fe9bbb08132d74cf32cec489df228faf668e547ac3e15844: Status 404 returned error can't find the container with id bae544aa0168a3a9fe9bbb08132d74cf32cec489df228faf668e547ac3e15844 Oct 31 04:19:15.131838 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:15.128703 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-bpgxq" Oct 31 04:19:15.131838 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:15.129560 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-4c2387/distinct-w3-4c2387-kserve-768bdd84c7-6jmtl"] Oct 31 04:19:15.741405 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:15.741222 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-4c2387/distinct-w3-4c2387-kserve-768bdd84c7-6jmtl" event={"ID":"b2c75691-4cf4-484b-96e2-3c2503b8d78c","Type":"ContainerStarted","Data":"e601b65852ff9096dfc957fb7cdb571aef3670633546efe666ee6fc47a19f36b"} Oct 31 04:19:15.741405 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:15.741277 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-4c2387/distinct-w3-4c2387-kserve-768bdd84c7-6jmtl" event={"ID":"b2c75691-4cf4-484b-96e2-3c2503b8d78c","Type":"ContainerStarted","Data":"bae544aa0168a3a9fe9bbb08132d74cf32cec489df228faf668e547ac3e15844"} Oct 31 04:19:15.802312 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:15.801843 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-4c2387/distinct-w3-4c2387-kserve-768bdd84c7-6jmtl" podStartSLOduration=3.80182221 podStartE2EDuration="3.80182221s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:19:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:19:15.792742959 +0000 UTC m=+903.507077579" watchObservedRunningTime="2026-10-31 04:19:15.80182221 +0000 UTC m=+903.516156829" Oct 31 04:19:15.814112 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:15.814084 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-bpgxq"] Oct 31 04:19:15.815531 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:19:15.815505 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod880c7780_4eac_4a67_b948_3d52a354c554.slice/crio-3400214d725e8b114ad65d8f4a84013eedcb778e49621f7ed883c50f73787ca2 WatchSource:0}: Error finding container 3400214d725e8b114ad65d8f4a84013eedcb778e49621f7ed883c50f73787ca2: Status 404 returned error can't find the container with id 3400214d725e8b114ad65d8f4a84013eedcb778e49621f7ed883c50f73787ca2 Oct 31 04:19:15.818720 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:15.818685 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:19:15.818823 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:15.818752 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:19:15.818823 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:15.818784 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:19:15.897612 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:15.897581 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-4c2387/distinct-w3-4c2387-kserve-768bdd84c7-6jmtl" Oct 31 04:19:16.770022 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:16.769974 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-bpgxq" event={"ID":"880c7780-4eac-4a67-b948-3d52a354c554","Type":"ContainerStarted","Data":"b7f0d327394e7f57beaa3798867e475cf9ee3d8ad80ac864847d019d4ed24535"} Oct 31 04:19:16.770590 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:16.770049 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-bpgxq" event={"ID":"880c7780-4eac-4a67-b948-3d52a354c554","Type":"ContainerStarted","Data":"3400214d725e8b114ad65d8f4a84013eedcb778e49621f7ed883c50f73787ca2"} Oct 31 04:19:16.833833 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:16.833773 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-bpgxq" podStartSLOduration=3.833757525 podStartE2EDuration="3.833757525s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:19:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:19:16.832962155 +0000 UTC m=+904.547296789" watchObservedRunningTime="2026-10-31 04:19:16.833757525 +0000 UTC m=+904.548092142" Oct 31 04:19:17.129994 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:17.129954 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-bpgxq" Oct 31 04:19:17.131463 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:17.131433 4376 patch_prober.go:29] interesting pod/e2e-worker-w0-965f61-openshift-default-55854964c-bpgxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 31 04:19:17.131604 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:17.131494 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-bpgxq" podUID="880c7780-4eac-4a67-b948-3d52a354c554" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 31 04:19:18.130536 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:18.130500 4376 patch_prober.go:29] interesting pod/e2e-worker-w0-965f61-openshift-default-55854964c-bpgxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 31 04:19:18.130996 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:18.130558 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-bpgxq" podUID="880c7780-4eac-4a67-b948-3d52a354c554" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 31 04:19:19.130291 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:19.130255 4376 patch_prober.go:29] interesting pod/e2e-worker-w0-965f61-openshift-default-55854964c-bpgxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 31 04:19:19.130487 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:19.130314 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-bpgxq" podUID="880c7780-4eac-4a67-b948-3d52a354c554" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 31 04:19:19.492080 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:19.491934 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-bzbxb"] Oct 31 04:19:19.496742 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:19.496720 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-bzbxb" Oct 31 04:19:19.519304 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:19.519264 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-bzbxb"] Oct 31 04:19:19.634388 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:19.624409 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zg5gl\" (UniqueName: \"kubernetes.io/projected/1287989d-3ca3-48c0-bfa1-3e6e27b7f06d-kube-api-access-zg5gl\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-bzbxb\" (UID: \"1287989d-3ca3-48c0-bfa1-3e6e27b7f06d\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-bzbxb" Oct 31 04:19:19.635151 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:19.634781 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1287989d-3ca3-48c0-bfa1-3e6e27b7f06d-workload-socket\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-bzbxb\" (UID: \"1287989d-3ca3-48c0-bfa1-3e6e27b7f06d\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-bzbxb" Oct 31 04:19:19.635448 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:19.635427 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1287989d-3ca3-48c0-bfa1-3e6e27b7f06d-istio-data\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-bzbxb\" (UID: \"1287989d-3ca3-48c0-bfa1-3e6e27b7f06d\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-bzbxb" Oct 31 04:19:19.635625 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:19.635609 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1287989d-3ca3-48c0-bfa1-3e6e27b7f06d-istio-token\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-bzbxb\" (UID: \"1287989d-3ca3-48c0-bfa1-3e6e27b7f06d\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-bzbxb" Oct 31 04:19:19.635797 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:19.635779 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1287989d-3ca3-48c0-bfa1-3e6e27b7f06d-workload-certs\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-bzbxb\" (UID: \"1287989d-3ca3-48c0-bfa1-3e6e27b7f06d\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-bzbxb" Oct 31 04:19:19.635948 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:19.635934 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1287989d-3ca3-48c0-bfa1-3e6e27b7f06d-istiod-ca-cert\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-bzbxb\" (UID: \"1287989d-3ca3-48c0-bfa1-3e6e27b7f06d\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-bzbxb" Oct 31 04:19:19.636147 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:19.636129 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1287989d-3ca3-48c0-bfa1-3e6e27b7f06d-credential-socket\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-bzbxb\" (UID: \"1287989d-3ca3-48c0-bfa1-3e6e27b7f06d\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-bzbxb" Oct 31 04:19:19.636297 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:19.636282 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1287989d-3ca3-48c0-bfa1-3e6e27b7f06d-istio-envoy\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-bzbxb\" (UID: \"1287989d-3ca3-48c0-bfa1-3e6e27b7f06d\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-bzbxb" Oct 31 04:19:19.636544 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:19.636522 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1287989d-3ca3-48c0-bfa1-3e6e27b7f06d-istio-podinfo\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-bzbxb\" (UID: \"1287989d-3ca3-48c0-bfa1-3e6e27b7f06d\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-bzbxb" Oct 31 04:19:19.750406 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:19.744016 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1287989d-3ca3-48c0-bfa1-3e6e27b7f06d-workload-certs\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-bzbxb\" (UID: \"1287989d-3ca3-48c0-bfa1-3e6e27b7f06d\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-bzbxb" Oct 31 04:19:19.750406 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:19.744081 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1287989d-3ca3-48c0-bfa1-3e6e27b7f06d-istiod-ca-cert\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-bzbxb\" (UID: \"1287989d-3ca3-48c0-bfa1-3e6e27b7f06d\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-bzbxb" Oct 31 04:19:19.750406 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:19.744130 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1287989d-3ca3-48c0-bfa1-3e6e27b7f06d-credential-socket\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-bzbxb\" (UID: \"1287989d-3ca3-48c0-bfa1-3e6e27b7f06d\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-bzbxb" Oct 31 04:19:19.750406 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:19.744158 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1287989d-3ca3-48c0-bfa1-3e6e27b7f06d-istio-envoy\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-bzbxb\" (UID: \"1287989d-3ca3-48c0-bfa1-3e6e27b7f06d\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-bzbxb" Oct 31 04:19:19.750406 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:19.744232 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1287989d-3ca3-48c0-bfa1-3e6e27b7f06d-istio-podinfo\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-bzbxb\" (UID: \"1287989d-3ca3-48c0-bfa1-3e6e27b7f06d\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-bzbxb" Oct 31 04:19:19.750406 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:19.744305 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zg5gl\" (UniqueName: \"kubernetes.io/projected/1287989d-3ca3-48c0-bfa1-3e6e27b7f06d-kube-api-access-zg5gl\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-bzbxb\" (UID: \"1287989d-3ca3-48c0-bfa1-3e6e27b7f06d\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-bzbxb" Oct 31 04:19:19.750406 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:19.744402 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1287989d-3ca3-48c0-bfa1-3e6e27b7f06d-workload-socket\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-bzbxb\" (UID: \"1287989d-3ca3-48c0-bfa1-3e6e27b7f06d\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-bzbxb" Oct 31 04:19:19.750406 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:19.744489 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1287989d-3ca3-48c0-bfa1-3e6e27b7f06d-istio-data\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-bzbxb\" (UID: \"1287989d-3ca3-48c0-bfa1-3e6e27b7f06d\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-bzbxb" Oct 31 04:19:19.750406 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:19.744517 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1287989d-3ca3-48c0-bfa1-3e6e27b7f06d-istio-token\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-bzbxb\" (UID: \"1287989d-3ca3-48c0-bfa1-3e6e27b7f06d\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-bzbxb" Oct 31 04:19:19.750406 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:19.745290 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1287989d-3ca3-48c0-bfa1-3e6e27b7f06d-workload-certs\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-bzbxb\" (UID: \"1287989d-3ca3-48c0-bfa1-3e6e27b7f06d\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-bzbxb" Oct 31 04:19:19.750406 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:19.745975 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1287989d-3ca3-48c0-bfa1-3e6e27b7f06d-istiod-ca-cert\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-bzbxb\" (UID: \"1287989d-3ca3-48c0-bfa1-3e6e27b7f06d\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-bzbxb" Oct 31 04:19:19.750406 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:19.746238 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1287989d-3ca3-48c0-bfa1-3e6e27b7f06d-credential-socket\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-bzbxb\" (UID: \"1287989d-3ca3-48c0-bfa1-3e6e27b7f06d\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-bzbxb" Oct 31 04:19:19.752230 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:19.752207 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1287989d-3ca3-48c0-bfa1-3e6e27b7f06d-workload-socket\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-bzbxb\" (UID: \"1287989d-3ca3-48c0-bfa1-3e6e27b7f06d\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-bzbxb" Oct 31 04:19:19.752727 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:19.752649 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1287989d-3ca3-48c0-bfa1-3e6e27b7f06d-istio-data\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-bzbxb\" (UID: \"1287989d-3ca3-48c0-bfa1-3e6e27b7f06d\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-bzbxb" Oct 31 04:19:19.763318 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:19.763287 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1287989d-3ca3-48c0-bfa1-3e6e27b7f06d-istio-podinfo\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-bzbxb\" (UID: \"1287989d-3ca3-48c0-bfa1-3e6e27b7f06d\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-bzbxb" Oct 31 04:19:19.771350 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:19.771139 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1287989d-3ca3-48c0-bfa1-3e6e27b7f06d-istio-envoy\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-bzbxb\" (UID: \"1287989d-3ca3-48c0-bfa1-3e6e27b7f06d\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-bzbxb" Oct 31 04:19:19.774654 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:19.774630 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1287989d-3ca3-48c0-bfa1-3e6e27b7f06d-istio-token\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-bzbxb\" (UID: \"1287989d-3ca3-48c0-bfa1-3e6e27b7f06d\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-bzbxb" Oct 31 04:19:19.777098 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:19.777072 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zg5gl\" (UniqueName: \"kubernetes.io/projected/1287989d-3ca3-48c0-bfa1-3e6e27b7f06d-kube-api-access-zg5gl\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-bzbxb\" (UID: \"1287989d-3ca3-48c0-bfa1-3e6e27b7f06d\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-bzbxb" Oct 31 04:19:19.810985 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:19.810959 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-bzbxb" Oct 31 04:19:20.130565 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:20.130529 4376 patch_prober.go:29] interesting pod/e2e-worker-w0-965f61-openshift-default-55854964c-bpgxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 31 04:19:20.130757 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:20.130589 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-bpgxq" podUID="880c7780-4eac-4a67-b948-3d52a354c554" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 31 04:19:20.169276 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:20.169040 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-bzbxb"] Oct 31 04:19:20.171929 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:19:20.171898 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1287989d_3ca3_48c0_bfa1_3e6e27b7f06d.slice/crio-fcf7203220e74f43754e1b30b33b98729c16438dafaaf273569bcb14f1671136 WatchSource:0}: Error finding container fcf7203220e74f43754e1b30b33b98729c16438dafaaf273569bcb14f1671136: Status 404 returned error can't find the container with id fcf7203220e74f43754e1b30b33b98729c16438dafaaf273569bcb14f1671136 Oct 31 04:19:20.174461 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:20.174429 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:19:20.174539 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:20.174497 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:19:20.174539 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:20.174526 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:19:20.795855 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:20.795822 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-bzbxb" event={"ID":"1287989d-3ca3-48c0-bfa1-3e6e27b7f06d","Type":"ContainerStarted","Data":"2d6d4ae8e4be174ccecadedfe719fbf0a903a33f1f647bdd228ba3a74dcd8aa3"} Oct 31 04:19:20.796236 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:20.795871 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-bzbxb" event={"ID":"1287989d-3ca3-48c0-bfa1-3e6e27b7f06d","Type":"ContainerStarted","Data":"fcf7203220e74f43754e1b30b33b98729c16438dafaaf273569bcb14f1671136"} Oct 31 04:19:20.811175 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:20.811149 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-bzbxb" Oct 31 04:19:20.852352 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:20.847115 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-bzbxb" podStartSLOduration=1.847097607 podStartE2EDuration="1.847097607s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:19:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:19:20.846792748 +0000 UTC m=+908.561127400" watchObservedRunningTime="2026-10-31 04:19:20.847097607 +0000 UTC m=+908.561432225" Oct 31 04:19:21.129888 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:21.129855 4376 patch_prober.go:29] interesting pod/e2e-worker-w0-965f61-openshift-default-55854964c-bpgxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 31 04:19:21.130034 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:21.129913 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-bpgxq" podUID="880c7780-4eac-4a67-b948-3d52a354c554" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 31 04:19:21.813343 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:21.813301 4376 patch_prober.go:29] interesting pod/e2e-worker-w1-439989-openshift-default-8448c6ff99-bzbxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Oct 31 04:19:21.813738 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:21.813377 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-bzbxb" podUID="1287989d-3ca3-48c0-bfa1-3e6e27b7f06d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Oct 31 04:19:22.130370 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:22.130321 4376 patch_prober.go:29] interesting pod/e2e-worker-w0-965f61-openshift-default-55854964c-bpgxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 31 04:19:22.130537 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:22.130388 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-bpgxq" podUID="880c7780-4eac-4a67-b948-3d52a354c554" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 31 04:19:22.811447 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:22.811409 4376 patch_prober.go:29] interesting pod/e2e-worker-w1-439989-openshift-default-8448c6ff99-bzbxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Oct 31 04:19:22.811624 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:22.811475 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-bzbxb" podUID="1287989d-3ca3-48c0-bfa1-3e6e27b7f06d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Oct 31 04:19:23.304399 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:23.304365 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-bpgxq" Oct 31 04:19:23.305646 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:23.305619 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-bpgxq" Oct 31 04:19:23.306927 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:23.306906 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-bpgxq" Oct 31 04:19:23.811287 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:23.811259 4376 patch_prober.go:29] interesting pod/e2e-worker-w1-439989-openshift-default-8448c6ff99-bzbxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Oct 31 04:19:23.811485 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:23.811315 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-bzbxb" podUID="1287989d-3ca3-48c0-bfa1-3e6e27b7f06d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Oct 31 04:19:23.898129 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:23.898093 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-4c2387/distinct-w3-4c2387-kserve-768bdd84c7-6jmtl" Oct 31 04:19:23.910735 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:23.910703 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-4c2387/distinct-w3-4c2387-kserve-768bdd84c7-6jmtl" Oct 31 04:19:24.811374 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:24.811322 4376 patch_prober.go:29] interesting pod/e2e-worker-w1-439989-openshift-default-8448c6ff99-bzbxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Oct 31 04:19:24.811763 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:24.811404 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-bzbxb" podUID="1287989d-3ca3-48c0-bfa1-3e6e27b7f06d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Oct 31 04:19:24.829722 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:24.829687 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-4c2387/distinct-w3-4c2387-kserve-768bdd84c7-6jmtl" Oct 31 04:19:25.819004 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:25.818962 4376 patch_prober.go:29] interesting pod/e2e-worker-w1-439989-openshift-default-8448c6ff99-bzbxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Oct 31 04:19:25.819479 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:25.819043 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-bzbxb" podUID="1287989d-3ca3-48c0-bfa1-3e6e27b7f06d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Oct 31 04:19:26.264244 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:26.264209 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 04:19:26.812116 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:26.812080 4376 patch_prober.go:29] interesting pod/e2e-worker-w1-439989-openshift-default-8448c6ff99-bzbxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Oct 31 04:19:26.812304 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:26.812145 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-bzbxb" podUID="1287989d-3ca3-48c0-bfa1-3e6e27b7f06d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Oct 31 04:19:27.119703 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:27.119669 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-09a20a/distinct-2-w2-09a20a-kserve-779d648d75-mshl7"] Oct 31 04:19:27.125108 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:27.125083 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-09a20a/distinct-2-w2-09a20a-kserve-779d648d75-mshl7" Oct 31 04:19:27.144937 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:27.144903 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-09a20a/distinct-2-w2-09a20a-kserve-779d648d75-mshl7"] Oct 31 04:19:27.155114 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:27.155083 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-09a20a\"/\"distinct-2-w2-09a20a-kserve-self-signed-certs\"" Oct 31 04:19:27.202994 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:27.202959 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c81f5d58-c5c7-46c1-b832-c550b092f02b-home\") pod \"distinct-2-w2-09a20a-kserve-779d648d75-mshl7\" (UID: \"c81f5d58-c5c7-46c1-b832-c550b092f02b\") " pod="e2e-models-e2e-worker-w2-09a20a/distinct-2-w2-09a20a-kserve-779d648d75-mshl7" Oct 31 04:19:27.202994 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:27.202993 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c81f5d58-c5c7-46c1-b832-c550b092f02b-tmp-dir\") pod \"distinct-2-w2-09a20a-kserve-779d648d75-mshl7\" (UID: \"c81f5d58-c5c7-46c1-b832-c550b092f02b\") " pod="e2e-models-e2e-worker-w2-09a20a/distinct-2-w2-09a20a-kserve-779d648d75-mshl7" Oct 31 04:19:27.203198 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:27.203022 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c81f5d58-c5c7-46c1-b832-c550b092f02b-model-cache\") pod \"distinct-2-w2-09a20a-kserve-779d648d75-mshl7\" (UID: \"c81f5d58-c5c7-46c1-b832-c550b092f02b\") " pod="e2e-models-e2e-worker-w2-09a20a/distinct-2-w2-09a20a-kserve-779d648d75-mshl7" Oct 31 04:19:27.203198 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:27.203135 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c81f5d58-c5c7-46c1-b832-c550b092f02b-tls-certs\") pod \"distinct-2-w2-09a20a-kserve-779d648d75-mshl7\" (UID: \"c81f5d58-c5c7-46c1-b832-c550b092f02b\") " pod="e2e-models-e2e-worker-w2-09a20a/distinct-2-w2-09a20a-kserve-779d648d75-mshl7" Oct 31 04:19:27.203198 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:27.203164 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sswx2\" (UniqueName: \"kubernetes.io/projected/c81f5d58-c5c7-46c1-b832-c550b092f02b-kube-api-access-sswx2\") pod \"distinct-2-w2-09a20a-kserve-779d648d75-mshl7\" (UID: \"c81f5d58-c5c7-46c1-b832-c550b092f02b\") " pod="e2e-models-e2e-worker-w2-09a20a/distinct-2-w2-09a20a-kserve-779d648d75-mshl7" Oct 31 04:19:27.203299 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:27.203291 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c81f5d58-c5c7-46c1-b832-c550b092f02b-dshm\") pod \"distinct-2-w2-09a20a-kserve-779d648d75-mshl7\" (UID: \"c81f5d58-c5c7-46c1-b832-c550b092f02b\") " pod="e2e-models-e2e-worker-w2-09a20a/distinct-2-w2-09a20a-kserve-779d648d75-mshl7" Oct 31 04:19:27.304781 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:27.304737 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c81f5d58-c5c7-46c1-b832-c550b092f02b-dshm\") pod \"distinct-2-w2-09a20a-kserve-779d648d75-mshl7\" (UID: \"c81f5d58-c5c7-46c1-b832-c550b092f02b\") " pod="e2e-models-e2e-worker-w2-09a20a/distinct-2-w2-09a20a-kserve-779d648d75-mshl7" Oct 31 04:19:27.304946 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:27.304872 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c81f5d58-c5c7-46c1-b832-c550b092f02b-home\") pod \"distinct-2-w2-09a20a-kserve-779d648d75-mshl7\" (UID: \"c81f5d58-c5c7-46c1-b832-c550b092f02b\") " pod="e2e-models-e2e-worker-w2-09a20a/distinct-2-w2-09a20a-kserve-779d648d75-mshl7" Oct 31 04:19:27.304946 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:27.304902 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c81f5d58-c5c7-46c1-b832-c550b092f02b-tmp-dir\") pod \"distinct-2-w2-09a20a-kserve-779d648d75-mshl7\" (UID: \"c81f5d58-c5c7-46c1-b832-c550b092f02b\") " pod="e2e-models-e2e-worker-w2-09a20a/distinct-2-w2-09a20a-kserve-779d648d75-mshl7" Oct 31 04:19:27.305030 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:27.304943 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c81f5d58-c5c7-46c1-b832-c550b092f02b-model-cache\") pod \"distinct-2-w2-09a20a-kserve-779d648d75-mshl7\" (UID: \"c81f5d58-c5c7-46c1-b832-c550b092f02b\") " pod="e2e-models-e2e-worker-w2-09a20a/distinct-2-w2-09a20a-kserve-779d648d75-mshl7" Oct 31 04:19:27.305082 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:27.305033 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c81f5d58-c5c7-46c1-b832-c550b092f02b-tls-certs\") pod \"distinct-2-w2-09a20a-kserve-779d648d75-mshl7\" (UID: \"c81f5d58-c5c7-46c1-b832-c550b092f02b\") " pod="e2e-models-e2e-worker-w2-09a20a/distinct-2-w2-09a20a-kserve-779d648d75-mshl7" Oct 31 04:19:27.305082 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:27.305063 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sswx2\" (UniqueName: \"kubernetes.io/projected/c81f5d58-c5c7-46c1-b832-c550b092f02b-kube-api-access-sswx2\") pod \"distinct-2-w2-09a20a-kserve-779d648d75-mshl7\" (UID: \"c81f5d58-c5c7-46c1-b832-c550b092f02b\") " pod="e2e-models-e2e-worker-w2-09a20a/distinct-2-w2-09a20a-kserve-779d648d75-mshl7" Oct 31 04:19:27.305301 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:27.305276 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c81f5d58-c5c7-46c1-b832-c550b092f02b-home\") pod \"distinct-2-w2-09a20a-kserve-779d648d75-mshl7\" (UID: \"c81f5d58-c5c7-46c1-b832-c550b092f02b\") " pod="e2e-models-e2e-worker-w2-09a20a/distinct-2-w2-09a20a-kserve-779d648d75-mshl7" Oct 31 04:19:27.305406 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:27.305297 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c81f5d58-c5c7-46c1-b832-c550b092f02b-tmp-dir\") pod \"distinct-2-w2-09a20a-kserve-779d648d75-mshl7\" (UID: \"c81f5d58-c5c7-46c1-b832-c550b092f02b\") " pod="e2e-models-e2e-worker-w2-09a20a/distinct-2-w2-09a20a-kserve-779d648d75-mshl7" Oct 31 04:19:27.305406 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:27.305359 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c81f5d58-c5c7-46c1-b832-c550b092f02b-model-cache\") pod \"distinct-2-w2-09a20a-kserve-779d648d75-mshl7\" (UID: \"c81f5d58-c5c7-46c1-b832-c550b092f02b\") " pod="e2e-models-e2e-worker-w2-09a20a/distinct-2-w2-09a20a-kserve-779d648d75-mshl7" Oct 31 04:19:27.307001 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:27.306978 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c81f5d58-c5c7-46c1-b832-c550b092f02b-dshm\") pod \"distinct-2-w2-09a20a-kserve-779d648d75-mshl7\" (UID: \"c81f5d58-c5c7-46c1-b832-c550b092f02b\") " pod="e2e-models-e2e-worker-w2-09a20a/distinct-2-w2-09a20a-kserve-779d648d75-mshl7" Oct 31 04:19:27.307556 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:27.307534 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c81f5d58-c5c7-46c1-b832-c550b092f02b-tls-certs\") pod \"distinct-2-w2-09a20a-kserve-779d648d75-mshl7\" (UID: \"c81f5d58-c5c7-46c1-b832-c550b092f02b\") " pod="e2e-models-e2e-worker-w2-09a20a/distinct-2-w2-09a20a-kserve-779d648d75-mshl7" Oct 31 04:19:27.331233 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:27.331198 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sswx2\" (UniqueName: \"kubernetes.io/projected/c81f5d58-c5c7-46c1-b832-c550b092f02b-kube-api-access-sswx2\") pod \"distinct-2-w2-09a20a-kserve-779d648d75-mshl7\" (UID: \"c81f5d58-c5c7-46c1-b832-c550b092f02b\") " pod="e2e-models-e2e-worker-w2-09a20a/distinct-2-w2-09a20a-kserve-779d648d75-mshl7" Oct 31 04:19:27.450404 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:27.449700 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-09a20a/distinct-2-w2-09a20a-kserve-779d648d75-mshl7" Oct 31 04:19:27.811786 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:27.811285 4376 patch_prober.go:29] interesting pod/e2e-worker-w1-439989-openshift-default-8448c6ff99-bzbxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Oct 31 04:19:27.811786 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:27.811358 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-bzbxb" podUID="1287989d-3ca3-48c0-bfa1-3e6e27b7f06d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Oct 31 04:19:27.947545 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:27.947506 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-09a20a/distinct-2-w2-09a20a-kserve-779d648d75-mshl7"] Oct 31 04:19:28.902362 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:28.894915 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-09a20a/distinct-2-w2-09a20a-kserve-779d648d75-mshl7" event={"ID":"c81f5d58-c5c7-46c1-b832-c550b092f02b","Type":"ContainerStarted","Data":"0b2f66ca1a9f304890933a12fda4dc76a29093853423773883f4866b86978ffd"} Oct 31 04:19:28.902362 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:28.894972 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-09a20a/distinct-2-w2-09a20a-kserve-779d648d75-mshl7" event={"ID":"c81f5d58-c5c7-46c1-b832-c550b092f02b","Type":"ContainerStarted","Data":"1a909083c986885280be5691dfc0f6a97c4ec21b654e1d93a5ab3a37ad209848"} Oct 31 04:19:28.949830 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:28.948824 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-09a20a/distinct-2-w2-09a20a-kserve-779d648d75-mshl7" podStartSLOduration=1.948804961 podStartE2EDuration="1.948804961s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:19:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:19:28.946885271 +0000 UTC m=+916.661219888" watchObservedRunningTime="2026-10-31 04:19:28.948804961 +0000 UTC m=+916.663139579" Oct 31 04:19:29.450879 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:29.450843 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-09a20a/distinct-2-w2-09a20a-kserve-779d648d75-mshl7" Oct 31 04:19:29.853549 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:29.853450 4376 patch_prober.go:29] interesting pod/e2e-worker-w1-439989-openshift-default-8448c6ff99-bzbxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 04:19:29.853549 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:29.853503 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-bzbxb" podUID="1287989d-3ca3-48c0-bfa1-3e6e27b7f06d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 04:19:30.047683 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:30.047653 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-bzbxb" Oct 31 04:19:30.048107 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:30.048002 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-bzbxb" Oct 31 04:19:30.049444 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:30.049421 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-bzbxb" Oct 31 04:19:34.544996 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:34.544963 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-nggns"] Oct 31 04:19:34.549005 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:34.548982 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-nggns" Oct 31 04:19:34.566616 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:34.566539 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-cqvwg"] Oct 31 04:19:34.581445 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:34.575070 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-cqvwg" Oct 31 04:19:34.595728 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:34.585549 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-nggns"] Oct 31 04:19:34.596562 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:34.596534 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-cqvwg"] Oct 31 04:19:34.605973 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:34.605945 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b871815-e7d5-42e1-a7b7-4777c82df055-istio-data\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-cqvwg\" (UID: \"4b871815-e7d5-42e1-a7b7-4777c82df055\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-cqvwg" Oct 31 04:19:34.606112 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:34.605996 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b871815-e7d5-42e1-a7b7-4777c82df055-istio-podinfo\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-cqvwg\" (UID: \"4b871815-e7d5-42e1-a7b7-4777c82df055\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-cqvwg" Oct 31 04:19:34.606157 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:34.606105 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d001c91-1967-4fb1-94f4-8f8a8db7216b-workload-certs\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-nggns\" (UID: \"8d001c91-1967-4fb1-94f4-8f8a8db7216b\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-nggns" Oct 31 04:19:34.606157 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:34.606138 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d001c91-1967-4fb1-94f4-8f8a8db7216b-istio-data\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-nggns\" (UID: \"8d001c91-1967-4fb1-94f4-8f8a8db7216b\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-nggns" Oct 31 04:19:34.606256 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:34.606185 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d001c91-1967-4fb1-94f4-8f8a8db7216b-istio-podinfo\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-nggns\" (UID: \"8d001c91-1967-4fb1-94f4-8f8a8db7216b\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-nggns" Oct 31 04:19:34.606307 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:34.606262 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b871815-e7d5-42e1-a7b7-4777c82df055-istiod-ca-cert\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-cqvwg\" (UID: \"4b871815-e7d5-42e1-a7b7-4777c82df055\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-cqvwg" Oct 31 04:19:34.606386 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:34.606368 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b871815-e7d5-42e1-a7b7-4777c82df055-istio-token\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-cqvwg\" (UID: \"4b871815-e7d5-42e1-a7b7-4777c82df055\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-cqvwg" Oct 31 04:19:34.606438 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:34.606400 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-662b5\" (UniqueName: \"kubernetes.io/projected/8d001c91-1967-4fb1-94f4-8f8a8db7216b-kube-api-access-662b5\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-nggns\" (UID: \"8d001c91-1967-4fb1-94f4-8f8a8db7216b\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-nggns" Oct 31 04:19:34.606438 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:34.606456 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k976b\" (UniqueName: \"kubernetes.io/projected/4b871815-e7d5-42e1-a7b7-4777c82df055-kube-api-access-k976b\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-cqvwg\" (UID: \"4b871815-e7d5-42e1-a7b7-4777c82df055\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-cqvwg" Oct 31 04:19:34.606618 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:34.606530 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d001c91-1967-4fb1-94f4-8f8a8db7216b-istiod-ca-cert\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-nggns\" (UID: \"8d001c91-1967-4fb1-94f4-8f8a8db7216b\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-nggns" Oct 31 04:19:34.606673 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:34.606656 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d001c91-1967-4fb1-94f4-8f8a8db7216b-credential-socket\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-nggns\" (UID: \"8d001c91-1967-4fb1-94f4-8f8a8db7216b\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-nggns" Oct 31 04:19:34.606783 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:34.606765 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b871815-e7d5-42e1-a7b7-4777c82df055-credential-socket\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-cqvwg\" (UID: \"4b871815-e7d5-42e1-a7b7-4777c82df055\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-cqvwg" Oct 31 04:19:34.606845 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:34.606796 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d001c91-1967-4fb1-94f4-8f8a8db7216b-istio-envoy\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-nggns\" (UID: \"8d001c91-1967-4fb1-94f4-8f8a8db7216b\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-nggns" Oct 31 04:19:34.606895 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:34.606855 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b871815-e7d5-42e1-a7b7-4777c82df055-istio-envoy\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-cqvwg\" (UID: \"4b871815-e7d5-42e1-a7b7-4777c82df055\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-cqvwg" Oct 31 04:19:34.607006 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:34.606991 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d001c91-1967-4fb1-94f4-8f8a8db7216b-workload-socket\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-nggns\" (UID: \"8d001c91-1967-4fb1-94f4-8f8a8db7216b\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-nggns" Oct 31 04:19:34.607063 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:34.607014 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b871815-e7d5-42e1-a7b7-4777c82df055-workload-socket\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-cqvwg\" (UID: \"4b871815-e7d5-42e1-a7b7-4777c82df055\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-cqvwg" Oct 31 04:19:34.607063 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:34.607036 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d001c91-1967-4fb1-94f4-8f8a8db7216b-istio-token\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-nggns\" (UID: \"8d001c91-1967-4fb1-94f4-8f8a8db7216b\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-nggns" Oct 31 04:19:34.607152 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:34.607083 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b871815-e7d5-42e1-a7b7-4777c82df055-workload-certs\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-cqvwg\" (UID: \"4b871815-e7d5-42e1-a7b7-4777c82df055\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-cqvwg" Oct 31 04:19:34.708852 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:34.708807 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b871815-e7d5-42e1-a7b7-4777c82df055-workload-certs\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-cqvwg\" (UID: \"4b871815-e7d5-42e1-a7b7-4777c82df055\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-cqvwg" Oct 31 04:19:34.709044 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:34.708878 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b871815-e7d5-42e1-a7b7-4777c82df055-istio-data\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-cqvwg\" (UID: \"4b871815-e7d5-42e1-a7b7-4777c82df055\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-cqvwg" Oct 31 04:19:34.709044 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:34.708930 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b871815-e7d5-42e1-a7b7-4777c82df055-istio-podinfo\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-cqvwg\" (UID: \"4b871815-e7d5-42e1-a7b7-4777c82df055\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-cqvwg" Oct 31 04:19:34.709151 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:34.709111 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d001c91-1967-4fb1-94f4-8f8a8db7216b-workload-certs\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-nggns\" (UID: \"8d001c91-1967-4fb1-94f4-8f8a8db7216b\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-nggns" Oct 31 04:19:34.709203 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:34.709183 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d001c91-1967-4fb1-94f4-8f8a8db7216b-istio-data\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-nggns\" (UID: \"8d001c91-1967-4fb1-94f4-8f8a8db7216b\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-nggns" Oct 31 04:19:34.709299 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:34.709282 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d001c91-1967-4fb1-94f4-8f8a8db7216b-istio-podinfo\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-nggns\" (UID: \"8d001c91-1967-4fb1-94f4-8f8a8db7216b\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-nggns" Oct 31 04:19:34.709382 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:34.709363 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b871815-e7d5-42e1-a7b7-4777c82df055-istiod-ca-cert\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-cqvwg\" (UID: \"4b871815-e7d5-42e1-a7b7-4777c82df055\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-cqvwg" Oct 31 04:19:34.709444 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:34.709396 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b871815-e7d5-42e1-a7b7-4777c82df055-istio-token\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-cqvwg\" (UID: \"4b871815-e7d5-42e1-a7b7-4777c82df055\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-cqvwg" Oct 31 04:19:34.709444 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:34.709423 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b871815-e7d5-42e1-a7b7-4777c82df055-istio-data\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-cqvwg\" (UID: \"4b871815-e7d5-42e1-a7b7-4777c82df055\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-cqvwg" Oct 31 04:19:34.709547 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:34.709441 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-662b5\" (UniqueName: \"kubernetes.io/projected/8d001c91-1967-4fb1-94f4-8f8a8db7216b-kube-api-access-662b5\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-nggns\" (UID: \"8d001c91-1967-4fb1-94f4-8f8a8db7216b\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-nggns" Oct 31 04:19:34.709547 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:34.709488 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k976b\" (UniqueName: \"kubernetes.io/projected/4b871815-e7d5-42e1-a7b7-4777c82df055-kube-api-access-k976b\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-cqvwg\" (UID: \"4b871815-e7d5-42e1-a7b7-4777c82df055\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-cqvwg" Oct 31 04:19:34.709547 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:34.709515 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d001c91-1967-4fb1-94f4-8f8a8db7216b-workload-certs\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-nggns\" (UID: \"8d001c91-1967-4fb1-94f4-8f8a8db7216b\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-nggns" Oct 31 04:19:34.709547 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:34.709533 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d001c91-1967-4fb1-94f4-8f8a8db7216b-istiod-ca-cert\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-nggns\" (UID: \"8d001c91-1967-4fb1-94f4-8f8a8db7216b\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-nggns" Oct 31 04:19:34.709747 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:34.709573 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d001c91-1967-4fb1-94f4-8f8a8db7216b-credential-socket\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-nggns\" (UID: \"8d001c91-1967-4fb1-94f4-8f8a8db7216b\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-nggns" Oct 31 04:19:34.709747 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:34.709652 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b871815-e7d5-42e1-a7b7-4777c82df055-credential-socket\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-cqvwg\" (UID: \"4b871815-e7d5-42e1-a7b7-4777c82df055\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-cqvwg" Oct 31 04:19:34.709747 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:34.709696 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d001c91-1967-4fb1-94f4-8f8a8db7216b-istio-envoy\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-nggns\" (UID: \"8d001c91-1967-4fb1-94f4-8f8a8db7216b\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-nggns" Oct 31 04:19:34.709895 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:34.709772 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b871815-e7d5-42e1-a7b7-4777c82df055-istio-envoy\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-cqvwg\" (UID: \"4b871815-e7d5-42e1-a7b7-4777c82df055\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-cqvwg" Oct 31 04:19:34.709895 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:34.709651 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d001c91-1967-4fb1-94f4-8f8a8db7216b-istio-data\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-nggns\" (UID: \"8d001c91-1967-4fb1-94f4-8f8a8db7216b\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-nggns" Oct 31 04:19:34.710090 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:34.709888 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d001c91-1967-4fb1-94f4-8f8a8db7216b-workload-socket\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-nggns\" (UID: \"8d001c91-1967-4fb1-94f4-8f8a8db7216b\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-nggns" Oct 31 04:19:34.710090 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:34.709391 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b871815-e7d5-42e1-a7b7-4777c82df055-workload-certs\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-cqvwg\" (UID: \"4b871815-e7d5-42e1-a7b7-4777c82df055\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-cqvwg" Oct 31 04:19:34.710090 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:34.709937 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b871815-e7d5-42e1-a7b7-4777c82df055-workload-socket\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-cqvwg\" (UID: \"4b871815-e7d5-42e1-a7b7-4777c82df055\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-cqvwg" Oct 31 04:19:34.710090 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:34.709983 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d001c91-1967-4fb1-94f4-8f8a8db7216b-istio-token\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-nggns\" (UID: \"8d001c91-1967-4fb1-94f4-8f8a8db7216b\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-nggns" Oct 31 04:19:34.710290 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:34.710093 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d001c91-1967-4fb1-94f4-8f8a8db7216b-credential-socket\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-nggns\" (UID: \"8d001c91-1967-4fb1-94f4-8f8a8db7216b\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-nggns" Oct 31 04:19:34.710362 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:34.710318 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d001c91-1967-4fb1-94f4-8f8a8db7216b-workload-socket\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-nggns\" (UID: \"8d001c91-1967-4fb1-94f4-8f8a8db7216b\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-nggns" Oct 31 04:19:34.710407 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:34.710361 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b871815-e7d5-42e1-a7b7-4777c82df055-credential-socket\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-cqvwg\" (UID: \"4b871815-e7d5-42e1-a7b7-4777c82df055\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-cqvwg" Oct 31 04:19:34.710602 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:34.710583 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b871815-e7d5-42e1-a7b7-4777c82df055-workload-socket\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-cqvwg\" (UID: \"4b871815-e7d5-42e1-a7b7-4777c82df055\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-cqvwg" Oct 31 04:19:34.710984 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:34.710956 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d001c91-1967-4fb1-94f4-8f8a8db7216b-istiod-ca-cert\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-nggns\" (UID: \"8d001c91-1967-4fb1-94f4-8f8a8db7216b\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-nggns" Oct 31 04:19:34.711068 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:34.710971 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b871815-e7d5-42e1-a7b7-4777c82df055-istiod-ca-cert\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-cqvwg\" (UID: \"4b871815-e7d5-42e1-a7b7-4777c82df055\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-cqvwg" Oct 31 04:19:34.712172 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:34.712143 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b871815-e7d5-42e1-a7b7-4777c82df055-istio-envoy\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-cqvwg\" (UID: \"4b871815-e7d5-42e1-a7b7-4777c82df055\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-cqvwg" Oct 31 04:19:34.712309 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:34.712285 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d001c91-1967-4fb1-94f4-8f8a8db7216b-istio-podinfo\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-nggns\" (UID: \"8d001c91-1967-4fb1-94f4-8f8a8db7216b\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-nggns" Oct 31 04:19:34.712424 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:34.712406 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d001c91-1967-4fb1-94f4-8f8a8db7216b-istio-envoy\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-nggns\" (UID: \"8d001c91-1967-4fb1-94f4-8f8a8db7216b\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-nggns" Oct 31 04:19:34.712891 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:34.712874 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b871815-e7d5-42e1-a7b7-4777c82df055-istio-podinfo\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-cqvwg\" (UID: \"4b871815-e7d5-42e1-a7b7-4777c82df055\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-cqvwg" Oct 31 04:19:34.735185 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:34.735146 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b871815-e7d5-42e1-a7b7-4777c82df055-istio-token\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-cqvwg\" (UID: \"4b871815-e7d5-42e1-a7b7-4777c82df055\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-cqvwg" Oct 31 04:19:34.735362 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:34.735214 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d001c91-1967-4fb1-94f4-8f8a8db7216b-istio-token\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-nggns\" (UID: \"8d001c91-1967-4fb1-94f4-8f8a8db7216b\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-nggns" Oct 31 04:19:34.735362 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:34.735287 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k976b\" (UniqueName: \"kubernetes.io/projected/4b871815-e7d5-42e1-a7b7-4777c82df055-kube-api-access-k976b\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-cqvwg\" (UID: \"4b871815-e7d5-42e1-a7b7-4777c82df055\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-cqvwg" Oct 31 04:19:34.735454 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:34.735388 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-662b5\" (UniqueName: \"kubernetes.io/projected/8d001c91-1967-4fb1-94f4-8f8a8db7216b-kube-api-access-662b5\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-nggns\" (UID: \"8d001c91-1967-4fb1-94f4-8f8a8db7216b\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-nggns" Oct 31 04:19:34.876844 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:34.876814 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-nggns" Oct 31 04:19:34.902618 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:34.902585 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-cqvwg" Oct 31 04:19:35.201720 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:19:35.201673 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d001c91_1967_4fb1_94f4_8f8a8db7216b.slice/crio-6820bb766a0abdae176ff0b224dfe8b652c9a2ae69ddc99c484006f16d3dafdd WatchSource:0}: Error finding container 6820bb766a0abdae176ff0b224dfe8b652c9a2ae69ddc99c484006f16d3dafdd: Status 404 returned error can't find the container with id 6820bb766a0abdae176ff0b224dfe8b652c9a2ae69ddc99c484006f16d3dafdd Oct 31 04:19:35.202902 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:35.202874 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-nggns"] Oct 31 04:19:35.204539 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:35.204507 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:19:35.204642 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:35.204587 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:19:35.204642 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:35.204634 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:19:35.228109 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:19:35.227210 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b871815_e7d5_42e1_a7b7_4777c82df055.slice/crio-2ee2e6604bbf7c8ee7d5c2f548c618dc0f4c1eef5d3e27b6a5a6193e5b55f5dc WatchSource:0}: Error finding container 2ee2e6604bbf7c8ee7d5c2f548c618dc0f4c1eef5d3e27b6a5a6193e5b55f5dc: Status 404 returned error can't find the container with id 2ee2e6604bbf7c8ee7d5c2f548c618dc0f4c1eef5d3e27b6a5a6193e5b55f5dc Oct 31 04:19:35.228109 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:35.227862 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-cqvwg"] Oct 31 04:19:35.229729 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:35.229683 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:19:35.229857 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:35.229760 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:19:35.229857 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:35.229810 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:19:35.959355 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:35.953493 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-cqvwg" event={"ID":"4b871815-e7d5-42e1-a7b7-4777c82df055","Type":"ContainerStarted","Data":"5f22b0d5ad7e86b40c8950f28b78f1537b10c187b38651b9b3faff39b698d58b"} Oct 31 04:19:35.959355 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:35.953566 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-cqvwg" event={"ID":"4b871815-e7d5-42e1-a7b7-4777c82df055","Type":"ContainerStarted","Data":"2ee2e6604bbf7c8ee7d5c2f548c618dc0f4c1eef5d3e27b6a5a6193e5b55f5dc"} Oct 31 04:19:35.963350 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:35.962319 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-nggns" event={"ID":"8d001c91-1967-4fb1-94f4-8f8a8db7216b","Type":"ContainerStarted","Data":"6896af4706d0f2de28de8772f390946c6406e8b80f07603c9ccc5934a88c144f"} Oct 31 04:19:35.963350 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:35.962415 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-nggns" event={"ID":"8d001c91-1967-4fb1-94f4-8f8a8db7216b","Type":"ContainerStarted","Data":"6820bb766a0abdae176ff0b224dfe8b652c9a2ae69ddc99c484006f16d3dafdd"} Oct 31 04:19:36.007789 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:36.007736 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-cqvwg" podStartSLOduration=2.007713262 podStartE2EDuration="2.007713262s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:19:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:19:36.004702458 +0000 UTC m=+923.719037074" watchObservedRunningTime="2026-10-31 04:19:36.007713262 +0000 UTC m=+923.722047880" Oct 31 04:19:36.881303 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:36.881255 4376 patch_prober.go:29] interesting pod/e2e-worker-w1-439989-openshift-default-8448c6ff99-nggns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 31 04:19:36.881623 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:36.881581 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-nggns" podUID="8d001c91-1967-4fb1-94f4-8f8a8db7216b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 31 04:19:36.882118 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:36.882031 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-nggns" Oct 31 04:19:36.903068 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:36.903034 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-cqvwg" Oct 31 04:19:36.904260 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:36.904228 4376 patch_prober.go:29] interesting pod/e2e-worker-w1-439989-openshift-default-8448c6ff99-cqvwg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Oct 31 04:19:36.904441 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:36.904422 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-cqvwg" podUID="4b871815-e7d5-42e1-a7b7-4777c82df055" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Oct 31 04:19:37.455354 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:37.452816 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-09a20a/distinct-2-w2-09a20a-kserve-779d648d75-mshl7" Oct 31 04:19:37.503725 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:37.503689 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-09a20a/distinct-2-w2-09a20a-kserve-779d648d75-mshl7" Oct 31 04:19:37.603037 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:37.597107 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-nggns" podStartSLOduration=3.5970847360000002 podStartE2EDuration="3.597084736s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:19:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:19:36.064781769 +0000 UTC m=+923.779116387" watchObservedRunningTime="2026-10-31 04:19:37.597084736 +0000 UTC m=+925.311419355" Oct 31 04:19:37.883919 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:37.883874 4376 patch_prober.go:29] interesting pod/e2e-worker-w1-439989-openshift-default-8448c6ff99-nggns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 31 04:19:37.884237 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:37.884190 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-nggns" podUID="8d001c91-1967-4fb1-94f4-8f8a8db7216b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 31 04:19:37.910053 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:37.910020 4376 patch_prober.go:29] interesting pod/e2e-worker-w1-439989-openshift-default-8448c6ff99-cqvwg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Oct 31 04:19:37.910367 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:37.910313 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-cqvwg" podUID="4b871815-e7d5-42e1-a7b7-4777c82df055" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Oct 31 04:19:38.023265 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:38.023238 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-09a20a/distinct-2-w2-09a20a-kserve-779d648d75-mshl7" Oct 31 04:19:38.883355 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:38.879776 4376 patch_prober.go:29] interesting pod/e2e-worker-w1-439989-openshift-default-8448c6ff99-nggns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 31 04:19:38.883355 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:38.879836 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-nggns" podUID="8d001c91-1967-4fb1-94f4-8f8a8db7216b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 31 04:19:38.903604 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:38.903572 4376 patch_prober.go:29] interesting pod/e2e-worker-w1-439989-openshift-default-8448c6ff99-cqvwg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Oct 31 04:19:38.903768 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:38.903622 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-cqvwg" podUID="4b871815-e7d5-42e1-a7b7-4777c82df055" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Oct 31 04:19:39.880361 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:39.879720 4376 patch_prober.go:29] interesting pod/e2e-worker-w1-439989-openshift-default-8448c6ff99-nggns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 31 04:19:39.880361 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:39.879804 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-nggns" podUID="8d001c91-1967-4fb1-94f4-8f8a8db7216b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 31 04:19:39.903545 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:39.903508 4376 patch_prober.go:29] interesting pod/e2e-worker-w1-439989-openshift-default-8448c6ff99-cqvwg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Oct 31 04:19:39.903988 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:39.903575 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-cqvwg" podUID="4b871815-e7d5-42e1-a7b7-4777c82df055" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Oct 31 04:19:40.881780 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:40.881740 4376 patch_prober.go:29] interesting pod/e2e-worker-w1-439989-openshift-default-8448c6ff99-nggns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 31 04:19:40.882171 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:40.882109 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-nggns" podUID="8d001c91-1967-4fb1-94f4-8f8a8db7216b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 31 04:19:40.903006 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:40.902972 4376 patch_prober.go:29] interesting pod/e2e-worker-w1-439989-openshift-default-8448c6ff99-cqvwg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Oct 31 04:19:40.903159 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:40.903030 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-cqvwg" podUID="4b871815-e7d5-42e1-a7b7-4777c82df055" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Oct 31 04:19:41.014578 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:19:41.014543 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 04:19:41.877712 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:41.877666 4376 patch_prober.go:29] interesting pod/e2e-worker-w1-439989-openshift-default-8448c6ff99-nggns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 31 04:19:41.877895 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:41.877732 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-nggns" podUID="8d001c91-1967-4fb1-94f4-8f8a8db7216b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 31 04:19:41.902965 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:41.902922 4376 patch_prober.go:29] interesting pod/e2e-worker-w1-439989-openshift-default-8448c6ff99-cqvwg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Oct 31 04:19:41.903137 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:41.902984 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-cqvwg" podUID="4b871815-e7d5-42e1-a7b7-4777c82df055" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Oct 31 04:19:42.930907 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:42.930870 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-cqvwg" Oct 31 04:19:42.932343 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:42.932303 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-cqvwg" Oct 31 04:19:42.933063 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:42.933042 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-cqvwg" Oct 31 04:19:43.062545 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:43.062393 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-nggns" Oct 31 04:19:43.064598 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:43.063861 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-nggns" Oct 31 04:19:43.064713 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:43.064687 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-nggns" Oct 31 04:19:49.176493 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:49.176431 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-09a20a/distinct-w2-09a20a-kserve-76557f6448-l8d69" podUID="7d298000-b65b-45fd-a1ee-092e47a7eb24" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:19:49.393699 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:49.393655 4376 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-c4006d/distinct-2-w4-c4006d-kserve-776b5b899-kpl6z" podUID="dc53083d-eadd-4cb8-8633-35e02d400b9c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": context deadline exceeded" Oct 31 04:19:49.419498 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:49.419446 4376 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-09a20a/premium-w2-09a20a-kserve-57c45db74f-s8glf" podUID="a3f9d0a0-a106-48bb-b6f2-a980c3aff4e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:19:49.605609 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:49.605479 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-d7npm" podUID="174171e7-100c-4b84-becd-0176672f1805" containerName="main" probeResult="failure" output="Get \"https://10.133.0.34:8000/ready\": context deadline exceeded" Oct 31 04:19:49.616474 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:49.616413 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-09a20a/premium-w2-09a20a-kserve-57c45db74f-s8glf" podUID="a3f9d0a0-a106-48bb-b6f2-a980c3aff4e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Oct 31 04:19:49.635426 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:49.635379 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c4006d/simulator-w4-c4006d-kserve-595455485b-4h46q" podUID="576e2c1f-1c80-43a9-a136-3692776642ae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:19:49.784488 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:49.784434 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-439989/premium-w1-439989-kserve-f9fc54cdf-sn2pp" podUID="17b2f181-2e66-4ce2-95b0-cb213f11a0e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:19:49.821469 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:49.821421 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4c2387/distinct-w3-4c2387-kserve-768bdd84c7-6jmtl" podUID="b2c75691-4cf4-484b-96e2-3c2503b8d78c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded" Oct 31 04:19:50.023820 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:50.023777 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-09a20a/distinct-2-w2-09a20a-kserve-779d648d75-mshl7" podUID="c81f5d58-c5c7-46c1-b832-c550b092f02b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded" Oct 31 04:19:50.130462 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:50.130417 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c4006d/distinct-2-w4-c4006d-kserve-776b5b899-kpl6z" podUID="dc53083d-eadd-4cb8-8633-35e02d400b9c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:19:50.181482 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:50.181424 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-09a20a/distinct-w2-09a20a-kserve-76557f6448-l8d69" podUID="7d298000-b65b-45fd-a1ee-092e47a7eb24" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:19:50.183514 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:50.183041 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-09a20a/distinct-w2-09a20a-kserve-76557f6448-l8d69" Oct 31 04:19:50.669356 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:50.629022 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-d7npm" podUID="174171e7-100c-4b84-becd-0176672f1805" containerName="main" probeResult="failure" output="Get \"https://10.133.0.34:8000/ready\": context deadline exceeded" Oct 31 04:19:50.717355 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:50.670080 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c4006d/simulator-w4-c4006d-kserve-595455485b-4h46q" podUID="576e2c1f-1c80-43a9-a136-3692776642ae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:19:50.717355 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:50.672294 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-09a20a/premium-w2-09a20a-kserve-57c45db74f-s8glf" podUID="a3f9d0a0-a106-48bb-b6f2-a980c3aff4e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:19:50.748651 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:50.748608 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-d7npm" Oct 31 04:19:50.753625 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:50.753579 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c4006d/simulator-w4-c4006d-kserve-595455485b-4h46q" Oct 31 04:19:50.757402 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:50.757371 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-09a20a/premium-w2-09a20a-kserve-57c45db74f-s8glf" Oct 31 04:19:50.788355 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:50.787403 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-439989/premium-w1-439989-kserve-f9fc54cdf-sn2pp" podUID="17b2f181-2e66-4ce2-95b0-cb213f11a0e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Oct 31 04:19:50.788355 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:50.788198 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-439989/premium-w1-439989-kserve-f9fc54cdf-sn2pp" Oct 31 04:19:51.267355 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:51.225255 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-09a20a/distinct-w2-09a20a-kserve-76557f6448-l8d69" podUID="7d298000-b65b-45fd-a1ee-092e47a7eb24" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:19:51.267355 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:51.226351 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c4006d/distinct-2-w4-c4006d-kserve-776b5b899-kpl6z" podUID="dc53083d-eadd-4cb8-8633-35e02d400b9c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 31 04:19:51.267355 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:51.247991 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c4006d/distinct-2-w4-c4006d-kserve-776b5b899-kpl6z" Oct 31 04:19:51.267355 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:51.248536 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4c2387/distinct-w3-4c2387-kserve-768bdd84c7-6jmtl" podUID="b2c75691-4cf4-484b-96e2-3c2503b8d78c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded" Oct 31 04:19:51.278872 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:51.276304 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-4c2387/distinct-w3-4c2387-kserve-768bdd84c7-6jmtl" Oct 31 04:19:51.380677 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:51.380626 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-09a20a/distinct-2-w2-09a20a-kserve-779d648d75-mshl7" podUID="c81f5d58-c5c7-46c1-b832-c550b092f02b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded" Oct 31 04:19:51.418110 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:51.418075 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-09a20a/distinct-2-w2-09a20a-kserve-779d648d75-mshl7" Oct 31 04:19:51.629354 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:51.629292 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-d7npm" podUID="174171e7-100c-4b84-becd-0176672f1805" containerName="main" probeResult="failure" output="Get \"https://10.133.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:19:51.717700 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:51.708033 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c4006d/simulator-w4-c4006d-kserve-595455485b-4h46q" podUID="576e2c1f-1c80-43a9-a136-3692776642ae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:19:51.924744 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:51.923983 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c4006d/distinct-2-w4-c4006d-kserve-776b5b899-kpl6z" Oct 31 04:19:51.925634 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:51.925556 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-09a20a/premium-w2-09a20a-kserve-57c45db74f-s8glf" podUID="a3f9d0a0-a106-48bb-b6f2-a980c3aff4e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:19:51.943238 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:51.943209 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-09a20a/distinct-w2-09a20a-kserve-76557f6448-l8d69" Oct 31 04:19:51.944127 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:51.944108 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c4006d/simulator-w4-c4006d-kserve-595455485b-4h46q" Oct 31 04:19:51.946637 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:51.946605 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-439989/premium-w1-439989-kserve-f9fc54cdf-sn2pp" podUID="17b2f181-2e66-4ce2-95b0-cb213f11a0e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:19:51.965351 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:51.958511 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-09a20a/distinct-2-w2-09a20a-kserve-779d648d75-mshl7" Oct 31 04:19:51.978813 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:51.978784 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-d7npm" Oct 31 04:19:52.032645 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:52.032614 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-09a20a/premium-w2-09a20a-kserve-57c45db74f-s8glf" Oct 31 04:19:52.034176 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:52.034156 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-439989/premium-w1-439989-kserve-f9fc54cdf-sn2pp" Oct 31 04:19:52.112361 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:52.111864 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-4c2387/distinct-w3-4c2387-kserve-768bdd84c7-6jmtl" Oct 31 04:19:53.605281 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:53.605248 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bgh5f"] Oct 31 04:19:53.610516 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:53.610491 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bgh5f" Oct 31 04:19:53.625933 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:53.625893 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/594f2d93-2f76-4632-904d-402fa126740c-workload-certs\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bgh5f\" (UID: \"594f2d93-2f76-4632-904d-402fa126740c\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bgh5f" Oct 31 04:19:53.626077 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:53.625952 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/594f2d93-2f76-4632-904d-402fa126740c-istio-token\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bgh5f\" (UID: \"594f2d93-2f76-4632-904d-402fa126740c\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bgh5f" Oct 31 04:19:53.626077 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:53.626011 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pfm2\" (UniqueName: \"kubernetes.io/projected/594f2d93-2f76-4632-904d-402fa126740c-kube-api-access-7pfm2\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bgh5f\" (UID: \"594f2d93-2f76-4632-904d-402fa126740c\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bgh5f" Oct 31 04:19:53.626153 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:53.626090 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/594f2d93-2f76-4632-904d-402fa126740c-workload-socket\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bgh5f\" (UID: \"594f2d93-2f76-4632-904d-402fa126740c\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bgh5f" Oct 31 04:19:53.626153 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:53.626147 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/594f2d93-2f76-4632-904d-402fa126740c-istio-envoy\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bgh5f\" (UID: \"594f2d93-2f76-4632-904d-402fa126740c\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bgh5f" Oct 31 04:19:53.626231 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:53.626181 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/594f2d93-2f76-4632-904d-402fa126740c-istio-podinfo\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bgh5f\" (UID: \"594f2d93-2f76-4632-904d-402fa126740c\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bgh5f" Oct 31 04:19:53.626276 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:53.626225 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/594f2d93-2f76-4632-904d-402fa126740c-istio-data\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bgh5f\" (UID: \"594f2d93-2f76-4632-904d-402fa126740c\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bgh5f" Oct 31 04:19:53.626276 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:53.626257 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/594f2d93-2f76-4632-904d-402fa126740c-istiod-ca-cert\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bgh5f\" (UID: \"594f2d93-2f76-4632-904d-402fa126740c\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bgh5f" Oct 31 04:19:53.626377 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:53.626286 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/594f2d93-2f76-4632-904d-402fa126740c-credential-socket\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bgh5f\" (UID: \"594f2d93-2f76-4632-904d-402fa126740c\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bgh5f" Oct 31 04:19:53.632038 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:53.632007 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bgh5f"] Oct 31 04:19:53.727797 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:53.727763 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/594f2d93-2f76-4632-904d-402fa126740c-workload-socket\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bgh5f\" (UID: \"594f2d93-2f76-4632-904d-402fa126740c\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bgh5f" Oct 31 04:19:53.727998 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:53.727850 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/594f2d93-2f76-4632-904d-402fa126740c-istio-envoy\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bgh5f\" (UID: \"594f2d93-2f76-4632-904d-402fa126740c\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bgh5f" Oct 31 04:19:53.727998 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:53.727888 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/594f2d93-2f76-4632-904d-402fa126740c-istio-podinfo\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bgh5f\" (UID: \"594f2d93-2f76-4632-904d-402fa126740c\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bgh5f" Oct 31 04:19:53.727998 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:53.727930 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/594f2d93-2f76-4632-904d-402fa126740c-istio-data\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bgh5f\" (UID: \"594f2d93-2f76-4632-904d-402fa126740c\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bgh5f" Oct 31 04:19:53.727998 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:53.727951 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/594f2d93-2f76-4632-904d-402fa126740c-istiod-ca-cert\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bgh5f\" (UID: \"594f2d93-2f76-4632-904d-402fa126740c\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bgh5f" Oct 31 04:19:53.727998 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:53.727982 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/594f2d93-2f76-4632-904d-402fa126740c-credential-socket\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bgh5f\" (UID: \"594f2d93-2f76-4632-904d-402fa126740c\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bgh5f" Oct 31 04:19:53.728305 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:53.728152 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/594f2d93-2f76-4632-904d-402fa126740c-workload-certs\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bgh5f\" (UID: \"594f2d93-2f76-4632-904d-402fa126740c\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bgh5f" Oct 31 04:19:53.728305 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:53.728203 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/594f2d93-2f76-4632-904d-402fa126740c-istio-token\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bgh5f\" (UID: \"594f2d93-2f76-4632-904d-402fa126740c\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bgh5f" Oct 31 04:19:53.728305 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:53.728236 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/594f2d93-2f76-4632-904d-402fa126740c-workload-socket\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bgh5f\" (UID: \"594f2d93-2f76-4632-904d-402fa126740c\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bgh5f" Oct 31 04:19:53.728305 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:53.728261 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pfm2\" (UniqueName: \"kubernetes.io/projected/594f2d93-2f76-4632-904d-402fa126740c-kube-api-access-7pfm2\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bgh5f\" (UID: \"594f2d93-2f76-4632-904d-402fa126740c\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bgh5f" Oct 31 04:19:53.728555 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:53.728385 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/594f2d93-2f76-4632-904d-402fa126740c-istio-data\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bgh5f\" (UID: \"594f2d93-2f76-4632-904d-402fa126740c\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bgh5f" Oct 31 04:19:53.728615 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:53.728576 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/594f2d93-2f76-4632-904d-402fa126740c-credential-socket\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bgh5f\" (UID: \"594f2d93-2f76-4632-904d-402fa126740c\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bgh5f" Oct 31 04:19:53.728700 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:53.728667 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/594f2d93-2f76-4632-904d-402fa126740c-workload-certs\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bgh5f\" (UID: \"594f2d93-2f76-4632-904d-402fa126740c\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bgh5f" Oct 31 04:19:53.728811 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:53.728788 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/594f2d93-2f76-4632-904d-402fa126740c-istiod-ca-cert\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bgh5f\" (UID: \"594f2d93-2f76-4632-904d-402fa126740c\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bgh5f" Oct 31 04:19:53.731014 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:53.730986 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/594f2d93-2f76-4632-904d-402fa126740c-istio-envoy\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bgh5f\" (UID: \"594f2d93-2f76-4632-904d-402fa126740c\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bgh5f" Oct 31 04:19:53.731182 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:53.731166 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/594f2d93-2f76-4632-904d-402fa126740c-istio-podinfo\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bgh5f\" (UID: \"594f2d93-2f76-4632-904d-402fa126740c\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bgh5f" Oct 31 04:19:53.756158 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:53.756124 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pfm2\" (UniqueName: \"kubernetes.io/projected/594f2d93-2f76-4632-904d-402fa126740c-kube-api-access-7pfm2\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bgh5f\" (UID: \"594f2d93-2f76-4632-904d-402fa126740c\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bgh5f" Oct 31 04:19:53.757883 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:53.757857 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/594f2d93-2f76-4632-904d-402fa126740c-istio-token\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bgh5f\" (UID: \"594f2d93-2f76-4632-904d-402fa126740c\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bgh5f" Oct 31 04:19:53.924524 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:53.924490 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bgh5f" Oct 31 04:19:54.239085 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:19:54.239051 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod594f2d93_2f76_4632_904d_402fa126740c.slice/crio-d48c395637a943f586d76ae3be49b97d9e35e76bfe452fb4146ec72928e2a82e WatchSource:0}: Error finding container d48c395637a943f586d76ae3be49b97d9e35e76bfe452fb4146ec72928e2a82e: Status 404 returned error can't find the container with id d48c395637a943f586d76ae3be49b97d9e35e76bfe452fb4146ec72928e2a82e Oct 31 04:19:54.239967 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:54.239946 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bgh5f"] Oct 31 04:19:54.241560 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:54.241528 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:19:54.241654 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:54.241602 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:19:54.241920 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:54.241655 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:19:54.729778 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:54.729739 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bgh5f" event={"ID":"594f2d93-2f76-4632-904d-402fa126740c","Type":"ContainerStarted","Data":"a907730c8fd50e968e3d15ba2fd87f6db56a4e6102701e117130f34be25d9b5d"} Oct 31 04:19:54.730305 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:54.729804 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bgh5f" event={"ID":"594f2d93-2f76-4632-904d-402fa126740c","Type":"ContainerStarted","Data":"d48c395637a943f586d76ae3be49b97d9e35e76bfe452fb4146ec72928e2a82e"} Oct 31 04:19:54.781121 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:54.781069 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bgh5f" podStartSLOduration=1.781049995 podStartE2EDuration="1.781049995s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:19:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:19:54.777757603 +0000 UTC m=+942.492092240" watchObservedRunningTime="2026-10-31 04:19:54.781049995 +0000 UTC m=+942.495384613" Oct 31 04:19:54.925067 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:54.924985 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bgh5f" Oct 31 04:19:55.926209 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:55.926174 4376 patch_prober.go:29] interesting pod/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bgh5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Oct 31 04:19:55.926613 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:55.926230 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bgh5f" podUID="594f2d93-2f76-4632-904d-402fa126740c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Oct 31 04:19:56.925200 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:56.925164 4376 patch_prober.go:29] interesting pod/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bgh5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Oct 31 04:19:56.925401 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:56.925242 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bgh5f" podUID="594f2d93-2f76-4632-904d-402fa126740c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Oct 31 04:19:57.925052 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:57.925017 4376 patch_prober.go:29] interesting pod/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bgh5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Oct 31 04:19:57.925498 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:57.925090 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bgh5f" podUID="594f2d93-2f76-4632-904d-402fa126740c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Oct 31 04:19:58.924998 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:58.924957 4376 patch_prober.go:29] interesting pod/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bgh5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Oct 31 04:19:58.925206 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:58.925043 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bgh5f" podUID="594f2d93-2f76-4632-904d-402fa126740c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Oct 31 04:19:59.924955 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:59.924917 4376 patch_prober.go:29] interesting pod/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bgh5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Oct 31 04:19:59.925203 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:19:59.925175 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bgh5f" podUID="594f2d93-2f76-4632-904d-402fa126740c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Oct 31 04:20:00.925469 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:00.925431 4376 patch_prober.go:29] interesting pod/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bgh5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Oct 31 04:20:00.926057 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:00.926026 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bgh5f" podUID="594f2d93-2f76-4632-904d-402fa126740c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Oct 31 04:20:01.942989 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:01.942953 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bgh5f" Oct 31 04:20:01.943964 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:01.943932 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bgh5f" Oct 31 04:20:01.944881 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:01.944851 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bgh5f" Oct 31 04:20:08.598132 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:08.597181 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 04:20:10.956022 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:10.955963 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c4006d/distinct-2-w4-c4006d-kserve-776b5b899-kpl6z" podUID="dc53083d-eadd-4cb8-8633-35e02d400b9c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 31 04:20:10.966252 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:10.966210 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-09a20a/distinct-2-w2-09a20a-kserve-779d648d75-mshl7" podUID="c81f5d58-c5c7-46c1-b832-c550b092f02b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:20:10.994559 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:10.994493 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-d7npm" podUID="174171e7-100c-4b84-becd-0176672f1805" containerName="main" probeResult="failure" output="Get \"https://10.133.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:20:11.045354 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:11.036886 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-439989/premium-w1-439989-kserve-f9fc54cdf-sn2pp" podUID="17b2f181-2e66-4ce2-95b0-cb213f11a0e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Oct 31 04:20:11.045354 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:11.037625 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-09a20a/premium-w2-09a20a-kserve-57c45db74f-s8glf" podUID="a3f9d0a0-a106-48bb-b6f2-a980c3aff4e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Oct 31 04:20:11.078357 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:11.077499 4376 patch_prober.go:29] interesting pod/e2e-worker-w1-439989-openshift-default-8448c6ff99-96rgs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 04:20:11.078357 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:11.077561 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-96rgs" podUID="2b3b5806-b3d7-48a8-b2d5-fb202004605d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 04:20:11.116538 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:11.116476 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4c2387/distinct-w3-4c2387-kserve-768bdd84c7-6jmtl" podUID="b2c75691-4cf4-484b-96e2-3c2503b8d78c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded" Oct 31 04:20:11.230640 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:11.230493 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-09a20a/distinct-w2-09a20a-kserve-76557f6448-l8d69" podUID="7d298000-b65b-45fd-a1ee-092e47a7eb24" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 04:20:11.363685 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:20:11.363641 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 04:20:11.727356 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:11.726425 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c4006d/simulator-w4-c4006d-kserve-595455485b-4h46q" podUID="576e2c1f-1c80-43a9-a136-3692776642ae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:20:11.957478 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:11.957427 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c4006d/distinct-2-w4-c4006d-kserve-776b5b899-kpl6z" podUID="dc53083d-eadd-4cb8-8633-35e02d400b9c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:20:11.958788 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:11.958764 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c4006d/distinct-2-w4-c4006d-kserve-776b5b899-kpl6z" Oct 31 04:20:11.968464 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:11.968430 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-09a20a/distinct-2-w2-09a20a-kserve-779d648d75-mshl7" podUID="c81f5d58-c5c7-46c1-b832-c550b092f02b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:20:11.968962 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:11.968943 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-09a20a/distinct-2-w2-09a20a-kserve-779d648d75-mshl7" Oct 31 04:20:11.998512 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:11.998410 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-d7npm" podUID="174171e7-100c-4b84-becd-0176672f1805" containerName="main" probeResult="failure" output="Get \"https://10.133.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:20:11.999174 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:11.999151 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-d7npm" Oct 31 04:20:12.047349 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:12.044678 4376 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-439989/premium-w1-439989-kserve-f9fc54cdf-sn2pp" podUID="17b2f181-2e66-4ce2-95b0-cb213f11a0e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:20:12.047349 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:12.044803 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-09a20a/premium-w2-09a20a-kserve-57c45db74f-s8glf" podUID="a3f9d0a0-a106-48bb-b6f2-a980c3aff4e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:20:12.047349 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:12.045125 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-439989/premium-w1-439989-kserve-f9fc54cdf-sn2pp" podUID="17b2f181-2e66-4ce2-95b0-cb213f11a0e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Oct 31 04:20:12.047349 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:12.045462 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-09a20a/premium-w2-09a20a-kserve-57c45db74f-s8glf" Oct 31 04:20:12.047349 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:12.045521 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-439989/premium-w1-439989-kserve-f9fc54cdf-sn2pp" Oct 31 04:20:12.161370 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:12.119318 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4c2387/distinct-w3-4c2387-kserve-768bdd84c7-6jmtl" podUID="b2c75691-4cf4-484b-96e2-3c2503b8d78c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:20:12.161370 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:12.119745 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-4c2387/distinct-w3-4c2387-kserve-768bdd84c7-6jmtl" Oct 31 04:20:12.285120 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:12.285008 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-09a20a/distinct-w2-09a20a-kserve-76557f6448-l8d69" podUID="7d298000-b65b-45fd-a1ee-092e47a7eb24" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded" Oct 31 04:20:12.306401 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:12.302110 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-09a20a/distinct-w2-09a20a-kserve-76557f6448-l8d69" Oct 31 04:20:12.781126 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:12.781075 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c4006d/simulator-w4-c4006d-kserve-595455485b-4h46q" podUID="576e2c1f-1c80-43a9-a136-3692776642ae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded" Oct 31 04:20:12.869557 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:12.869524 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c4006d/simulator-w4-c4006d-kserve-595455485b-4h46q" Oct 31 04:20:12.971438 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:12.971394 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c4006d/distinct-2-w4-c4006d-kserve-776b5b899-kpl6z" podUID="dc53083d-eadd-4cb8-8633-35e02d400b9c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:20:12.985353 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:12.974047 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-09a20a/distinct-2-w2-09a20a-kserve-779d648d75-mshl7" podUID="c81f5d58-c5c7-46c1-b832-c550b092f02b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded" Oct 31 04:20:13.007269 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:13.007213 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-d7npm" podUID="174171e7-100c-4b84-becd-0176672f1805" containerName="main" probeResult="failure" output="Get \"https://10.133.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:20:13.056411 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:13.053429 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-439989/premium-w1-439989-kserve-f9fc54cdf-sn2pp" podUID="17b2f181-2e66-4ce2-95b0-cb213f11a0e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:20:13.056411 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:13.053740 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-09a20a/premium-w2-09a20a-kserve-57c45db74f-s8glf" podUID="a3f9d0a0-a106-48bb-b6f2-a980c3aff4e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Oct 31 04:20:13.136407 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:13.136368 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4c2387/distinct-w3-4c2387-kserve-768bdd84c7-6jmtl" podUID="b2c75691-4cf4-484b-96e2-3c2503b8d78c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:20:13.309449 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:13.287536 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-09a20a/distinct-w2-09a20a-kserve-76557f6448-l8d69" podUID="7d298000-b65b-45fd-a1ee-092e47a7eb24" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:20:13.561420 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:13.555726 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c4006d/distinct-2-w4-c4006d-kserve-776b5b899-kpl6z" Oct 31 04:20:13.699580 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:13.699511 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-439989/premium-w1-439989-kserve-f9fc54cdf-sn2pp" Oct 31 04:20:13.772854 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:13.772407 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-09a20a/distinct-w2-09a20a-kserve-76557f6448-l8d69" Oct 31 04:20:13.790676 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:13.790635 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c4006d/simulator-w4-c4006d-kserve-595455485b-4h46q" podUID="576e2c1f-1c80-43a9-a136-3692776642ae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:20:13.856007 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:13.855928 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-d7npm" Oct 31 04:20:14.218668 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:14.218614 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-09a20a/distinct-2-w2-09a20a-kserve-779d648d75-mshl7" podUID="c81f5d58-c5c7-46c1-b832-c550b092f02b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:20:14.336354 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:14.335511 4376 patch_prober.go:29] interesting pod/e2e-worker-w1-439989-openshift-default-8448c6ff99-cqvwg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 04:20:14.336354 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:14.335598 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-cqvwg" podUID="4b871815-e7d5-42e1-a7b7-4777c82df055" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 04:20:14.362352 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:14.361318 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-4c2387/distinct-w3-4c2387-kserve-768bdd84c7-6jmtl" Oct 31 04:20:14.662943 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:14.642829 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-09a20a/premium-w2-09a20a-kserve-57c45db74f-s8glf" podUID="a3f9d0a0-a106-48bb-b6f2-a980c3aff4e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Oct 31 04:20:14.946580 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:14.942507 4376 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-4c2387/distinct-w3-4c2387-kserve-768bdd84c7-6jmtl" podUID="b2c75691-4cf4-484b-96e2-3c2503b8d78c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:20:14.954530 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:14.950495 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-d7npm" podUID="174171e7-100c-4b84-becd-0176672f1805" containerName="main" probeResult="failure" output="Get \"https://10.133.0.34:8000/ready\": context deadline exceeded" Oct 31 04:20:14.959904 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:14.959871 4376 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-c4006d/simulator-w4-c4006d-kserve-595455485b-4h46q" podUID="576e2c1f-1c80-43a9-a136-3692776642ae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:20:15.022346 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:15.017370 4376 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-d7npm" podUID="174171e7-100c-4b84-becd-0176672f1805" containerName="main" probeResult="failure" output="Get \"https://10.133.0.34:8000/health\": context deadline exceeded" Oct 31 04:20:15.054066 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:15.053417 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c4006d/simulator-w4-c4006d-kserve-595455485b-4h46q" podUID="576e2c1f-1c80-43a9-a136-3692776642ae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 04:20:15.088362 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:15.077302 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c4006d/simulator-w4-c4006d-kserve-595455485b-4h46q" Oct 31 04:20:15.100501 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:15.100461 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-6rs9r"] Oct 31 04:20:15.112801 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:15.112764 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-6rs9r"] Oct 31 04:20:15.112934 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:15.112892 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-6rs9r" Oct 31 04:20:15.125730 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:15.125694 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-09a20a/distinct-2-w2-09a20a-kserve-779d648d75-mshl7" Oct 31 04:20:15.148704 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:15.146713 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28a0b862-cb3b-4f82-990e-049381077d85-workload-certs\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-6rs9r\" (UID: \"28a0b862-cb3b-4f82-990e-049381077d85\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-6rs9r" Oct 31 04:20:15.148704 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:15.146755 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28a0b862-cb3b-4f82-990e-049381077d85-istio-token\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-6rs9r\" (UID: \"28a0b862-cb3b-4f82-990e-049381077d85\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-6rs9r" Oct 31 04:20:15.148704 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:15.146796 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxkjn\" (UniqueName: \"kubernetes.io/projected/28a0b862-cb3b-4f82-990e-049381077d85-kube-api-access-pxkjn\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-6rs9r\" (UID: \"28a0b862-cb3b-4f82-990e-049381077d85\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-6rs9r" Oct 31 04:20:15.148704 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:15.146830 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28a0b862-cb3b-4f82-990e-049381077d85-istiod-ca-cert\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-6rs9r\" (UID: \"28a0b862-cb3b-4f82-990e-049381077d85\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-6rs9r" Oct 31 04:20:15.148704 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:15.146885 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28a0b862-cb3b-4f82-990e-049381077d85-credential-socket\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-6rs9r\" (UID: \"28a0b862-cb3b-4f82-990e-049381077d85\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-6rs9r" Oct 31 04:20:15.148704 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:15.146913 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28a0b862-cb3b-4f82-990e-049381077d85-istio-podinfo\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-6rs9r\" (UID: \"28a0b862-cb3b-4f82-990e-049381077d85\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-6rs9r" Oct 31 04:20:15.148704 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:15.146961 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28a0b862-cb3b-4f82-990e-049381077d85-istio-envoy\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-6rs9r\" (UID: \"28a0b862-cb3b-4f82-990e-049381077d85\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-6rs9r" Oct 31 04:20:15.148704 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:15.147006 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28a0b862-cb3b-4f82-990e-049381077d85-istio-data\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-6rs9r\" (UID: \"28a0b862-cb3b-4f82-990e-049381077d85\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-6rs9r" Oct 31 04:20:15.148704 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:15.147047 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28a0b862-cb3b-4f82-990e-049381077d85-workload-socket\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-6rs9r\" (UID: \"28a0b862-cb3b-4f82-990e-049381077d85\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-6rs9r" Oct 31 04:20:15.233000 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:15.232927 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-09a20a/premium-w2-09a20a-kserve-57c45db74f-s8glf" Oct 31 04:20:15.268974 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:15.268919 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28a0b862-cb3b-4f82-990e-049381077d85-workload-certs\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-6rs9r\" (UID: \"28a0b862-cb3b-4f82-990e-049381077d85\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-6rs9r" Oct 31 04:20:15.269257 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:15.269235 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28a0b862-cb3b-4f82-990e-049381077d85-istio-token\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-6rs9r\" (UID: \"28a0b862-cb3b-4f82-990e-049381077d85\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-6rs9r" Oct 31 04:20:15.269488 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:15.269466 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxkjn\" (UniqueName: \"kubernetes.io/projected/28a0b862-cb3b-4f82-990e-049381077d85-kube-api-access-pxkjn\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-6rs9r\" (UID: \"28a0b862-cb3b-4f82-990e-049381077d85\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-6rs9r" Oct 31 04:20:15.269644 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:15.269626 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28a0b862-cb3b-4f82-990e-049381077d85-istiod-ca-cert\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-6rs9r\" (UID: \"28a0b862-cb3b-4f82-990e-049381077d85\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-6rs9r" Oct 31 04:20:15.269808 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:15.269787 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28a0b862-cb3b-4f82-990e-049381077d85-credential-socket\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-6rs9r\" (UID: \"28a0b862-cb3b-4f82-990e-049381077d85\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-6rs9r" Oct 31 04:20:15.271053 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:15.271032 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28a0b862-cb3b-4f82-990e-049381077d85-istio-podinfo\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-6rs9r\" (UID: \"28a0b862-cb3b-4f82-990e-049381077d85\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-6rs9r" Oct 31 04:20:15.271375 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:15.271355 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28a0b862-cb3b-4f82-990e-049381077d85-istio-envoy\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-6rs9r\" (UID: \"28a0b862-cb3b-4f82-990e-049381077d85\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-6rs9r" Oct 31 04:20:15.271552 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:15.271534 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28a0b862-cb3b-4f82-990e-049381077d85-istio-data\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-6rs9r\" (UID: \"28a0b862-cb3b-4f82-990e-049381077d85\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-6rs9r" Oct 31 04:20:15.271726 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:15.271706 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28a0b862-cb3b-4f82-990e-049381077d85-workload-socket\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-6rs9r\" (UID: \"28a0b862-cb3b-4f82-990e-049381077d85\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-6rs9r" Oct 31 04:20:15.272278 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:15.272256 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28a0b862-cb3b-4f82-990e-049381077d85-workload-socket\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-6rs9r\" (UID: \"28a0b862-cb3b-4f82-990e-049381077d85\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-6rs9r" Oct 31 04:20:15.272971 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:15.272948 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28a0b862-cb3b-4f82-990e-049381077d85-workload-certs\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-6rs9r\" (UID: \"28a0b862-cb3b-4f82-990e-049381077d85\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-6rs9r" Oct 31 04:20:15.274207 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:15.274184 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28a0b862-cb3b-4f82-990e-049381077d85-istiod-ca-cert\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-6rs9r\" (UID: \"28a0b862-cb3b-4f82-990e-049381077d85\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-6rs9r" Oct 31 04:20:15.274647 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:15.274622 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28a0b862-cb3b-4f82-990e-049381077d85-credential-socket\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-6rs9r\" (UID: \"28a0b862-cb3b-4f82-990e-049381077d85\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-6rs9r" Oct 31 04:20:15.291876 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:15.284647 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28a0b862-cb3b-4f82-990e-049381077d85-istio-data\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-6rs9r\" (UID: \"28a0b862-cb3b-4f82-990e-049381077d85\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-6rs9r" Oct 31 04:20:15.316050 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:15.315664 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28a0b862-cb3b-4f82-990e-049381077d85-istio-envoy\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-6rs9r\" (UID: \"28a0b862-cb3b-4f82-990e-049381077d85\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-6rs9r" Oct 31 04:20:15.318995 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:15.318875 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28a0b862-cb3b-4f82-990e-049381077d85-istio-podinfo\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-6rs9r\" (UID: \"28a0b862-cb3b-4f82-990e-049381077d85\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-6rs9r" Oct 31 04:20:15.324963 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:15.324923 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28a0b862-cb3b-4f82-990e-049381077d85-istio-token\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-6rs9r\" (UID: \"28a0b862-cb3b-4f82-990e-049381077d85\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-6rs9r" Oct 31 04:20:15.325163 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:15.325050 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxkjn\" (UniqueName: \"kubernetes.io/projected/28a0b862-cb3b-4f82-990e-049381077d85-kube-api-access-pxkjn\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-6rs9r\" (UID: \"28a0b862-cb3b-4f82-990e-049381077d85\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-6rs9r" Oct 31 04:20:15.526347 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:15.526245 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-6rs9r" Oct 31 04:20:15.869621 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:20:15.869593 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28a0b862_cb3b_4f82_990e_049381077d85.slice/crio-6d29eb77b394818b09e208f416d6b78193010755e7d4fcc5b3d3fad89733c585 WatchSource:0}: Error finding container 6d29eb77b394818b09e208f416d6b78193010755e7d4fcc5b3d3fad89733c585: Status 404 returned error can't find the container with id 6d29eb77b394818b09e208f416d6b78193010755e7d4fcc5b3d3fad89733c585 Oct 31 04:20:15.870883 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:15.870856 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-6rs9r"] Oct 31 04:20:15.873044 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:15.873003 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:20:15.873165 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:15.873095 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:20:15.873165 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:15.873140 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:20:16.005821 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:16.005785 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-6rs9r" event={"ID":"28a0b862-cb3b-4f82-990e-049381077d85","Type":"ContainerStarted","Data":"cca1910d69469194747b37b4b6fc5083ef9615228e12581e276f44c5a7d0363d"} Oct 31 04:20:16.006000 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:16.005896 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-6rs9r" event={"ID":"28a0b862-cb3b-4f82-990e-049381077d85","Type":"ContainerStarted","Data":"6d29eb77b394818b09e208f416d6b78193010755e7d4fcc5b3d3fad89733c585"} Oct 31 04:20:16.056117 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:16.056062 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-6rs9r" podStartSLOduration=3.05604267 podStartE2EDuration="3.05604267s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:20:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:20:16.054181043 +0000 UTC m=+963.768515658" watchObservedRunningTime="2026-10-31 04:20:16.05604267 +0000 UTC m=+963.770377287" Oct 31 04:20:16.527495 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:16.527454 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-6rs9r" Oct 31 04:20:16.529214 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:16.529182 4376 patch_prober.go:29] interesting pod/e2e-worker-w0-965f61-openshift-default-55854964c-6rs9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 31 04:20:16.529393 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:16.529242 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-6rs9r" podUID="28a0b862-cb3b-4f82-990e-049381077d85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 31 04:20:17.427089 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:17.427056 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86b5bede-5sf9t"] Oct 31 04:20:17.431294 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:17.431270 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86b5bede-5sf9t" Oct 31 04:20:17.475676 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:17.475638 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86b5bede-5sf9t"] Oct 31 04:20:17.479547 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:17.479520 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzb8r\"" Oct 31 04:20:17.479812 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:17.479795 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 04:20:17.510839 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:17.510811 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7af9e9fd-7429-41b0-a38e-637c94a20fe4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-86b5bede-5sf9t\" (UID: \"7af9e9fd-7429-41b0-a38e-637c94a20fe4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86b5bede-5sf9t" Oct 31 04:20:17.511004 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:17.510979 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7tnbb\" (UniqueName: \"kubernetes.io/projected/7af9e9fd-7429-41b0-a38e-637c94a20fe4-kube-api-access-7tnbb\") pod \"maas-api-revoke-sub-86b5bede-5sf9t\" (UID: \"7af9e9fd-7429-41b0-a38e-637c94a20fe4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86b5bede-5sf9t" Oct 31 04:20:17.511216 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:17.511196 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7af9e9fd-7429-41b0-a38e-637c94a20fe4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-86b5bede-5sf9t\" (UID: \"7af9e9fd-7429-41b0-a38e-637c94a20fe4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86b5bede-5sf9t" Oct 31 04:20:17.527077 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:17.527053 4376 patch_prober.go:29] interesting pod/e2e-worker-w0-965f61-openshift-default-55854964c-6rs9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 31 04:20:17.527187 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:17.527098 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-6rs9r" podUID="28a0b862-cb3b-4f82-990e-049381077d85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 31 04:20:17.612467 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:17.612438 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7af9e9fd-7429-41b0-a38e-637c94a20fe4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-86b5bede-5sf9t\" (UID: \"7af9e9fd-7429-41b0-a38e-637c94a20fe4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86b5bede-5sf9t" Oct 31 04:20:17.612955 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:17.612525 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tnbb\" (UniqueName: \"kubernetes.io/projected/7af9e9fd-7429-41b0-a38e-637c94a20fe4-kube-api-access-7tnbb\") pod \"maas-api-revoke-sub-86b5bede-5sf9t\" (UID: \"7af9e9fd-7429-41b0-a38e-637c94a20fe4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86b5bede-5sf9t" Oct 31 04:20:17.612955 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:17.612637 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7af9e9fd-7429-41b0-a38e-637c94a20fe4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-86b5bede-5sf9t\" (UID: \"7af9e9fd-7429-41b0-a38e-637c94a20fe4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86b5bede-5sf9t" Oct 31 04:20:17.613251 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:17.613226 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7af9e9fd-7429-41b0-a38e-637c94a20fe4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-86b5bede-5sf9t\" (UID: \"7af9e9fd-7429-41b0-a38e-637c94a20fe4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86b5bede-5sf9t" Oct 31 04:20:17.614929 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:17.614908 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7af9e9fd-7429-41b0-a38e-637c94a20fe4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-86b5bede-5sf9t\" (UID: \"7af9e9fd-7429-41b0-a38e-637c94a20fe4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86b5bede-5sf9t" Oct 31 04:20:17.661349 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:17.637800 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tnbb\" (UniqueName: \"kubernetes.io/projected/7af9e9fd-7429-41b0-a38e-637c94a20fe4-kube-api-access-7tnbb\") pod \"maas-api-revoke-sub-86b5bede-5sf9t\" (UID: \"7af9e9fd-7429-41b0-a38e-637c94a20fe4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86b5bede-5sf9t" Oct 31 04:20:17.787911 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:17.787822 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86b5bede-5sf9t" Oct 31 04:20:18.105182 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:20:18.105143 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7af9e9fd_7429_41b0_a38e_637c94a20fe4.slice/crio-3be24a429760e599a735ccd0f44f80226600fff460d8f9bdeec744b16514f9f3 WatchSource:0}: Error finding container 3be24a429760e599a735ccd0f44f80226600fff460d8f9bdeec744b16514f9f3: Status 404 returned error can't find the container with id 3be24a429760e599a735ccd0f44f80226600fff460d8f9bdeec744b16514f9f3 Oct 31 04:20:18.106396 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:18.106370 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86b5bede-5sf9t"] Oct 31 04:20:18.539357 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:18.536408 4376 patch_prober.go:29] interesting pod/e2e-worker-w0-965f61-openshift-default-55854964c-6rs9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 31 04:20:18.539357 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:18.536468 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-6rs9r" podUID="28a0b862-cb3b-4f82-990e-049381077d85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 31 04:20:19.034066 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:19.034032 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86b5bede-5sf9t" event={"ID":"7af9e9fd-7429-41b0-a38e-637c94a20fe4","Type":"ContainerStarted","Data":"7fd115bca1878e75a1e201abd476d9f7cf3b445b90b7cecb9d8c841751a56d26"} Oct 31 04:20:19.034477 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:19.034083 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86b5bede-5sf9t" event={"ID":"7af9e9fd-7429-41b0-a38e-637c94a20fe4","Type":"ContainerStarted","Data":"3be24a429760e599a735ccd0f44f80226600fff460d8f9bdeec744b16514f9f3"} Oct 31 04:20:19.080438 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:19.080385 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86b5bede-5sf9t" podStartSLOduration=2.080372755 podStartE2EDuration="2.080372755s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:20:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:20:19.079509059 +0000 UTC m=+966.793843675" watchObservedRunningTime="2026-10-31 04:20:19.080372755 +0000 UTC m=+966.794707371" Oct 31 04:20:19.533495 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:19.532799 4376 patch_prober.go:29] interesting pod/e2e-worker-w0-965f61-openshift-default-55854964c-6rs9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 31 04:20:19.533495 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:19.532945 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-6rs9r" podUID="28a0b862-cb3b-4f82-990e-049381077d85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 31 04:20:20.066349 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:20.065283 4376 generic.go:356] "Generic (PLEG): container finished" podID="7af9e9fd-7429-41b0-a38e-637c94a20fe4" containerID="7fd115bca1878e75a1e201abd476d9f7cf3b445b90b7cecb9d8c841751a56d26" exitCode=0 Oct 31 04:20:20.066349 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:20.065404 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86b5bede-5sf9t" event={"ID":"7af9e9fd-7429-41b0-a38e-637c94a20fe4","Type":"ContainerDied","Data":"7fd115bca1878e75a1e201abd476d9f7cf3b445b90b7cecb9d8c841751a56d26"} Oct 31 04:20:20.527615 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:20.527582 4376 patch_prober.go:29] interesting pod/e2e-worker-w0-965f61-openshift-default-55854964c-6rs9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 31 04:20:20.527778 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:20.527630 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-6rs9r" podUID="28a0b862-cb3b-4f82-990e-049381077d85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 31 04:20:21.275620 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:21.275582 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-scmn8"] Oct 31 04:20:21.283498 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:21.283468 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-scmn8"] Oct 31 04:20:21.283646 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:21.283559 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-scmn8" Oct 31 04:20:21.285235 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:21.285209 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86b5bede-5sf9t" Oct 31 04:20:21.317394 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:21.317372 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f95ddb3e-3a81-433a-aead-9a220d1ae61b-workload-certs\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-scmn8\" (UID: \"f95ddb3e-3a81-433a-aead-9a220d1ae61b\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-scmn8" Oct 31 04:20:21.317503 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:21.317410 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f95ddb3e-3a81-433a-aead-9a220d1ae61b-istio-podinfo\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-scmn8\" (UID: \"f95ddb3e-3a81-433a-aead-9a220d1ae61b\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-scmn8" Oct 31 04:20:21.317559 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:21.317506 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxlcr\" (UniqueName: \"kubernetes.io/projected/f95ddb3e-3a81-433a-aead-9a220d1ae61b-kube-api-access-lxlcr\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-scmn8\" (UID: \"f95ddb3e-3a81-433a-aead-9a220d1ae61b\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-scmn8" Oct 31 04:20:21.317559 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:21.317523 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f95ddb3e-3a81-433a-aead-9a220d1ae61b-istio-data\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-scmn8\" (UID: \"f95ddb3e-3a81-433a-aead-9a220d1ae61b\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-scmn8" Oct 31 04:20:21.317559 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:21.317542 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f95ddb3e-3a81-433a-aead-9a220d1ae61b-istio-envoy\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-scmn8\" (UID: \"f95ddb3e-3a81-433a-aead-9a220d1ae61b\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-scmn8" Oct 31 04:20:21.317666 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:21.317584 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f95ddb3e-3a81-433a-aead-9a220d1ae61b-credential-socket\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-scmn8\" (UID: \"f95ddb3e-3a81-433a-aead-9a220d1ae61b\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-scmn8" Oct 31 04:20:21.317666 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:21.317626 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f95ddb3e-3a81-433a-aead-9a220d1ae61b-istiod-ca-cert\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-scmn8\" (UID: \"f95ddb3e-3a81-433a-aead-9a220d1ae61b\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-scmn8" Oct 31 04:20:21.317736 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:21.317682 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f95ddb3e-3a81-433a-aead-9a220d1ae61b-workload-socket\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-scmn8\" (UID: \"f95ddb3e-3a81-433a-aead-9a220d1ae61b\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-scmn8" Oct 31 04:20:21.317736 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:21.317702 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f95ddb3e-3a81-433a-aead-9a220d1ae61b-istio-token\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-scmn8\" (UID: \"f95ddb3e-3a81-433a-aead-9a220d1ae61b\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-scmn8" Oct 31 04:20:21.418723 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:21.418630 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7af9e9fd-7429-41b0-a38e-637c94a20fe4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7af9e9fd-7429-41b0-a38e-637c94a20fe4-maas-api-openshift-ca\") pod \"7af9e9fd-7429-41b0-a38e-637c94a20fe4\" (UID: \"7af9e9fd-7429-41b0-a38e-637c94a20fe4\") " Oct 31 04:20:21.419025 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:21.419003 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7af9e9fd-7429-41b0-a38e-637c94a20fe4-kube-api-access-7tnbb\" (UniqueName: \"kubernetes.io/projected/7af9e9fd-7429-41b0-a38e-637c94a20fe4-kube-api-access-7tnbb\") pod \"7af9e9fd-7429-41b0-a38e-637c94a20fe4\" (UID: \"7af9e9fd-7429-41b0-a38e-637c94a20fe4\") " Oct 31 04:20:21.419204 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:21.419186 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7af9e9fd-7429-41b0-a38e-637c94a20fe4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7af9e9fd-7429-41b0-a38e-637c94a20fe4-maas-api-xks-ca\") pod \"7af9e9fd-7429-41b0-a38e-637c94a20fe4\" (UID: \"7af9e9fd-7429-41b0-a38e-637c94a20fe4\") " Oct 31 04:20:21.419656 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:21.419065 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7af9e9fd-7429-41b0-a38e-637c94a20fe4-maas-api-openshift-ca" pod "7af9e9fd-7429-41b0-a38e-637c94a20fe4" (UID: "7af9e9fd-7429-41b0-a38e-637c94a20fe4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:20:21.419776 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:21.419732 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxlcr\" (UniqueName: \"kubernetes.io/projected/f95ddb3e-3a81-433a-aead-9a220d1ae61b-kube-api-access-lxlcr\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-scmn8\" (UID: \"f95ddb3e-3a81-433a-aead-9a220d1ae61b\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-scmn8" Oct 31 04:20:21.419847 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:21.419799 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f95ddb3e-3a81-433a-aead-9a220d1ae61b-istio-data\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-scmn8\" (UID: \"f95ddb3e-3a81-433a-aead-9a220d1ae61b\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-scmn8" Oct 31 04:20:21.419847 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:21.419830 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f95ddb3e-3a81-433a-aead-9a220d1ae61b-istio-envoy\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-scmn8\" (UID: \"f95ddb3e-3a81-433a-aead-9a220d1ae61b\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-scmn8" Oct 31 04:20:21.419962 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:21.419880 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f95ddb3e-3a81-433a-aead-9a220d1ae61b-credential-socket\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-scmn8\" (UID: \"f95ddb3e-3a81-433a-aead-9a220d1ae61b\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-scmn8" Oct 31 04:20:21.419962 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:21.419926 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f95ddb3e-3a81-433a-aead-9a220d1ae61b-istiod-ca-cert\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-scmn8\" (UID: \"f95ddb3e-3a81-433a-aead-9a220d1ae61b\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-scmn8" Oct 31 04:20:21.420069 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:21.419980 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f95ddb3e-3a81-433a-aead-9a220d1ae61b-workload-socket\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-scmn8\" (UID: \"f95ddb3e-3a81-433a-aead-9a220d1ae61b\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-scmn8" Oct 31 04:20:21.420069 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:21.420006 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f95ddb3e-3a81-433a-aead-9a220d1ae61b-istio-token\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-scmn8\" (UID: \"f95ddb3e-3a81-433a-aead-9a220d1ae61b\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-scmn8" Oct 31 04:20:21.420199 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:21.420091 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f95ddb3e-3a81-433a-aead-9a220d1ae61b-workload-certs\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-scmn8\" (UID: \"f95ddb3e-3a81-433a-aead-9a220d1ae61b\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-scmn8" Oct 31 04:20:21.420199 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:21.420134 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f95ddb3e-3a81-433a-aead-9a220d1ae61b-istio-podinfo\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-scmn8\" (UID: \"f95ddb3e-3a81-433a-aead-9a220d1ae61b\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-scmn8" Oct 31 04:20:21.420725 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:21.420407 4376 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7af9e9fd-7429-41b0-a38e-637c94a20fe4-maas-api-openshift-ca\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:20:21.420725 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:21.420510 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f95ddb3e-3a81-433a-aead-9a220d1ae61b-credential-socket\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-scmn8\" (UID: \"f95ddb3e-3a81-433a-aead-9a220d1ae61b\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-scmn8" Oct 31 04:20:21.420896 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:21.420830 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f95ddb3e-3a81-433a-aead-9a220d1ae61b-istio-data\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-scmn8\" (UID: \"f95ddb3e-3a81-433a-aead-9a220d1ae61b\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-scmn8" Oct 31 04:20:21.421036 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:21.420639 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f95ddb3e-3a81-433a-aead-9a220d1ae61b-workload-socket\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-scmn8\" (UID: \"f95ddb3e-3a81-433a-aead-9a220d1ae61b\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-scmn8" Oct 31 04:20:21.421129 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:21.421066 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f95ddb3e-3a81-433a-aead-9a220d1ae61b-workload-certs\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-scmn8\" (UID: \"f95ddb3e-3a81-433a-aead-9a220d1ae61b\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-scmn8" Oct 31 04:20:21.421786 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:21.421751 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7af9e9fd-7429-41b0-a38e-637c94a20fe4-maas-api-xks-ca" pod "7af9e9fd-7429-41b0-a38e-637c94a20fe4" (UID: "7af9e9fd-7429-41b0-a38e-637c94a20fe4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:20:21.421889 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:21.421791 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7af9e9fd-7429-41b0-a38e-637c94a20fe4-kube-api-access-7tnbb" pod "7af9e9fd-7429-41b0-a38e-637c94a20fe4" (UID: "7af9e9fd-7429-41b0-a38e-637c94a20fe4"). InnerVolumeSpecName "kube-api-access-7tnbb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:20:21.421889 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:21.421837 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f95ddb3e-3a81-433a-aead-9a220d1ae61b-istiod-ca-cert\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-scmn8\" (UID: \"f95ddb3e-3a81-433a-aead-9a220d1ae61b\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-scmn8" Oct 31 04:20:21.422971 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:21.422944 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f95ddb3e-3a81-433a-aead-9a220d1ae61b-istio-envoy\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-scmn8\" (UID: \"f95ddb3e-3a81-433a-aead-9a220d1ae61b\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-scmn8" Oct 31 04:20:21.423518 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:21.423498 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f95ddb3e-3a81-433a-aead-9a220d1ae61b-istio-podinfo\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-scmn8\" (UID: \"f95ddb3e-3a81-433a-aead-9a220d1ae61b\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-scmn8" Oct 31 04:20:21.468447 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:21.468408 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxlcr\" (UniqueName: \"kubernetes.io/projected/f95ddb3e-3a81-433a-aead-9a220d1ae61b-kube-api-access-lxlcr\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-scmn8\" (UID: \"f95ddb3e-3a81-433a-aead-9a220d1ae61b\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-scmn8" Oct 31 04:20:21.469785 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:21.469767 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f95ddb3e-3a81-433a-aead-9a220d1ae61b-istio-token\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-scmn8\" (UID: \"f95ddb3e-3a81-433a-aead-9a220d1ae61b\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-scmn8" Oct 31 04:20:21.522470 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:21.522448 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7tnbb\" (UniqueName: \"kubernetes.io/projected/7af9e9fd-7429-41b0-a38e-637c94a20fe4-kube-api-access-7tnbb\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:20:21.522470 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:21.522470 4376 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7af9e9fd-7429-41b0-a38e-637c94a20fe4-maas-api-xks-ca\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:20:21.527569 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:21.527538 4376 patch_prober.go:29] interesting pod/e2e-worker-w0-965f61-openshift-default-55854964c-6rs9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 31 04:20:21.527666 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:21.527576 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-6rs9r" podUID="28a0b862-cb3b-4f82-990e-049381077d85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 31 04:20:21.598160 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:21.598134 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-scmn8" Oct 31 04:20:21.964421 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:20:21.963965 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf95ddb3e_3a81_433a_aead_9a220d1ae61b.slice/crio-fccc9b34eb8ff7e03c26aadbd1da0cc62c5060540a9de0f6bba436048098848b WatchSource:0}: Error finding container fccc9b34eb8ff7e03c26aadbd1da0cc62c5060540a9de0f6bba436048098848b: Status 404 returned error can't find the container with id fccc9b34eb8ff7e03c26aadbd1da0cc62c5060540a9de0f6bba436048098848b Oct 31 04:20:21.965862 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:21.965839 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-scmn8"] Oct 31 04:20:21.967406 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:21.967366 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:20:21.967511 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:21.967459 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:20:21.967511 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:21.967503 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:20:22.081407 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:22.081378 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86b5bede-5sf9t" Oct 31 04:20:22.081742 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:22.081379 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86b5bede-5sf9t" event={"ID":"7af9e9fd-7429-41b0-a38e-637c94a20fe4","Type":"ContainerDied","Data":"3be24a429760e599a735ccd0f44f80226600fff460d8f9bdeec744b16514f9f3"} Oct 31 04:20:22.081836 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:22.081782 4376 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3be24a429760e599a735ccd0f44f80226600fff460d8f9bdeec744b16514f9f3" Oct 31 04:20:22.083383 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:22.083323 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-scmn8" event={"ID":"f95ddb3e-3a81-433a-aead-9a220d1ae61b","Type":"ContainerStarted","Data":"0ec9f3db650e8b422b9517c0fd438d67096a750bc3a1ab2ace78cc32cb43788c"} Oct 31 04:20:22.083535 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:22.083398 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-scmn8" event={"ID":"f95ddb3e-3a81-433a-aead-9a220d1ae61b","Type":"ContainerStarted","Data":"fccc9b34eb8ff7e03c26aadbd1da0cc62c5060540a9de0f6bba436048098848b"} Oct 31 04:20:22.133968 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:22.133920 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-scmn8" podStartSLOduration=1.1339074980000001 podStartE2EDuration="1.133907498s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:20:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:20:22.133579953 +0000 UTC m=+969.847914593" watchObservedRunningTime="2026-10-31 04:20:22.133907498 +0000 UTC m=+969.848242115" Oct 31 04:20:22.557349 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:22.556833 4376 patch_prober.go:29] interesting pod/e2e-worker-w0-965f61-openshift-default-55854964c-6rs9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 31 04:20:22.557349 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:22.556888 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-6rs9r" podUID="28a0b862-cb3b-4f82-990e-049381077d85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 31 04:20:22.599678 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:22.599596 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-scmn8" Oct 31 04:20:23.600036 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:23.599997 4376 patch_prober.go:29] interesting pod/e2e-worker-w2-09a20a-openshift-default-55d8647ff-scmn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Oct 31 04:20:23.600481 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:23.600059 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-scmn8" podUID="f95ddb3e-3a81-433a-aead-9a220d1ae61b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Oct 31 04:20:23.631861 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:23.631824 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-6rs9r" Oct 31 04:20:23.632265 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:23.632240 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-6rs9r" Oct 31 04:20:23.633271 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:23.633241 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-6rs9r" Oct 31 04:20:23.734192 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:23.734161 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-cjmdq"] Oct 31 04:20:23.735127 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:23.735104 4376 cpu_manager.go:414] "containerMap: removing container" podUID="7af9e9fd-7429-41b0-a38e-637c94a20fe4" containerName="revoke-keys" Oct 31 04:20:23.735211 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:23.735129 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7af9e9fd-7429-41b0-a38e-637c94a20fe4" containerName="revoke-keys" Oct 31 04:20:23.735286 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:23.735274 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="7af9e9fd-7429-41b0-a38e-637c94a20fe4" containerName="revoke-keys" Oct 31 04:20:23.744357 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:23.739608 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-cjmdq" Oct 31 04:20:23.811839 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:23.811778 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-cjmdq"] Oct 31 04:20:23.883508 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:23.883465 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6213a9f9-302f-4896-a1ce-dc0c7323082c-istio-data\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-cjmdq\" (UID: \"6213a9f9-302f-4896-a1ce-dc0c7323082c\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-cjmdq" Oct 31 04:20:23.883689 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:23.883567 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6213a9f9-302f-4896-a1ce-dc0c7323082c-istio-token\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-cjmdq\" (UID: \"6213a9f9-302f-4896-a1ce-dc0c7323082c\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-cjmdq" Oct 31 04:20:23.883689 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:23.883639 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6213a9f9-302f-4896-a1ce-dc0c7323082c-workload-certs\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-cjmdq\" (UID: \"6213a9f9-302f-4896-a1ce-dc0c7323082c\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-cjmdq" Oct 31 04:20:23.883689 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:23.883686 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6213a9f9-302f-4896-a1ce-dc0c7323082c-istio-envoy\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-cjmdq\" (UID: \"6213a9f9-302f-4896-a1ce-dc0c7323082c\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-cjmdq" Oct 31 04:20:23.883867 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:23.883724 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6213a9f9-302f-4896-a1ce-dc0c7323082c-istiod-ca-cert\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-cjmdq\" (UID: \"6213a9f9-302f-4896-a1ce-dc0c7323082c\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-cjmdq" Oct 31 04:20:23.883867 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:23.883838 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6213a9f9-302f-4896-a1ce-dc0c7323082c-istio-podinfo\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-cjmdq\" (UID: \"6213a9f9-302f-4896-a1ce-dc0c7323082c\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-cjmdq" Oct 31 04:20:23.884080 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:23.884058 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6213a9f9-302f-4896-a1ce-dc0c7323082c-credential-socket\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-cjmdq\" (UID: \"6213a9f9-302f-4896-a1ce-dc0c7323082c\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-cjmdq" Oct 31 04:20:23.884166 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:23.884149 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7pq2\" (UniqueName: \"kubernetes.io/projected/6213a9f9-302f-4896-a1ce-dc0c7323082c-kube-api-access-n7pq2\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-cjmdq\" (UID: \"6213a9f9-302f-4896-a1ce-dc0c7323082c\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-cjmdq" Oct 31 04:20:23.884226 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:23.884208 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6213a9f9-302f-4896-a1ce-dc0c7323082c-workload-socket\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-cjmdq\" (UID: \"6213a9f9-302f-4896-a1ce-dc0c7323082c\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-cjmdq" Oct 31 04:20:23.985522 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:23.985494 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6213a9f9-302f-4896-a1ce-dc0c7323082c-istio-data\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-cjmdq\" (UID: \"6213a9f9-302f-4896-a1ce-dc0c7323082c\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-cjmdq" Oct 31 04:20:23.985687 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:23.985541 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6213a9f9-302f-4896-a1ce-dc0c7323082c-istio-token\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-cjmdq\" (UID: \"6213a9f9-302f-4896-a1ce-dc0c7323082c\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-cjmdq" Oct 31 04:20:23.985687 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:23.985573 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6213a9f9-302f-4896-a1ce-dc0c7323082c-workload-certs\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-cjmdq\" (UID: \"6213a9f9-302f-4896-a1ce-dc0c7323082c\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-cjmdq" Oct 31 04:20:23.985687 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:23.985588 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6213a9f9-302f-4896-a1ce-dc0c7323082c-istio-envoy\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-cjmdq\" (UID: \"6213a9f9-302f-4896-a1ce-dc0c7323082c\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-cjmdq" Oct 31 04:20:23.985687 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:23.985608 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6213a9f9-302f-4896-a1ce-dc0c7323082c-istiod-ca-cert\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-cjmdq\" (UID: \"6213a9f9-302f-4896-a1ce-dc0c7323082c\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-cjmdq" Oct 31 04:20:23.985687 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:23.985656 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6213a9f9-302f-4896-a1ce-dc0c7323082c-istio-podinfo\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-cjmdq\" (UID: \"6213a9f9-302f-4896-a1ce-dc0c7323082c\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-cjmdq" Oct 31 04:20:23.985867 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:23.985777 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6213a9f9-302f-4896-a1ce-dc0c7323082c-credential-socket\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-cjmdq\" (UID: \"6213a9f9-302f-4896-a1ce-dc0c7323082c\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-cjmdq" Oct 31 04:20:23.985867 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:23.985828 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7pq2\" (UniqueName: \"kubernetes.io/projected/6213a9f9-302f-4896-a1ce-dc0c7323082c-kube-api-access-n7pq2\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-cjmdq\" (UID: \"6213a9f9-302f-4896-a1ce-dc0c7323082c\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-cjmdq" Oct 31 04:20:23.985867 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:23.985849 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6213a9f9-302f-4896-a1ce-dc0c7323082c-workload-socket\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-cjmdq\" (UID: \"6213a9f9-302f-4896-a1ce-dc0c7323082c\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-cjmdq" Oct 31 04:20:23.985973 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:23.985952 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6213a9f9-302f-4896-a1ce-dc0c7323082c-istio-data\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-cjmdq\" (UID: \"6213a9f9-302f-4896-a1ce-dc0c7323082c\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-cjmdq" Oct 31 04:20:23.985973 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:23.985964 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6213a9f9-302f-4896-a1ce-dc0c7323082c-workload-certs\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-cjmdq\" (UID: \"6213a9f9-302f-4896-a1ce-dc0c7323082c\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-cjmdq" Oct 31 04:20:23.986137 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:23.986112 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6213a9f9-302f-4896-a1ce-dc0c7323082c-workload-socket\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-cjmdq\" (UID: \"6213a9f9-302f-4896-a1ce-dc0c7323082c\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-cjmdq" Oct 31 04:20:23.986841 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:23.986814 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6213a9f9-302f-4896-a1ce-dc0c7323082c-istiod-ca-cert\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-cjmdq\" (UID: \"6213a9f9-302f-4896-a1ce-dc0c7323082c\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-cjmdq" Oct 31 04:20:23.986992 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:23.986973 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6213a9f9-302f-4896-a1ce-dc0c7323082c-credential-socket\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-cjmdq\" (UID: \"6213a9f9-302f-4896-a1ce-dc0c7323082c\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-cjmdq" Oct 31 04:20:23.988205 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:23.988176 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6213a9f9-302f-4896-a1ce-dc0c7323082c-istio-envoy\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-cjmdq\" (UID: \"6213a9f9-302f-4896-a1ce-dc0c7323082c\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-cjmdq" Oct 31 04:20:23.988476 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:23.988458 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6213a9f9-302f-4896-a1ce-dc0c7323082c-istio-podinfo\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-cjmdq\" (UID: \"6213a9f9-302f-4896-a1ce-dc0c7323082c\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-cjmdq" Oct 31 04:20:24.011012 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:24.010988 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6213a9f9-302f-4896-a1ce-dc0c7323082c-istio-token\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-cjmdq\" (UID: \"6213a9f9-302f-4896-a1ce-dc0c7323082c\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-cjmdq" Oct 31 04:20:24.011250 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:24.011231 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7pq2\" (UniqueName: \"kubernetes.io/projected/6213a9f9-302f-4896-a1ce-dc0c7323082c-kube-api-access-n7pq2\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-cjmdq\" (UID: \"6213a9f9-302f-4896-a1ce-dc0c7323082c\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-cjmdq" Oct 31 04:20:24.118184 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:24.118153 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-cjmdq" Oct 31 04:20:24.468022 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:20:24.467994 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6213a9f9_302f_4896_a1ce_dc0c7323082c.slice/crio-87e82dd28b2a85b773db740984da597bd9ddb9ac7cbdc2bf75078d00aadc9e66 WatchSource:0}: Error finding container 87e82dd28b2a85b773db740984da597bd9ddb9ac7cbdc2bf75078d00aadc9e66: Status 404 returned error can't find the container with id 87e82dd28b2a85b773db740984da597bd9ddb9ac7cbdc2bf75078d00aadc9e66 Oct 31 04:20:24.468323 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:24.468305 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-cjmdq"] Oct 31 04:20:24.470579 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:24.470547 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:20:24.470657 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:24.470614 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:20:24.470657 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:24.470644 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:20:24.609547 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:24.604166 4376 patch_prober.go:29] interesting pod/e2e-worker-w2-09a20a-openshift-default-55d8647ff-scmn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Oct 31 04:20:24.609547 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:24.604222 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-scmn8" podUID="f95ddb3e-3a81-433a-aead-9a220d1ae61b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Oct 31 04:20:25.118388 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:25.116490 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-cjmdq" event={"ID":"6213a9f9-302f-4896-a1ce-dc0c7323082c","Type":"ContainerStarted","Data":"76596c78a36ea8bc117ade55c61089b261bb2d8c0405ac33a1c0324f5787eccb"} Oct 31 04:20:25.118388 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:25.116551 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-cjmdq" event={"ID":"6213a9f9-302f-4896-a1ce-dc0c7323082c","Type":"ContainerStarted","Data":"87e82dd28b2a85b773db740984da597bd9ddb9ac7cbdc2bf75078d00aadc9e66"} Oct 31 04:20:25.118921 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:25.118896 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-cjmdq" Oct 31 04:20:25.120578 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:25.120552 4376 patch_prober.go:29] interesting pod/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-cjmdq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Oct 31 04:20:25.120683 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:25.120604 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-cjmdq" podUID="6213a9f9-302f-4896-a1ce-dc0c7323082c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Oct 31 04:20:25.169605 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:25.169533 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-cjmdq" podStartSLOduration=2.169513436 podStartE2EDuration="2.169513436s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:20:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:20:25.168887439 +0000 UTC m=+972.883222067" watchObservedRunningTime="2026-10-31 04:20:25.169513436 +0000 UTC m=+972.883848054" Oct 31 04:20:25.599272 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:25.599185 4376 patch_prober.go:29] interesting pod/e2e-worker-w2-09a20a-openshift-default-55d8647ff-scmn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Oct 31 04:20:25.599577 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:25.599543 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-scmn8" podUID="f95ddb3e-3a81-433a-aead-9a220d1ae61b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Oct 31 04:20:26.030373 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:26.029522 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-8c0086-openshift-default-58b446d58c-cdgtp"] Oct 31 04:20:26.035058 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:26.035030 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-8c0086-openshift-default-58b446d58c-cdgtp" Oct 31 04:20:26.071350 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:26.070790 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-8c0086-openshift-default-dockercfg-hl2fb\"" Oct 31 04:20:26.106046 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:26.105436 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-8c0086-openshift-default-58b446d58c-cdgtp"] Oct 31 04:20:26.119899 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:26.119866 4376 patch_prober.go:29] interesting pod/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-cjmdq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Oct 31 04:20:26.120081 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:26.119956 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-cjmdq" podUID="6213a9f9-302f-4896-a1ce-dc0c7323082c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Oct 31 04:20:26.130406 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:26.130380 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ab2d121-68d1-4cc1-9f0a-96755f3782f4-workload-certs\") pod \"e2e-api-a-8c0086-openshift-default-58b446d58c-cdgtp\" (UID: \"5ab2d121-68d1-4cc1-9f0a-96755f3782f4\") " pod="openshift-ingress/e2e-api-a-8c0086-openshift-default-58b446d58c-cdgtp" Oct 31 04:20:26.130630 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:26.130611 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ab2d121-68d1-4cc1-9f0a-96755f3782f4-istio-envoy\") pod \"e2e-api-a-8c0086-openshift-default-58b446d58c-cdgtp\" (UID: \"5ab2d121-68d1-4cc1-9f0a-96755f3782f4\") " pod="openshift-ingress/e2e-api-a-8c0086-openshift-default-58b446d58c-cdgtp" Oct 31 04:20:26.130728 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:26.130717 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ab2d121-68d1-4cc1-9f0a-96755f3782f4-istio-podinfo\") pod \"e2e-api-a-8c0086-openshift-default-58b446d58c-cdgtp\" (UID: \"5ab2d121-68d1-4cc1-9f0a-96755f3782f4\") " pod="openshift-ingress/e2e-api-a-8c0086-openshift-default-58b446d58c-cdgtp" Oct 31 04:20:26.130949 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:26.130923 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rvtx\" (UniqueName: \"kubernetes.io/projected/5ab2d121-68d1-4cc1-9f0a-96755f3782f4-kube-api-access-7rvtx\") pod \"e2e-api-a-8c0086-openshift-default-58b446d58c-cdgtp\" (UID: \"5ab2d121-68d1-4cc1-9f0a-96755f3782f4\") " pod="openshift-ingress/e2e-api-a-8c0086-openshift-default-58b446d58c-cdgtp" Oct 31 04:20:26.131077 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:26.131058 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ab2d121-68d1-4cc1-9f0a-96755f3782f4-credential-socket\") pod \"e2e-api-a-8c0086-openshift-default-58b446d58c-cdgtp\" (UID: \"5ab2d121-68d1-4cc1-9f0a-96755f3782f4\") " pod="openshift-ingress/e2e-api-a-8c0086-openshift-default-58b446d58c-cdgtp" Oct 31 04:20:26.131138 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:26.131104 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ab2d121-68d1-4cc1-9f0a-96755f3782f4-istio-token\") pod \"e2e-api-a-8c0086-openshift-default-58b446d58c-cdgtp\" (UID: \"5ab2d121-68d1-4cc1-9f0a-96755f3782f4\") " pod="openshift-ingress/e2e-api-a-8c0086-openshift-default-58b446d58c-cdgtp" Oct 31 04:20:26.131213 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:26.131195 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ab2d121-68d1-4cc1-9f0a-96755f3782f4-workload-socket\") pod \"e2e-api-a-8c0086-openshift-default-58b446d58c-cdgtp\" (UID: \"5ab2d121-68d1-4cc1-9f0a-96755f3782f4\") " pod="openshift-ingress/e2e-api-a-8c0086-openshift-default-58b446d58c-cdgtp" Oct 31 04:20:26.131277 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:26.131231 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ab2d121-68d1-4cc1-9f0a-96755f3782f4-istio-data\") pod \"e2e-api-a-8c0086-openshift-default-58b446d58c-cdgtp\" (UID: \"5ab2d121-68d1-4cc1-9f0a-96755f3782f4\") " pod="openshift-ingress/e2e-api-a-8c0086-openshift-default-58b446d58c-cdgtp" Oct 31 04:20:26.131377 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:26.131359 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ab2d121-68d1-4cc1-9f0a-96755f3782f4-istiod-ca-cert\") pod \"e2e-api-a-8c0086-openshift-default-58b446d58c-cdgtp\" (UID: \"5ab2d121-68d1-4cc1-9f0a-96755f3782f4\") " pod="openshift-ingress/e2e-api-a-8c0086-openshift-default-58b446d58c-cdgtp" Oct 31 04:20:26.240359 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:26.239697 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rvtx\" (UniqueName: \"kubernetes.io/projected/5ab2d121-68d1-4cc1-9f0a-96755f3782f4-kube-api-access-7rvtx\") pod \"e2e-api-a-8c0086-openshift-default-58b446d58c-cdgtp\" (UID: \"5ab2d121-68d1-4cc1-9f0a-96755f3782f4\") " pod="openshift-ingress/e2e-api-a-8c0086-openshift-default-58b446d58c-cdgtp" Oct 31 04:20:26.244350 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:26.240615 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ab2d121-68d1-4cc1-9f0a-96755f3782f4-credential-socket\") pod \"e2e-api-a-8c0086-openshift-default-58b446d58c-cdgtp\" (UID: \"5ab2d121-68d1-4cc1-9f0a-96755f3782f4\") " pod="openshift-ingress/e2e-api-a-8c0086-openshift-default-58b446d58c-cdgtp" Oct 31 04:20:26.244350 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:26.240651 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ab2d121-68d1-4cc1-9f0a-96755f3782f4-istio-token\") pod \"e2e-api-a-8c0086-openshift-default-58b446d58c-cdgtp\" (UID: \"5ab2d121-68d1-4cc1-9f0a-96755f3782f4\") " pod="openshift-ingress/e2e-api-a-8c0086-openshift-default-58b446d58c-cdgtp" Oct 31 04:20:26.244350 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:26.240687 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ab2d121-68d1-4cc1-9f0a-96755f3782f4-workload-socket\") pod \"e2e-api-a-8c0086-openshift-default-58b446d58c-cdgtp\" (UID: \"5ab2d121-68d1-4cc1-9f0a-96755f3782f4\") " pod="openshift-ingress/e2e-api-a-8c0086-openshift-default-58b446d58c-cdgtp" Oct 31 04:20:26.244350 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:26.240745 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ab2d121-68d1-4cc1-9f0a-96755f3782f4-istio-data\") pod \"e2e-api-a-8c0086-openshift-default-58b446d58c-cdgtp\" (UID: \"5ab2d121-68d1-4cc1-9f0a-96755f3782f4\") " pod="openshift-ingress/e2e-api-a-8c0086-openshift-default-58b446d58c-cdgtp" Oct 31 04:20:26.244350 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:26.240853 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ab2d121-68d1-4cc1-9f0a-96755f3782f4-istiod-ca-cert\") pod \"e2e-api-a-8c0086-openshift-default-58b446d58c-cdgtp\" (UID: \"5ab2d121-68d1-4cc1-9f0a-96755f3782f4\") " pod="openshift-ingress/e2e-api-a-8c0086-openshift-default-58b446d58c-cdgtp" Oct 31 04:20:26.259344 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:26.251543 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ab2d121-68d1-4cc1-9f0a-96755f3782f4-istio-data\") pod \"e2e-api-a-8c0086-openshift-default-58b446d58c-cdgtp\" (UID: \"5ab2d121-68d1-4cc1-9f0a-96755f3782f4\") " pod="openshift-ingress/e2e-api-a-8c0086-openshift-default-58b446d58c-cdgtp" Oct 31 04:20:26.259344 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:26.252200 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ab2d121-68d1-4cc1-9f0a-96755f3782f4-istiod-ca-cert\") pod \"e2e-api-a-8c0086-openshift-default-58b446d58c-cdgtp\" (UID: \"5ab2d121-68d1-4cc1-9f0a-96755f3782f4\") " pod="openshift-ingress/e2e-api-a-8c0086-openshift-default-58b446d58c-cdgtp" Oct 31 04:20:26.259344 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:26.252476 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ab2d121-68d1-4cc1-9f0a-96755f3782f4-credential-socket\") pod \"e2e-api-a-8c0086-openshift-default-58b446d58c-cdgtp\" (UID: \"5ab2d121-68d1-4cc1-9f0a-96755f3782f4\") " pod="openshift-ingress/e2e-api-a-8c0086-openshift-default-58b446d58c-cdgtp" Oct 31 04:20:26.259344 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:26.257603 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ab2d121-68d1-4cc1-9f0a-96755f3782f4-workload-socket\") pod \"e2e-api-a-8c0086-openshift-default-58b446d58c-cdgtp\" (UID: \"5ab2d121-68d1-4cc1-9f0a-96755f3782f4\") " pod="openshift-ingress/e2e-api-a-8c0086-openshift-default-58b446d58c-cdgtp" Oct 31 04:20:26.259344 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:26.257887 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ab2d121-68d1-4cc1-9f0a-96755f3782f4-workload-certs\") pod \"e2e-api-a-8c0086-openshift-default-58b446d58c-cdgtp\" (UID: \"5ab2d121-68d1-4cc1-9f0a-96755f3782f4\") " pod="openshift-ingress/e2e-api-a-8c0086-openshift-default-58b446d58c-cdgtp" Oct 31 04:20:26.259344 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:26.258017 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ab2d121-68d1-4cc1-9f0a-96755f3782f4-istio-envoy\") pod \"e2e-api-a-8c0086-openshift-default-58b446d58c-cdgtp\" (UID: \"5ab2d121-68d1-4cc1-9f0a-96755f3782f4\") " pod="openshift-ingress/e2e-api-a-8c0086-openshift-default-58b446d58c-cdgtp" Oct 31 04:20:26.259344 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:26.258043 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ab2d121-68d1-4cc1-9f0a-96755f3782f4-istio-podinfo\") pod \"e2e-api-a-8c0086-openshift-default-58b446d58c-cdgtp\" (UID: \"5ab2d121-68d1-4cc1-9f0a-96755f3782f4\") " pod="openshift-ingress/e2e-api-a-8c0086-openshift-default-58b446d58c-cdgtp" Oct 31 04:20:26.280891 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:26.262636 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ab2d121-68d1-4cc1-9f0a-96755f3782f4-workload-certs\") pod \"e2e-api-a-8c0086-openshift-default-58b446d58c-cdgtp\" (UID: \"5ab2d121-68d1-4cc1-9f0a-96755f3782f4\") " pod="openshift-ingress/e2e-api-a-8c0086-openshift-default-58b446d58c-cdgtp" Oct 31 04:20:26.280891 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:26.272769 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ab2d121-68d1-4cc1-9f0a-96755f3782f4-istio-podinfo\") pod \"e2e-api-a-8c0086-openshift-default-58b446d58c-cdgtp\" (UID: \"5ab2d121-68d1-4cc1-9f0a-96755f3782f4\") " pod="openshift-ingress/e2e-api-a-8c0086-openshift-default-58b446d58c-cdgtp" Oct 31 04:20:26.281592 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:26.281566 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ab2d121-68d1-4cc1-9f0a-96755f3782f4-istio-envoy\") pod \"e2e-api-a-8c0086-openshift-default-58b446d58c-cdgtp\" (UID: \"5ab2d121-68d1-4cc1-9f0a-96755f3782f4\") " pod="openshift-ingress/e2e-api-a-8c0086-openshift-default-58b446d58c-cdgtp" Oct 31 04:20:26.312351 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:26.310814 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rvtx\" (UniqueName: \"kubernetes.io/projected/5ab2d121-68d1-4cc1-9f0a-96755f3782f4-kube-api-access-7rvtx\") pod \"e2e-api-a-8c0086-openshift-default-58b446d58c-cdgtp\" (UID: \"5ab2d121-68d1-4cc1-9f0a-96755f3782f4\") " pod="openshift-ingress/e2e-api-a-8c0086-openshift-default-58b446d58c-cdgtp" Oct 31 04:20:26.327383 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:26.320932 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ab2d121-68d1-4cc1-9f0a-96755f3782f4-istio-token\") pod \"e2e-api-a-8c0086-openshift-default-58b446d58c-cdgtp\" (UID: \"5ab2d121-68d1-4cc1-9f0a-96755f3782f4\") " pod="openshift-ingress/e2e-api-a-8c0086-openshift-default-58b446d58c-cdgtp" Oct 31 04:20:26.419433 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:26.418122 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-8c0086-openshift-default-58b446d58c-cdgtp" Oct 31 04:20:26.603886 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:26.601049 4376 patch_prober.go:29] interesting pod/e2e-worker-w2-09a20a-openshift-default-55d8647ff-scmn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Oct 31 04:20:26.603886 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:26.601108 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-scmn8" podUID="f95ddb3e-3a81-433a-aead-9a220d1ae61b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Oct 31 04:20:26.828046 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:26.828015 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-8c0086-openshift-default-58b446d58c-cdgtp"] Oct 31 04:20:26.828167 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:20:26.828138 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ab2d121_68d1_4cc1_9f0a_96755f3782f4.slice/crio-da13116685351b425ee141cd219b0b05a848647a269527e758f64e4e61d78c1a WatchSource:0}: Error finding container da13116685351b425ee141cd219b0b05a848647a269527e758f64e4e61d78c1a: Status 404 returned error can't find the container with id da13116685351b425ee141cd219b0b05a848647a269527e758f64e4e61d78c1a Oct 31 04:20:26.830656 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:26.830625 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:20:26.830817 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:26.830801 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:20:26.830899 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:26.830888 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:20:27.120227 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:27.119354 4376 patch_prober.go:29] interesting pod/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-cjmdq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Oct 31 04:20:27.120227 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:27.119408 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-cjmdq" podUID="6213a9f9-302f-4896-a1ce-dc0c7323082c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Oct 31 04:20:27.133706 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:27.133617 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-8c0086-openshift-default-58b446d58c-cdgtp" event={"ID":"5ab2d121-68d1-4cc1-9f0a-96755f3782f4","Type":"ContainerStarted","Data":"98e53ff19986f018955a4600a3874f577718448d591b945669c50655a96d8981"} Oct 31 04:20:27.133898 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:27.133876 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-8c0086-openshift-default-58b446d58c-cdgtp" event={"ID":"5ab2d121-68d1-4cc1-9f0a-96755f3782f4","Type":"ContainerStarted","Data":"da13116685351b425ee141cd219b0b05a848647a269527e758f64e4e61d78c1a"} Oct 31 04:20:27.203186 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:27.203078 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-8c0086-openshift-default-58b446d58c-cdgtp" podStartSLOduration=1.203056709 podStartE2EDuration="1.203056709s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:20:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:20:27.188767212 +0000 UTC m=+974.903101858" watchObservedRunningTime="2026-10-31 04:20:27.203056709 +0000 UTC m=+974.917391328" Oct 31 04:20:27.429551 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:27.429417 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-8c0086-openshift-default-58b446d58c-cdgtp" Oct 31 04:20:27.648774 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:27.633693 4376 patch_prober.go:29] interesting pod/e2e-worker-w2-09a20a-openshift-default-55d8647ff-scmn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Oct 31 04:20:27.648774 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:27.633750 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-scmn8" podUID="f95ddb3e-3a81-433a-aead-9a220d1ae61b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Oct 31 04:20:28.118559 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:28.118514 4376 patch_prober.go:29] interesting pod/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-cjmdq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Oct 31 04:20:28.118740 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:28.118670 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-cjmdq" podUID="6213a9f9-302f-4896-a1ce-dc0c7323082c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Oct 31 04:20:28.426130 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:28.426052 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-8c0086-openshift-default-58b446d58c-cdgtp" Oct 31 04:20:28.598784 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:28.598753 4376 patch_prober.go:29] interesting pod/e2e-worker-w2-09a20a-openshift-default-55d8647ff-scmn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Oct 31 04:20:28.598987 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:28.598815 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-scmn8" podUID="f95ddb3e-3a81-433a-aead-9a220d1ae61b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Oct 31 04:20:29.118853 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:29.118818 4376 patch_prober.go:29] interesting pod/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-cjmdq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Oct 31 04:20:29.119040 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:29.118880 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-cjmdq" podUID="6213a9f9-302f-4896-a1ce-dc0c7323082c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Oct 31 04:20:29.172110 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:29.172070 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-8c0086-openshift-default-58b446d58c-cdgtp" Oct 31 04:20:29.173135 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:29.173118 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-8c0086-openshift-default-58b446d58c-cdgtp" Oct 31 04:20:29.985827 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:29.985795 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-scmn8" Oct 31 04:20:29.986513 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:29.986490 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-scmn8" Oct 31 04:20:29.988298 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:29.988271 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-scmn8" Oct 31 04:20:30.118808 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:30.118776 4376 patch_prober.go:29] interesting pod/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-cjmdq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Oct 31 04:20:30.118978 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:30.118882 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-cjmdq" podUID="6213a9f9-302f-4896-a1ce-dc0c7323082c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Oct 31 04:20:30.601117 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:30.601083 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-9883e4-openshift-default-5666f48c8c-g9v2t"] Oct 31 04:20:30.606137 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:30.606110 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-9883e4-openshift-default-5666f48c8c-g9v2t" Oct 31 04:20:30.627721 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:30.627684 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-9883e4-openshift-default-dockercfg-5j476\"" Oct 31 04:20:30.628684 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:30.628661 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-9883e4-openshift-default-5666f48c8c-g9v2t"] Oct 31 04:20:30.678868 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:30.678821 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dc8f6\" (UniqueName: \"kubernetes.io/projected/5c611e14-54b1-42de-b10d-faf45c6f6c04-kube-api-access-dc8f6\") pod \"e2e-api-a-9883e4-openshift-default-5666f48c8c-g9v2t\" (UID: \"5c611e14-54b1-42de-b10d-faf45c6f6c04\") " pod="openshift-ingress/e2e-api-a-9883e4-openshift-default-5666f48c8c-g9v2t" Oct 31 04:20:30.679018 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:30.678950 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c611e14-54b1-42de-b10d-faf45c6f6c04-istio-token\") pod \"e2e-api-a-9883e4-openshift-default-5666f48c8c-g9v2t\" (UID: \"5c611e14-54b1-42de-b10d-faf45c6f6c04\") " pod="openshift-ingress/e2e-api-a-9883e4-openshift-default-5666f48c8c-g9v2t" Oct 31 04:20:30.679076 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:30.679023 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c611e14-54b1-42de-b10d-faf45c6f6c04-istiod-ca-cert\") pod \"e2e-api-a-9883e4-openshift-default-5666f48c8c-g9v2t\" (UID: \"5c611e14-54b1-42de-b10d-faf45c6f6c04\") " pod="openshift-ingress/e2e-api-a-9883e4-openshift-default-5666f48c8c-g9v2t" Oct 31 04:20:30.679076 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:30.679041 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c611e14-54b1-42de-b10d-faf45c6f6c04-credential-socket\") pod \"e2e-api-a-9883e4-openshift-default-5666f48c8c-g9v2t\" (UID: \"5c611e14-54b1-42de-b10d-faf45c6f6c04\") " pod="openshift-ingress/e2e-api-a-9883e4-openshift-default-5666f48c8c-g9v2t" Oct 31 04:20:30.679147 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:30.679089 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c611e14-54b1-42de-b10d-faf45c6f6c04-workload-certs\") pod \"e2e-api-a-9883e4-openshift-default-5666f48c8c-g9v2t\" (UID: \"5c611e14-54b1-42de-b10d-faf45c6f6c04\") " pod="openshift-ingress/e2e-api-a-9883e4-openshift-default-5666f48c8c-g9v2t" Oct 31 04:20:30.679195 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:30.679179 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c611e14-54b1-42de-b10d-faf45c6f6c04-istio-envoy\") pod \"e2e-api-a-9883e4-openshift-default-5666f48c8c-g9v2t\" (UID: \"5c611e14-54b1-42de-b10d-faf45c6f6c04\") " pod="openshift-ingress/e2e-api-a-9883e4-openshift-default-5666f48c8c-g9v2t" Oct 31 04:20:30.679262 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:30.679247 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c611e14-54b1-42de-b10d-faf45c6f6c04-workload-socket\") pod \"e2e-api-a-9883e4-openshift-default-5666f48c8c-g9v2t\" (UID: \"5c611e14-54b1-42de-b10d-faf45c6f6c04\") " pod="openshift-ingress/e2e-api-a-9883e4-openshift-default-5666f48c8c-g9v2t" Oct 31 04:20:30.679299 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:30.679281 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c611e14-54b1-42de-b10d-faf45c6f6c04-istio-podinfo\") pod \"e2e-api-a-9883e4-openshift-default-5666f48c8c-g9v2t\" (UID: \"5c611e14-54b1-42de-b10d-faf45c6f6c04\") " pod="openshift-ingress/e2e-api-a-9883e4-openshift-default-5666f48c8c-g9v2t" Oct 31 04:20:30.679395 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:30.679384 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c611e14-54b1-42de-b10d-faf45c6f6c04-istio-data\") pod \"e2e-api-a-9883e4-openshift-default-5666f48c8c-g9v2t\" (UID: \"5c611e14-54b1-42de-b10d-faf45c6f6c04\") " pod="openshift-ingress/e2e-api-a-9883e4-openshift-default-5666f48c8c-g9v2t" Oct 31 04:20:30.786359 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:30.785173 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c611e14-54b1-42de-b10d-faf45c6f6c04-istiod-ca-cert\") pod \"e2e-api-a-9883e4-openshift-default-5666f48c8c-g9v2t\" (UID: \"5c611e14-54b1-42de-b10d-faf45c6f6c04\") " pod="openshift-ingress/e2e-api-a-9883e4-openshift-default-5666f48c8c-g9v2t" Oct 31 04:20:30.786359 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:30.785219 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c611e14-54b1-42de-b10d-faf45c6f6c04-credential-socket\") pod \"e2e-api-a-9883e4-openshift-default-5666f48c8c-g9v2t\" (UID: \"5c611e14-54b1-42de-b10d-faf45c6f6c04\") " pod="openshift-ingress/e2e-api-a-9883e4-openshift-default-5666f48c8c-g9v2t" Oct 31 04:20:30.786359 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:30.785245 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c611e14-54b1-42de-b10d-faf45c6f6c04-workload-certs\") pod \"e2e-api-a-9883e4-openshift-default-5666f48c8c-g9v2t\" (UID: \"5c611e14-54b1-42de-b10d-faf45c6f6c04\") " pod="openshift-ingress/e2e-api-a-9883e4-openshift-default-5666f48c8c-g9v2t" Oct 31 04:20:30.786359 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:30.785316 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c611e14-54b1-42de-b10d-faf45c6f6c04-istio-envoy\") pod \"e2e-api-a-9883e4-openshift-default-5666f48c8c-g9v2t\" (UID: \"5c611e14-54b1-42de-b10d-faf45c6f6c04\") " pod="openshift-ingress/e2e-api-a-9883e4-openshift-default-5666f48c8c-g9v2t" Oct 31 04:20:30.786359 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:30.785399 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c611e14-54b1-42de-b10d-faf45c6f6c04-workload-socket\") pod \"e2e-api-a-9883e4-openshift-default-5666f48c8c-g9v2t\" (UID: \"5c611e14-54b1-42de-b10d-faf45c6f6c04\") " pod="openshift-ingress/e2e-api-a-9883e4-openshift-default-5666f48c8c-g9v2t" Oct 31 04:20:30.786359 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:30.785433 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c611e14-54b1-42de-b10d-faf45c6f6c04-istio-podinfo\") pod \"e2e-api-a-9883e4-openshift-default-5666f48c8c-g9v2t\" (UID: \"5c611e14-54b1-42de-b10d-faf45c6f6c04\") " pod="openshift-ingress/e2e-api-a-9883e4-openshift-default-5666f48c8c-g9v2t" Oct 31 04:20:30.786359 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:30.785555 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c611e14-54b1-42de-b10d-faf45c6f6c04-istio-data\") pod \"e2e-api-a-9883e4-openshift-default-5666f48c8c-g9v2t\" (UID: \"5c611e14-54b1-42de-b10d-faf45c6f6c04\") " pod="openshift-ingress/e2e-api-a-9883e4-openshift-default-5666f48c8c-g9v2t" Oct 31 04:20:30.786359 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:30.785620 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dc8f6\" (UniqueName: \"kubernetes.io/projected/5c611e14-54b1-42de-b10d-faf45c6f6c04-kube-api-access-dc8f6\") pod \"e2e-api-a-9883e4-openshift-default-5666f48c8c-g9v2t\" (UID: \"5c611e14-54b1-42de-b10d-faf45c6f6c04\") " pod="openshift-ingress/e2e-api-a-9883e4-openshift-default-5666f48c8c-g9v2t" Oct 31 04:20:30.786359 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:30.785695 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c611e14-54b1-42de-b10d-faf45c6f6c04-istio-token\") pod \"e2e-api-a-9883e4-openshift-default-5666f48c8c-g9v2t\" (UID: \"5c611e14-54b1-42de-b10d-faf45c6f6c04\") " pod="openshift-ingress/e2e-api-a-9883e4-openshift-default-5666f48c8c-g9v2t" Oct 31 04:20:30.792659 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:30.787360 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c611e14-54b1-42de-b10d-faf45c6f6c04-istio-data\") pod \"e2e-api-a-9883e4-openshift-default-5666f48c8c-g9v2t\" (UID: \"5c611e14-54b1-42de-b10d-faf45c6f6c04\") " pod="openshift-ingress/e2e-api-a-9883e4-openshift-default-5666f48c8c-g9v2t" Oct 31 04:20:30.792659 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:30.787656 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c611e14-54b1-42de-b10d-faf45c6f6c04-workload-certs\") pod \"e2e-api-a-9883e4-openshift-default-5666f48c8c-g9v2t\" (UID: \"5c611e14-54b1-42de-b10d-faf45c6f6c04\") " pod="openshift-ingress/e2e-api-a-9883e4-openshift-default-5666f48c8c-g9v2t" Oct 31 04:20:30.792659 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:30.788008 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c611e14-54b1-42de-b10d-faf45c6f6c04-workload-socket\") pod \"e2e-api-a-9883e4-openshift-default-5666f48c8c-g9v2t\" (UID: \"5c611e14-54b1-42de-b10d-faf45c6f6c04\") " pod="openshift-ingress/e2e-api-a-9883e4-openshift-default-5666f48c8c-g9v2t" Oct 31 04:20:30.792659 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:30.788453 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c611e14-54b1-42de-b10d-faf45c6f6c04-istiod-ca-cert\") pod \"e2e-api-a-9883e4-openshift-default-5666f48c8c-g9v2t\" (UID: \"5c611e14-54b1-42de-b10d-faf45c6f6c04\") " pod="openshift-ingress/e2e-api-a-9883e4-openshift-default-5666f48c8c-g9v2t" Oct 31 04:20:30.824358 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:30.799709 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c611e14-54b1-42de-b10d-faf45c6f6c04-credential-socket\") pod \"e2e-api-a-9883e4-openshift-default-5666f48c8c-g9v2t\" (UID: \"5c611e14-54b1-42de-b10d-faf45c6f6c04\") " pod="openshift-ingress/e2e-api-a-9883e4-openshift-default-5666f48c8c-g9v2t" Oct 31 04:20:30.824358 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:30.810122 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c611e14-54b1-42de-b10d-faf45c6f6c04-istio-podinfo\") pod \"e2e-api-a-9883e4-openshift-default-5666f48c8c-g9v2t\" (UID: \"5c611e14-54b1-42de-b10d-faf45c6f6c04\") " pod="openshift-ingress/e2e-api-a-9883e4-openshift-default-5666f48c8c-g9v2t" Oct 31 04:20:30.828353 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:30.827640 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c611e14-54b1-42de-b10d-faf45c6f6c04-istio-envoy\") pod \"e2e-api-a-9883e4-openshift-default-5666f48c8c-g9v2t\" (UID: \"5c611e14-54b1-42de-b10d-faf45c6f6c04\") " pod="openshift-ingress/e2e-api-a-9883e4-openshift-default-5666f48c8c-g9v2t" Oct 31 04:20:30.851395 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:30.835959 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c611e14-54b1-42de-b10d-faf45c6f6c04-istio-token\") pod \"e2e-api-a-9883e4-openshift-default-5666f48c8c-g9v2t\" (UID: \"5c611e14-54b1-42de-b10d-faf45c6f6c04\") " pod="openshift-ingress/e2e-api-a-9883e4-openshift-default-5666f48c8c-g9v2t" Oct 31 04:20:30.851395 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:30.847645 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dc8f6\" (UniqueName: \"kubernetes.io/projected/5c611e14-54b1-42de-b10d-faf45c6f6c04-kube-api-access-dc8f6\") pod \"e2e-api-a-9883e4-openshift-default-5666f48c8c-g9v2t\" (UID: \"5c611e14-54b1-42de-b10d-faf45c6f6c04\") " pod="openshift-ingress/e2e-api-a-9883e4-openshift-default-5666f48c8c-g9v2t" Oct 31 04:20:30.932348 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:30.931657 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-9883e4-openshift-default-5666f48c8c-g9v2t" Oct 31 04:20:31.121214 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:31.120504 4376 patch_prober.go:29] interesting pod/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-cjmdq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Oct 31 04:20:31.121214 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:31.120563 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-cjmdq" podUID="6213a9f9-302f-4896-a1ce-dc0c7323082c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Oct 31 04:20:31.372905 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:20:31.372877 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c611e14_54b1_42de_b10d_faf45c6f6c04.slice/crio-8efcb68892655affb7e852e917fb2cad3b66712aacebdd57072207b20e430614 WatchSource:0}: Error finding container 8efcb68892655affb7e852e917fb2cad3b66712aacebdd57072207b20e430614: Status 404 returned error can't find the container with id 8efcb68892655affb7e852e917fb2cad3b66712aacebdd57072207b20e430614 Oct 31 04:20:31.374518 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:31.374497 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-9883e4-openshift-default-5666f48c8c-g9v2t"] Oct 31 04:20:31.375878 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:31.375848 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:20:31.375934 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:31.375912 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:20:31.375975 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:31.375941 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:20:32.239356 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:32.237514 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-8c0086-fd5c48549-k5ltz"] Oct 31 04:20:32.295810 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:32.288888 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-8c0086-fd5c48549-k5ltz" Oct 31 04:20:32.355370 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:32.354121 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-8c0086-dockercfg-hd9jv\"" Oct 31 04:20:32.383574 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:32.363631 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-8c0086\"" Oct 31 04:20:32.409885 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:32.409164 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9883e4-openshift-default-5666f48c8c-g9v2t" event={"ID":"5c611e14-54b1-42de-b10d-faf45c6f6c04","Type":"ContainerStarted","Data":"a5402ceb186f4948fa5368278a004b1f1ae679eea48803c4f687e9fba6d03f26"} Oct 31 04:20:32.409885 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:32.409224 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9883e4-openshift-default-5666f48c8c-g9v2t" event={"ID":"5c611e14-54b1-42de-b10d-faf45c6f6c04","Type":"ContainerStarted","Data":"8efcb68892655affb7e852e917fb2cad3b66712aacebdd57072207b20e430614"} Oct 31 04:20:32.448577 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:32.447747 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9de59006-1b3d-48f4-a990-a51968da6a23-plugins-config-volume\") pod \"payload-processing-e2e-api-a-8c0086-fd5c48549-k5ltz\" (UID: \"9de59006-1b3d-48f4-a990-a51968da6a23\") " pod="openshift-ingress/payload-processing-e2e-api-a-8c0086-fd5c48549-k5ltz" Oct 31 04:20:32.448577 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:32.447827 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgw85\" (UniqueName: \"kubernetes.io/projected/9de59006-1b3d-48f4-a990-a51968da6a23-kube-api-access-jgw85\") pod \"payload-processing-e2e-api-a-8c0086-fd5c48549-k5ltz\" (UID: \"9de59006-1b3d-48f4-a990-a51968da6a23\") " pod="openshift-ingress/payload-processing-e2e-api-a-8c0086-fd5c48549-k5ltz" Oct 31 04:20:32.467641 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:32.467608 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-8c0086-fd5c48549-k5ltz"] Oct 31 04:20:32.472737 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:32.472674 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-9883e4-openshift-default-5666f48c8c-g9v2t" podStartSLOduration=2.472658154 podStartE2EDuration="2.472658154s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:20:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:20:32.470855007 +0000 UTC m=+980.185189644" watchObservedRunningTime="2026-10-31 04:20:32.472658154 +0000 UTC m=+980.186992772" Oct 31 04:20:32.549135 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:32.549045 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9de59006-1b3d-48f4-a990-a51968da6a23-plugins-config-volume\") pod \"payload-processing-e2e-api-a-8c0086-fd5c48549-k5ltz\" (UID: \"9de59006-1b3d-48f4-a990-a51968da6a23\") " pod="openshift-ingress/payload-processing-e2e-api-a-8c0086-fd5c48549-k5ltz" Oct 31 04:20:32.549135 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:32.549107 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgw85\" (UniqueName: \"kubernetes.io/projected/9de59006-1b3d-48f4-a990-a51968da6a23-kube-api-access-jgw85\") pod \"payload-processing-e2e-api-a-8c0086-fd5c48549-k5ltz\" (UID: \"9de59006-1b3d-48f4-a990-a51968da6a23\") " pod="openshift-ingress/payload-processing-e2e-api-a-8c0086-fd5c48549-k5ltz" Oct 31 04:20:32.549946 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:32.549921 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9de59006-1b3d-48f4-a990-a51968da6a23-plugins-config-volume\") pod \"payload-processing-e2e-api-a-8c0086-fd5c48549-k5ltz\" (UID: \"9de59006-1b3d-48f4-a990-a51968da6a23\") " pod="openshift-ingress/payload-processing-e2e-api-a-8c0086-fd5c48549-k5ltz" Oct 31 04:20:32.594589 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:32.594552 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgw85\" (UniqueName: \"kubernetes.io/projected/9de59006-1b3d-48f4-a990-a51968da6a23-kube-api-access-jgw85\") pod \"payload-processing-e2e-api-a-8c0086-fd5c48549-k5ltz\" (UID: \"9de59006-1b3d-48f4-a990-a51968da6a23\") " pod="openshift-ingress/payload-processing-e2e-api-a-8c0086-fd5c48549-k5ltz" Oct 31 04:20:32.721727 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:32.721694 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-8c0086-fd5c48549-k5ltz" Oct 31 04:20:32.798373 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:32.796594 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-cjmdq" Oct 31 04:20:32.798373 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:32.797419 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-cjmdq" Oct 31 04:20:32.798731 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:32.798712 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-cjmdq" Oct 31 04:20:32.932191 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:32.932149 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-9883e4-openshift-default-5666f48c8c-g9v2t" Oct 31 04:20:32.937106 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:32.937084 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-9883e4-openshift-default-5666f48c8c-g9v2t" Oct 31 04:20:33.070976 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:20:33.070946 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9de59006_1b3d_48f4_a990_a51968da6a23.slice/crio-878dea6b7da3ee2620e2e6f08baa981267863a333bb5462d9ed744b0b6c3d2c5 WatchSource:0}: Error finding container 878dea6b7da3ee2620e2e6f08baa981267863a333bb5462d9ed744b0b6c3d2c5: Status 404 returned error can't find the container with id 878dea6b7da3ee2620e2e6f08baa981267863a333bb5462d9ed744b0b6c3d2c5 Oct 31 04:20:33.073016 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:33.072866 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-8c0086-fd5c48549-k5ltz"] Oct 31 04:20:33.416491 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:33.416455 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-8c0086-fd5c48549-k5ltz" event={"ID":"9de59006-1b3d-48f4-a990-a51968da6a23","Type":"ContainerStarted","Data":"14a6dbaf540610df7d6709f2591b0f2830255ba50682869ea00d3a5d58840c4a"} Oct 31 04:20:33.416925 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:33.416519 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-8c0086-fd5c48549-k5ltz" event={"ID":"9de59006-1b3d-48f4-a990-a51968da6a23","Type":"ContainerStarted","Data":"878dea6b7da3ee2620e2e6f08baa981267863a333bb5462d9ed744b0b6c3d2c5"} Oct 31 04:20:33.416925 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:33.416546 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-8c0086-fd5c48549-k5ltz" Oct 31 04:20:33.417082 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:33.417059 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-9883e4-openshift-default-5666f48c8c-g9v2t" Oct 31 04:20:33.417893 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:33.417871 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-9883e4-openshift-default-5666f48c8c-g9v2t" Oct 31 04:20:33.465277 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:33.465229 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-8c0086-fd5c48549-k5ltz" podStartSLOduration=1.4652144489999999 podStartE2EDuration="1.465214449s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:20:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:20:33.462100497 +0000 UTC m=+981.176435133" watchObservedRunningTime="2026-10-31 04:20:33.465214449 +0000 UTC m=+981.179549065" Oct 31 04:20:41.220352 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:20:41.218523 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 04:20:41.950355 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:41.938762 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-d7npm" podUID="174171e7-100c-4b84-becd-0176672f1805" containerName="main" probeResult="failure" output="Get \"https://10.133.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:20:41.950355 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:41.939203 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c4006d/simulator-w4-c4006d-kserve-595455485b-4h46q" podUID="576e2c1f-1c80-43a9-a136-3692776642ae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded" Oct 31 04:20:41.950355 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:41.946441 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-09a20a/distinct-w2-09a20a-kserve-76557f6448-l8d69" podUID="7d298000-b65b-45fd-a1ee-092e47a7eb24" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded" Oct 31 04:20:41.950355 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:41.946798 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4c2387/distinct-w3-4c2387-kserve-768bdd84c7-6jmtl" podUID="b2c75691-4cf4-484b-96e2-3c2503b8d78c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:20:41.988393 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:41.981419 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c4006d/distinct-2-w4-c4006d-kserve-776b5b899-kpl6z" podUID="dc53083d-eadd-4cb8-8633-35e02d400b9c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 31 04:20:42.010839 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:42.010062 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-09a20a/distinct-2-w2-09a20a-kserve-779d648d75-mshl7" podUID="c81f5d58-c5c7-46c1-b832-c550b092f02b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:20:42.053591 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:42.053550 4376 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-439989/premium-w1-439989-kserve-f9fc54cdf-sn2pp" podUID="17b2f181-2e66-4ce2-95b0-cb213f11a0e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": context deadline exceeded" Oct 31 04:20:42.065495 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:42.065450 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-439989/premium-w1-439989-kserve-f9fc54cdf-sn2pp" podUID="17b2f181-2e66-4ce2-95b0-cb213f11a0e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:20:42.701352 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:42.700696 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-09a20a/premium-w2-09a20a-kserve-57c45db74f-s8glf" podUID="a3f9d0a0-a106-48bb-b6f2-a980c3aff4e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:20:42.950068 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:42.950026 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-09a20a/distinct-w2-09a20a-kserve-76557f6448-l8d69" podUID="7d298000-b65b-45fd-a1ee-092e47a7eb24" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:20:42.951793 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:42.951706 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-09a20a/distinct-w2-09a20a-kserve-76557f6448-l8d69" Oct 31 04:20:43.100048 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:43.100018 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-09a20a/distinct-w2-09a20a-kserve-76557f6448-l8d69" Oct 31 04:20:44.426834 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:44.426805 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-8c0086-fd5c48549-k5ltz" Oct 31 04:20:51.647110 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:51.647078 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-9883e4-7f5fbccbbd-tv8v6"] Oct 31 04:20:51.653471 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:51.653441 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9883e4-7f5fbccbbd-tv8v6" Oct 31 04:20:51.670851 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:51.670821 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-9883e4-7f5fbccbbd-tv8v6"] Oct 31 04:20:51.678657 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:51.678636 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-9883e4-serving-cert\"" Oct 31 04:20:51.716739 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:51.716712 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47ce5c05-1ed0-471c-8a53-7d6f581e5f48-metrics-certs\") pod \"maas-api-e2e-api-a-9883e4-7f5fbccbbd-tv8v6\" (UID: \"47ce5c05-1ed0-471c-8a53-7d6f581e5f48\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9883e4-7f5fbccbbd-tv8v6" Oct 31 04:20:51.716887 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:51.716853 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/47ce5c05-1ed0-471c-8a53-7d6f581e5f48-openshift-service-ca\") pod \"maas-api-e2e-api-a-9883e4-7f5fbccbbd-tv8v6\" (UID: \"47ce5c05-1ed0-471c-8a53-7d6f581e5f48\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9883e4-7f5fbccbbd-tv8v6" Oct 31 04:20:51.716928 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:51.716916 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47ce5c05-1ed0-471c-8a53-7d6f581e5f48-maas-api-tls\") pod \"maas-api-e2e-api-a-9883e4-7f5fbccbbd-tv8v6\" (UID: \"47ce5c05-1ed0-471c-8a53-7d6f581e5f48\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9883e4-7f5fbccbbd-tv8v6" Oct 31 04:20:51.716975 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:51.716963 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppcsd\" (UniqueName: \"kubernetes.io/projected/47ce5c05-1ed0-471c-8a53-7d6f581e5f48-kube-api-access-ppcsd\") pod \"maas-api-e2e-api-a-9883e4-7f5fbccbbd-tv8v6\" (UID: \"47ce5c05-1ed0-471c-8a53-7d6f581e5f48\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9883e4-7f5fbccbbd-tv8v6" Oct 31 04:20:51.817886 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:51.817853 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47ce5c05-1ed0-471c-8a53-7d6f581e5f48-metrics-certs\") pod \"maas-api-e2e-api-a-9883e4-7f5fbccbbd-tv8v6\" (UID: \"47ce5c05-1ed0-471c-8a53-7d6f581e5f48\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9883e4-7f5fbccbbd-tv8v6" Oct 31 04:20:51.818078 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:51.817944 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/47ce5c05-1ed0-471c-8a53-7d6f581e5f48-openshift-service-ca\") pod \"maas-api-e2e-api-a-9883e4-7f5fbccbbd-tv8v6\" (UID: \"47ce5c05-1ed0-471c-8a53-7d6f581e5f48\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9883e4-7f5fbccbbd-tv8v6" Oct 31 04:20:51.818078 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:51.817965 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47ce5c05-1ed0-471c-8a53-7d6f581e5f48-maas-api-tls\") pod \"maas-api-e2e-api-a-9883e4-7f5fbccbbd-tv8v6\" (UID: \"47ce5c05-1ed0-471c-8a53-7d6f581e5f48\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9883e4-7f5fbccbbd-tv8v6" Oct 31 04:20:51.818078 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:51.818004 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppcsd\" (UniqueName: \"kubernetes.io/projected/47ce5c05-1ed0-471c-8a53-7d6f581e5f48-kube-api-access-ppcsd\") pod \"maas-api-e2e-api-a-9883e4-7f5fbccbbd-tv8v6\" (UID: \"47ce5c05-1ed0-471c-8a53-7d6f581e5f48\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9883e4-7f5fbccbbd-tv8v6" Oct 31 04:20:51.818258 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:20:51.818111 4376 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-9883e4-serving-cert: secret "maas-api-e2e-api-a-9883e4-serving-cert" not found Oct 31 04:20:51.818258 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:20:51.818204 4376 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/47ce5c05-1ed0-471c-8a53-7d6f581e5f48-maas-api-tls podName:47ce5c05-1ed0-471c-8a53-7d6f581e5f48 nodeName:}" failed. No retries permitted until 2026-10-31 04:20:52.318182633 +0000 UTC m=+1000.032517231 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/47ce5c05-1ed0-471c-8a53-7d6f581e5f48-maas-api-tls") pod "maas-api-e2e-api-a-9883e4-7f5fbccbbd-tv8v6" (UID: "47ce5c05-1ed0-471c-8a53-7d6f581e5f48") : secret "maas-api-e2e-api-a-9883e4-serving-cert" not found Oct 31 04:20:51.818620 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:51.818601 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/47ce5c05-1ed0-471c-8a53-7d6f581e5f48-openshift-service-ca\") pod \"maas-api-e2e-api-a-9883e4-7f5fbccbbd-tv8v6\" (UID: \"47ce5c05-1ed0-471c-8a53-7d6f581e5f48\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9883e4-7f5fbccbbd-tv8v6" Oct 31 04:20:51.820298 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:51.820279 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47ce5c05-1ed0-471c-8a53-7d6f581e5f48-metrics-certs\") pod \"maas-api-e2e-api-a-9883e4-7f5fbccbbd-tv8v6\" (UID: \"47ce5c05-1ed0-471c-8a53-7d6f581e5f48\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9883e4-7f5fbccbbd-tv8v6" Oct 31 04:20:51.869350 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:51.869268 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppcsd\" (UniqueName: \"kubernetes.io/projected/47ce5c05-1ed0-471c-8a53-7d6f581e5f48-kube-api-access-ppcsd\") pod \"maas-api-e2e-api-a-9883e4-7f5fbccbbd-tv8v6\" (UID: \"47ce5c05-1ed0-471c-8a53-7d6f581e5f48\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9883e4-7f5fbccbbd-tv8v6" Oct 31 04:20:52.326178 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:52.326144 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47ce5c05-1ed0-471c-8a53-7d6f581e5f48-maas-api-tls\") pod \"maas-api-e2e-api-a-9883e4-7f5fbccbbd-tv8v6\" (UID: \"47ce5c05-1ed0-471c-8a53-7d6f581e5f48\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9883e4-7f5fbccbbd-tv8v6" Oct 31 04:20:52.328631 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:52.328600 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47ce5c05-1ed0-471c-8a53-7d6f581e5f48-maas-api-tls\") pod \"maas-api-e2e-api-a-9883e4-7f5fbccbbd-tv8v6\" (UID: \"47ce5c05-1ed0-471c-8a53-7d6f581e5f48\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9883e4-7f5fbccbbd-tv8v6" Oct 31 04:20:52.587912 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:52.587593 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9883e4-7f5fbccbbd-tv8v6" Oct 31 04:20:53.051827 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:53.049726 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-9883e4-7f5fbccbbd-tv8v6"] Oct 31 04:20:53.055115 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:53.054155 4376 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 04:20:53.291996 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:53.291906 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9883e4-7f5fbccbbd-tv8v6" event={"ID":"47ce5c05-1ed0-471c-8a53-7d6f581e5f48","Type":"ContainerStarted","Data":"89010ab1d2b8f3bfaca477df0009bd4d9ecf9878aac38c3f7fb2d581e18885e7"} Oct 31 04:20:54.299392 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:54.299357 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9883e4-7f5fbccbbd-tv8v6" event={"ID":"47ce5c05-1ed0-471c-8a53-7d6f581e5f48","Type":"ContainerStarted","Data":"e45f86710c598eb9820798d43a15f9a01d91eaa40453b889891157095d91aa8b"} Oct 31 04:20:54.299777 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:54.299428 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9883e4-7f5fbccbbd-tv8v6" Oct 31 04:20:54.344615 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:20:54.344569 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9883e4-7f5fbccbbd-tv8v6" podStartSLOduration=2.9246194 podStartE2EDuration="3.344557609s" totalImagesPullingTime="419.938209ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:20:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:20:54.342678917 +0000 UTC m=+1002.057013534" watchObservedRunningTime="2026-10-31 04:20:54.344557609 +0000 UTC m=+1002.058892226" Oct 31 04:21:00.311185 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:00.311113 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9883e4-7f5fbccbbd-tv8v6" Oct 31 04:21:08.040080 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:08.040046 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 04:21:08.893351 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:08.890530 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-k2jb2"] Oct 31 04:21:08.940352 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:08.939590 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-k2jb2"] Oct 31 04:21:08.940352 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:08.939720 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-k2jb2" Oct 31 04:21:09.057810 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:09.047490 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/716d4a99-414f-449d-89c2-fa06e5751c60-workload-certs\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-k2jb2\" (UID: \"716d4a99-414f-449d-89c2-fa06e5751c60\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-k2jb2" Oct 31 04:21:09.057810 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:09.047533 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/716d4a99-414f-449d-89c2-fa06e5751c60-istio-podinfo\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-k2jb2\" (UID: \"716d4a99-414f-449d-89c2-fa06e5751c60\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-k2jb2" Oct 31 04:21:09.057810 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:09.047610 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/716d4a99-414f-449d-89c2-fa06e5751c60-istiod-ca-cert\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-k2jb2\" (UID: \"716d4a99-414f-449d-89c2-fa06e5751c60\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-k2jb2" Oct 31 04:21:09.057810 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:09.047845 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/716d4a99-414f-449d-89c2-fa06e5751c60-istio-envoy\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-k2jb2\" (UID: \"716d4a99-414f-449d-89c2-fa06e5751c60\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-k2jb2" Oct 31 04:21:09.057810 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:09.047947 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhz97\" (UniqueName: \"kubernetes.io/projected/716d4a99-414f-449d-89c2-fa06e5751c60-kube-api-access-dhz97\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-k2jb2\" (UID: \"716d4a99-414f-449d-89c2-fa06e5751c60\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-k2jb2" Oct 31 04:21:09.057810 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:09.047991 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/716d4a99-414f-449d-89c2-fa06e5751c60-workload-socket\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-k2jb2\" (UID: \"716d4a99-414f-449d-89c2-fa06e5751c60\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-k2jb2" Oct 31 04:21:09.057810 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:09.048017 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/716d4a99-414f-449d-89c2-fa06e5751c60-istio-token\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-k2jb2\" (UID: \"716d4a99-414f-449d-89c2-fa06e5751c60\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-k2jb2" Oct 31 04:21:09.057810 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:09.048062 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/716d4a99-414f-449d-89c2-fa06e5751c60-credential-socket\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-k2jb2\" (UID: \"716d4a99-414f-449d-89c2-fa06e5751c60\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-k2jb2" Oct 31 04:21:09.057810 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:09.048131 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/716d4a99-414f-449d-89c2-fa06e5751c60-istio-data\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-k2jb2\" (UID: \"716d4a99-414f-449d-89c2-fa06e5751c60\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-k2jb2" Oct 31 04:21:09.169753 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:09.159234 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/716d4a99-414f-449d-89c2-fa06e5751c60-istio-envoy\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-k2jb2\" (UID: \"716d4a99-414f-449d-89c2-fa06e5751c60\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-k2jb2" Oct 31 04:21:09.169753 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:09.159289 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhz97\" (UniqueName: \"kubernetes.io/projected/716d4a99-414f-449d-89c2-fa06e5751c60-kube-api-access-dhz97\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-k2jb2\" (UID: \"716d4a99-414f-449d-89c2-fa06e5751c60\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-k2jb2" Oct 31 04:21:09.169753 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:09.159351 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/716d4a99-414f-449d-89c2-fa06e5751c60-workload-socket\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-k2jb2\" (UID: \"716d4a99-414f-449d-89c2-fa06e5751c60\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-k2jb2" Oct 31 04:21:09.169753 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:09.159376 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/716d4a99-414f-449d-89c2-fa06e5751c60-istio-token\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-k2jb2\" (UID: \"716d4a99-414f-449d-89c2-fa06e5751c60\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-k2jb2" Oct 31 04:21:09.169753 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:09.159425 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/716d4a99-414f-449d-89c2-fa06e5751c60-credential-socket\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-k2jb2\" (UID: \"716d4a99-414f-449d-89c2-fa06e5751c60\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-k2jb2" Oct 31 04:21:09.169753 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:09.159486 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/716d4a99-414f-449d-89c2-fa06e5751c60-istio-data\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-k2jb2\" (UID: \"716d4a99-414f-449d-89c2-fa06e5751c60\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-k2jb2" Oct 31 04:21:09.169753 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:09.159584 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/716d4a99-414f-449d-89c2-fa06e5751c60-workload-certs\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-k2jb2\" (UID: \"716d4a99-414f-449d-89c2-fa06e5751c60\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-k2jb2" Oct 31 04:21:09.169753 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:09.159610 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/716d4a99-414f-449d-89c2-fa06e5751c60-istio-podinfo\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-k2jb2\" (UID: \"716d4a99-414f-449d-89c2-fa06e5751c60\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-k2jb2" Oct 31 04:21:09.169753 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:09.159677 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/716d4a99-414f-449d-89c2-fa06e5751c60-istiod-ca-cert\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-k2jb2\" (UID: \"716d4a99-414f-449d-89c2-fa06e5751c60\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-k2jb2" Oct 31 04:21:09.169753 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:09.168588 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/716d4a99-414f-449d-89c2-fa06e5751c60-istiod-ca-cert\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-k2jb2\" (UID: \"716d4a99-414f-449d-89c2-fa06e5751c60\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-k2jb2" Oct 31 04:21:09.189627 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:09.189589 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/716d4a99-414f-449d-89c2-fa06e5751c60-credential-socket\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-k2jb2\" (UID: \"716d4a99-414f-449d-89c2-fa06e5751c60\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-k2jb2" Oct 31 04:21:09.207136 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:09.191670 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/716d4a99-414f-449d-89c2-fa06e5751c60-workload-certs\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-k2jb2\" (UID: \"716d4a99-414f-449d-89c2-fa06e5751c60\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-k2jb2" Oct 31 04:21:09.207369 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:09.206546 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/716d4a99-414f-449d-89c2-fa06e5751c60-istio-data\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-k2jb2\" (UID: \"716d4a99-414f-449d-89c2-fa06e5751c60\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-k2jb2" Oct 31 04:21:09.207475 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:09.206994 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/716d4a99-414f-449d-89c2-fa06e5751c60-workload-socket\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-k2jb2\" (UID: \"716d4a99-414f-449d-89c2-fa06e5751c60\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-k2jb2" Oct 31 04:21:09.210189 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:09.210140 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/716d4a99-414f-449d-89c2-fa06e5751c60-istio-podinfo\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-k2jb2\" (UID: \"716d4a99-414f-449d-89c2-fa06e5751c60\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-k2jb2" Oct 31 04:21:09.309463 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:09.308372 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/716d4a99-414f-449d-89c2-fa06e5751c60-istio-token\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-k2jb2\" (UID: \"716d4a99-414f-449d-89c2-fa06e5751c60\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-k2jb2" Oct 31 04:21:09.309973 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:09.309946 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/716d4a99-414f-449d-89c2-fa06e5751c60-istio-envoy\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-k2jb2\" (UID: \"716d4a99-414f-449d-89c2-fa06e5751c60\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-k2jb2" Oct 31 04:21:09.313020 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:09.312940 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhz97\" (UniqueName: \"kubernetes.io/projected/716d4a99-414f-449d-89c2-fa06e5751c60-kube-api-access-dhz97\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-k2jb2\" (UID: \"716d4a99-414f-449d-89c2-fa06e5751c60\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-k2jb2" Oct 31 04:21:09.369279 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:09.369246 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-k2jb2" Oct 31 04:21:09.689482 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:21:09.689448 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod716d4a99_414f_449d_89c2_fa06e5751c60.slice/crio-4890131363f9d90b67b89971ca6e9361848d4401391ff4f6b87b4af9b195dc5e WatchSource:0}: Error finding container 4890131363f9d90b67b89971ca6e9361848d4401391ff4f6b87b4af9b195dc5e: Status 404 returned error can't find the container with id 4890131363f9d90b67b89971ca6e9361848d4401391ff4f6b87b4af9b195dc5e Oct 31 04:21:09.690185 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:09.690156 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-k2jb2"] Oct 31 04:21:09.692107 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:09.692071 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:21:09.692219 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:09.692149 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:21:09.692219 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:09.692196 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:21:10.436257 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:10.435077 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-k2jb2" event={"ID":"716d4a99-414f-449d-89c2-fa06e5751c60","Type":"ContainerStarted","Data":"45d080900be9400444f7c02487989b189b36ce89754d4a8a50599929415695a7"} Oct 31 04:21:10.436257 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:10.435149 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-k2jb2" event={"ID":"716d4a99-414f-449d-89c2-fa06e5751c60","Type":"ContainerStarted","Data":"4890131363f9d90b67b89971ca6e9361848d4401391ff4f6b87b4af9b195dc5e"} Oct 31 04:21:10.577353 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:10.563230 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-k2jb2" podStartSLOduration=2.563210012 podStartE2EDuration="2.563210012s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:21:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:21:10.530673526 +0000 UTC m=+1018.245008167" watchObservedRunningTime="2026-10-31 04:21:10.563210012 +0000 UTC m=+1018.277544635" Oct 31 04:21:11.200353 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:21:11.197759 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 04:21:11.377422 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:11.377385 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-k2jb2" Oct 31 04:21:11.377735 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:11.377529 4376 patch_prober.go:29] interesting pod/e2e-worker-w3-4c2387-openshift-default-5b68d44445-k2jb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Oct 31 04:21:11.377907 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:11.377883 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-k2jb2" podUID="716d4a99-414f-449d-89c2-fa06e5751c60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Oct 31 04:21:12.383831 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:12.383788 4376 patch_prober.go:29] interesting pod/e2e-worker-w3-4c2387-openshift-default-5b68d44445-k2jb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Oct 31 04:21:12.385150 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:12.385111 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-k2jb2" podUID="716d4a99-414f-449d-89c2-fa06e5751c60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Oct 31 04:21:13.371460 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:13.371420 4376 patch_prober.go:29] interesting pod/e2e-worker-w3-4c2387-openshift-default-5b68d44445-k2jb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Oct 31 04:21:13.371767 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:13.371737 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-k2jb2" podUID="716d4a99-414f-449d-89c2-fa06e5751c60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Oct 31 04:21:14.372183 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:14.371724 4376 patch_prober.go:29] interesting pod/e2e-worker-w3-4c2387-openshift-default-5b68d44445-k2jb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Oct 31 04:21:14.372183 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:14.371803 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-k2jb2" podUID="716d4a99-414f-449d-89c2-fa06e5751c60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Oct 31 04:21:15.369925 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:15.369876 4376 patch_prober.go:29] interesting pod/e2e-worker-w3-4c2387-openshift-default-5b68d44445-k2jb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Oct 31 04:21:15.370240 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:15.370195 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-k2jb2" podUID="716d4a99-414f-449d-89c2-fa06e5751c60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Oct 31 04:21:16.370812 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:16.370770 4376 patch_prober.go:29] interesting pod/e2e-worker-w3-4c2387-openshift-default-5b68d44445-k2jb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Oct 31 04:21:16.371365 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:16.370865 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-k2jb2" podUID="716d4a99-414f-449d-89c2-fa06e5751c60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Oct 31 04:21:16.687821 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:16.687781 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-a43a70-openshift-default-59d5b5dcf-rlkth"] Oct 31 04:21:16.694594 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:16.694560 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-a43a70-openshift-default-59d5b5dcf-rlkth" Oct 31 04:21:16.715587 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:16.715374 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-a43a70-openshift-default-59d5b5dcf-rlkth"] Oct 31 04:21:16.722314 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:16.722256 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-a43a70-openshift-default-dockercfg-wmncs\"" Oct 31 04:21:16.776433 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:16.775780 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c4d4b3d-2929-4563-ac6a-18d5f01c0db6-istio-envoy\") pod \"e2e-api-b-a43a70-openshift-default-59d5b5dcf-rlkth\" (UID: \"3c4d4b3d-2929-4563-ac6a-18d5f01c0db6\") " pod="openshift-ingress/e2e-api-b-a43a70-openshift-default-59d5b5dcf-rlkth" Oct 31 04:21:16.776433 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:16.775835 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c4d4b3d-2929-4563-ac6a-18d5f01c0db6-istio-token\") pod \"e2e-api-b-a43a70-openshift-default-59d5b5dcf-rlkth\" (UID: \"3c4d4b3d-2929-4563-ac6a-18d5f01c0db6\") " pod="openshift-ingress/e2e-api-b-a43a70-openshift-default-59d5b5dcf-rlkth" Oct 31 04:21:16.776433 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:16.775914 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c4d4b3d-2929-4563-ac6a-18d5f01c0db6-workload-socket\") pod \"e2e-api-b-a43a70-openshift-default-59d5b5dcf-rlkth\" (UID: \"3c4d4b3d-2929-4563-ac6a-18d5f01c0db6\") " pod="openshift-ingress/e2e-api-b-a43a70-openshift-default-59d5b5dcf-rlkth" Oct 31 04:21:16.776433 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:16.776022 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c4d4b3d-2929-4563-ac6a-18d5f01c0db6-istio-podinfo\") pod \"e2e-api-b-a43a70-openshift-default-59d5b5dcf-rlkth\" (UID: \"3c4d4b3d-2929-4563-ac6a-18d5f01c0db6\") " pod="openshift-ingress/e2e-api-b-a43a70-openshift-default-59d5b5dcf-rlkth" Oct 31 04:21:16.776433 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:16.776064 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79zsd\" (UniqueName: \"kubernetes.io/projected/3c4d4b3d-2929-4563-ac6a-18d5f01c0db6-kube-api-access-79zsd\") pod \"e2e-api-b-a43a70-openshift-default-59d5b5dcf-rlkth\" (UID: \"3c4d4b3d-2929-4563-ac6a-18d5f01c0db6\") " pod="openshift-ingress/e2e-api-b-a43a70-openshift-default-59d5b5dcf-rlkth" Oct 31 04:21:16.776433 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:16.776101 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c4d4b3d-2929-4563-ac6a-18d5f01c0db6-istiod-ca-cert\") pod \"e2e-api-b-a43a70-openshift-default-59d5b5dcf-rlkth\" (UID: \"3c4d4b3d-2929-4563-ac6a-18d5f01c0db6\") " pod="openshift-ingress/e2e-api-b-a43a70-openshift-default-59d5b5dcf-rlkth" Oct 31 04:21:16.776433 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:16.776146 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c4d4b3d-2929-4563-ac6a-18d5f01c0db6-workload-certs\") pod \"e2e-api-b-a43a70-openshift-default-59d5b5dcf-rlkth\" (UID: \"3c4d4b3d-2929-4563-ac6a-18d5f01c0db6\") " pod="openshift-ingress/e2e-api-b-a43a70-openshift-default-59d5b5dcf-rlkth" Oct 31 04:21:16.776433 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:16.776208 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c4d4b3d-2929-4563-ac6a-18d5f01c0db6-credential-socket\") pod \"e2e-api-b-a43a70-openshift-default-59d5b5dcf-rlkth\" (UID: \"3c4d4b3d-2929-4563-ac6a-18d5f01c0db6\") " pod="openshift-ingress/e2e-api-b-a43a70-openshift-default-59d5b5dcf-rlkth" Oct 31 04:21:16.776433 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:16.776239 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c4d4b3d-2929-4563-ac6a-18d5f01c0db6-istio-data\") pod \"e2e-api-b-a43a70-openshift-default-59d5b5dcf-rlkth\" (UID: \"3c4d4b3d-2929-4563-ac6a-18d5f01c0db6\") " pod="openshift-ingress/e2e-api-b-a43a70-openshift-default-59d5b5dcf-rlkth" Oct 31 04:21:16.919827 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:16.883639 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c4d4b3d-2929-4563-ac6a-18d5f01c0db6-istio-envoy\") pod \"e2e-api-b-a43a70-openshift-default-59d5b5dcf-rlkth\" (UID: \"3c4d4b3d-2929-4563-ac6a-18d5f01c0db6\") " pod="openshift-ingress/e2e-api-b-a43a70-openshift-default-59d5b5dcf-rlkth" Oct 31 04:21:16.919827 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:16.883705 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c4d4b3d-2929-4563-ac6a-18d5f01c0db6-istio-token\") pod \"e2e-api-b-a43a70-openshift-default-59d5b5dcf-rlkth\" (UID: \"3c4d4b3d-2929-4563-ac6a-18d5f01c0db6\") " pod="openshift-ingress/e2e-api-b-a43a70-openshift-default-59d5b5dcf-rlkth" Oct 31 04:21:16.919827 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:16.883799 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c4d4b3d-2929-4563-ac6a-18d5f01c0db6-workload-socket\") pod \"e2e-api-b-a43a70-openshift-default-59d5b5dcf-rlkth\" (UID: \"3c4d4b3d-2929-4563-ac6a-18d5f01c0db6\") " pod="openshift-ingress/e2e-api-b-a43a70-openshift-default-59d5b5dcf-rlkth" Oct 31 04:21:16.919827 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:16.883902 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c4d4b3d-2929-4563-ac6a-18d5f01c0db6-istio-podinfo\") pod \"e2e-api-b-a43a70-openshift-default-59d5b5dcf-rlkth\" (UID: \"3c4d4b3d-2929-4563-ac6a-18d5f01c0db6\") " pod="openshift-ingress/e2e-api-b-a43a70-openshift-default-59d5b5dcf-rlkth" Oct 31 04:21:16.919827 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:16.883945 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79zsd\" (UniqueName: \"kubernetes.io/projected/3c4d4b3d-2929-4563-ac6a-18d5f01c0db6-kube-api-access-79zsd\") pod \"e2e-api-b-a43a70-openshift-default-59d5b5dcf-rlkth\" (UID: \"3c4d4b3d-2929-4563-ac6a-18d5f01c0db6\") " pod="openshift-ingress/e2e-api-b-a43a70-openshift-default-59d5b5dcf-rlkth" Oct 31 04:21:16.919827 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:16.883981 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c4d4b3d-2929-4563-ac6a-18d5f01c0db6-istiod-ca-cert\") pod \"e2e-api-b-a43a70-openshift-default-59d5b5dcf-rlkth\" (UID: \"3c4d4b3d-2929-4563-ac6a-18d5f01c0db6\") " pod="openshift-ingress/e2e-api-b-a43a70-openshift-default-59d5b5dcf-rlkth" Oct 31 04:21:16.919827 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:16.884026 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c4d4b3d-2929-4563-ac6a-18d5f01c0db6-workload-certs\") pod \"e2e-api-b-a43a70-openshift-default-59d5b5dcf-rlkth\" (UID: \"3c4d4b3d-2929-4563-ac6a-18d5f01c0db6\") " pod="openshift-ingress/e2e-api-b-a43a70-openshift-default-59d5b5dcf-rlkth" Oct 31 04:21:16.919827 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:16.884088 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c4d4b3d-2929-4563-ac6a-18d5f01c0db6-credential-socket\") pod \"e2e-api-b-a43a70-openshift-default-59d5b5dcf-rlkth\" (UID: \"3c4d4b3d-2929-4563-ac6a-18d5f01c0db6\") " pod="openshift-ingress/e2e-api-b-a43a70-openshift-default-59d5b5dcf-rlkth" Oct 31 04:21:16.919827 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:16.884116 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c4d4b3d-2929-4563-ac6a-18d5f01c0db6-istio-data\") pod \"e2e-api-b-a43a70-openshift-default-59d5b5dcf-rlkth\" (UID: \"3c4d4b3d-2929-4563-ac6a-18d5f01c0db6\") " pod="openshift-ingress/e2e-api-b-a43a70-openshift-default-59d5b5dcf-rlkth" Oct 31 04:21:16.919827 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:16.884751 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c4d4b3d-2929-4563-ac6a-18d5f01c0db6-istio-data\") pod \"e2e-api-b-a43a70-openshift-default-59d5b5dcf-rlkth\" (UID: \"3c4d4b3d-2929-4563-ac6a-18d5f01c0db6\") " pod="openshift-ingress/e2e-api-b-a43a70-openshift-default-59d5b5dcf-rlkth" Oct 31 04:21:16.930026 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:16.924644 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c4d4b3d-2929-4563-ac6a-18d5f01c0db6-workload-certs\") pod \"e2e-api-b-a43a70-openshift-default-59d5b5dcf-rlkth\" (UID: \"3c4d4b3d-2929-4563-ac6a-18d5f01c0db6\") " pod="openshift-ingress/e2e-api-b-a43a70-openshift-default-59d5b5dcf-rlkth" Oct 31 04:21:16.930026 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:16.925301 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c4d4b3d-2929-4563-ac6a-18d5f01c0db6-istiod-ca-cert\") pod \"e2e-api-b-a43a70-openshift-default-59d5b5dcf-rlkth\" (UID: \"3c4d4b3d-2929-4563-ac6a-18d5f01c0db6\") " pod="openshift-ingress/e2e-api-b-a43a70-openshift-default-59d5b5dcf-rlkth" Oct 31 04:21:16.930026 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:16.925644 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c4d4b3d-2929-4563-ac6a-18d5f01c0db6-istio-envoy\") pod \"e2e-api-b-a43a70-openshift-default-59d5b5dcf-rlkth\" (UID: \"3c4d4b3d-2929-4563-ac6a-18d5f01c0db6\") " pod="openshift-ingress/e2e-api-b-a43a70-openshift-default-59d5b5dcf-rlkth" Oct 31 04:21:16.978180 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:16.978097 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c4d4b3d-2929-4563-ac6a-18d5f01c0db6-credential-socket\") pod \"e2e-api-b-a43a70-openshift-default-59d5b5dcf-rlkth\" (UID: \"3c4d4b3d-2929-4563-ac6a-18d5f01c0db6\") " pod="openshift-ingress/e2e-api-b-a43a70-openshift-default-59d5b5dcf-rlkth" Oct 31 04:21:16.979366 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:16.978949 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c4d4b3d-2929-4563-ac6a-18d5f01c0db6-workload-socket\") pod \"e2e-api-b-a43a70-openshift-default-59d5b5dcf-rlkth\" (UID: \"3c4d4b3d-2929-4563-ac6a-18d5f01c0db6\") " pod="openshift-ingress/e2e-api-b-a43a70-openshift-default-59d5b5dcf-rlkth" Oct 31 04:21:17.018991 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:17.018182 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79zsd\" (UniqueName: \"kubernetes.io/projected/3c4d4b3d-2929-4563-ac6a-18d5f01c0db6-kube-api-access-79zsd\") pod \"e2e-api-b-a43a70-openshift-default-59d5b5dcf-rlkth\" (UID: \"3c4d4b3d-2929-4563-ac6a-18d5f01c0db6\") " pod="openshift-ingress/e2e-api-b-a43a70-openshift-default-59d5b5dcf-rlkth" Oct 31 04:21:17.054347 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:17.053971 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c4d4b3d-2929-4563-ac6a-18d5f01c0db6-istio-token\") pod \"e2e-api-b-a43a70-openshift-default-59d5b5dcf-rlkth\" (UID: \"3c4d4b3d-2929-4563-ac6a-18d5f01c0db6\") " pod="openshift-ingress/e2e-api-b-a43a70-openshift-default-59d5b5dcf-rlkth" Oct 31 04:21:17.062566 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:17.062497 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c4d4b3d-2929-4563-ac6a-18d5f01c0db6-istio-podinfo\") pod \"e2e-api-b-a43a70-openshift-default-59d5b5dcf-rlkth\" (UID: \"3c4d4b3d-2929-4563-ac6a-18d5f01c0db6\") " pod="openshift-ingress/e2e-api-b-a43a70-openshift-default-59d5b5dcf-rlkth" Oct 31 04:21:17.315846 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:17.315759 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-a43a70-openshift-default-59d5b5dcf-rlkth" Oct 31 04:21:17.369641 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:17.369603 4376 patch_prober.go:29] interesting pod/e2e-worker-w3-4c2387-openshift-default-5b68d44445-k2jb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Oct 31 04:21:17.369815 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:17.369666 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-k2jb2" podUID="716d4a99-414f-449d-89c2-fa06e5751c60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Oct 31 04:21:17.729449 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:17.729418 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-a43a70-openshift-default-59d5b5dcf-rlkth"] Oct 31 04:21:17.752350 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:17.737043 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:21:17.752350 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:17.737145 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:21:17.752350 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:17.737189 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:21:17.943076 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:17.942096 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a43a70-openshift-default-59d5b5dcf-rlkth" event={"ID":"3c4d4b3d-2929-4563-ac6a-18d5f01c0db6","Type":"ContainerStarted","Data":"0bad7043dcb2db1128e5638cd787770f06bb54cbe2f6ed979fd9b483785e4360"} Oct 31 04:21:17.943076 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:17.942165 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a43a70-openshift-default-59d5b5dcf-rlkth" event={"ID":"3c4d4b3d-2929-4563-ac6a-18d5f01c0db6","Type":"ContainerStarted","Data":"497d35c488a8cee933855c318627762b77d2fb083d9791b6497041826c19b741"} Oct 31 04:21:18.327401 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:18.321268 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-a43a70-openshift-default-59d5b5dcf-rlkth" Oct 31 04:21:18.327401 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:18.325443 4376 patch_prober.go:29] interesting pod/e2e-api-b-a43a70-openshift-default-59d5b5dcf-rlkth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Oct 31 04:21:18.327401 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:18.325497 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a43a70-openshift-default-59d5b5dcf-rlkth" podUID="3c4d4b3d-2929-4563-ac6a-18d5f01c0db6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Oct 31 04:21:19.104598 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:19.103509 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-k2jb2" Oct 31 04:21:19.104598 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:19.104468 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-k2jb2" Oct 31 04:21:19.105777 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:19.105755 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-k2jb2" Oct 31 04:21:19.176026 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:19.175969 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-a43a70-openshift-default-59d5b5dcf-rlkth" podStartSLOduration=3.175955184 podStartE2EDuration="3.175955184s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:21:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:21:18.003303621 +0000 UTC m=+1025.717638239" watchObservedRunningTime="2026-10-31 04:21:19.175955184 +0000 UTC m=+1026.890289801" Oct 31 04:21:19.321629 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:19.321593 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-a43a70-openshift-default-59d5b5dcf-rlkth" Oct 31 04:21:19.464530 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:19.464485 4376 patch_prober.go:29] interesting pod/e2e-api-a-9883e4-openshift-default-5666f48c8c-g9v2t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 04:21:19.464741 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:19.464554 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-9883e4-openshift-default-5666f48c8c-g9v2t" podUID="5c611e14-54b1-42de-b10d-faf45c6f6c04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 04:21:20.003872 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:20.000299 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-a43a70-openshift-default-59d5b5dcf-rlkth" Oct 31 04:21:20.003872 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:20.001837 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-a43a70-openshift-default-59d5b5dcf-rlkth" Oct 31 04:21:22.019143 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:22.019111 4376 generic.go:356] "Generic (PLEG): container finished" podID="5ab2d121-68d1-4cc1-9f0a-96755f3782f4" containerID="98e53ff19986f018955a4600a3874f577718448d591b945669c50655a96d8981" exitCode=0 Oct 31 04:21:22.019701 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:22.019185 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-8c0086-openshift-default-58b446d58c-cdgtp" event={"ID":"5ab2d121-68d1-4cc1-9f0a-96755f3782f4","Type":"ContainerDied","Data":"98e53ff19986f018955a4600a3874f577718448d591b945669c50655a96d8981"} Oct 31 04:21:22.019773 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:22.019699 4376 scope.go:122] "RemoveContainer" containerID="98e53ff19986f018955a4600a3874f577718448d591b945669c50655a96d8981" Oct 31 04:21:22.021083 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:22.021058 4376 generic.go:356] "Generic (PLEG): container finished" podID="5c611e14-54b1-42de-b10d-faf45c6f6c04" containerID="a5402ceb186f4948fa5368278a004b1f1ae679eea48803c4f687e9fba6d03f26" exitCode=0 Oct 31 04:21:22.021188 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:22.021080 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9883e4-openshift-default-5666f48c8c-g9v2t" event={"ID":"5c611e14-54b1-42de-b10d-faf45c6f6c04","Type":"ContainerDied","Data":"a5402ceb186f4948fa5368278a004b1f1ae679eea48803c4f687e9fba6d03f26"} Oct 31 04:21:22.021403 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:22.021375 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:21:22.021497 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:22.021449 4376 scope.go:122] "RemoveContainer" containerID="a5402ceb186f4948fa5368278a004b1f1ae679eea48803c4f687e9fba6d03f26" Oct 31 04:21:22.021538 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:22.021447 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:21:22.021585 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:22.021568 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:21:22.022869 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:22.022835 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:21:22.022990 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:22.022919 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:21:22.022990 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:22.022971 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:21:22.564853 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:22.564813 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a43a70-85d6b96975-wzq5w"] Oct 31 04:21:22.567755 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:22.567735 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a43a70-85d6b96975-wzq5w" Oct 31 04:21:22.583077 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:22.583055 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-a43a70\"" Oct 31 04:21:22.583200 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:22.583177 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-a43a70-dockercfg-8mx9p\"" Oct 31 04:21:22.589673 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:22.589651 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a43a70-85d6b96975-wzq5w"] Oct 31 04:21:22.621896 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:22.621801 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92fkn\" (UniqueName: \"kubernetes.io/projected/830c4180-0a49-43d9-bfba-ed9f0cadc4ec-kube-api-access-92fkn\") pod \"payload-pre-processing-e2e-api-b-a43a70-85d6b96975-wzq5w\" (UID: \"830c4180-0a49-43d9-bfba-ed9f0cadc4ec\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a43a70-85d6b96975-wzq5w" Oct 31 04:21:22.622053 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:22.621936 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/830c4180-0a49-43d9-bfba-ed9f0cadc4ec-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-a43a70-85d6b96975-wzq5w\" (UID: \"830c4180-0a49-43d9-bfba-ed9f0cadc4ec\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a43a70-85d6b96975-wzq5w" Oct 31 04:21:22.735354 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:22.731299 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/830c4180-0a49-43d9-bfba-ed9f0cadc4ec-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-a43a70-85d6b96975-wzq5w\" (UID: \"830c4180-0a49-43d9-bfba-ed9f0cadc4ec\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a43a70-85d6b96975-wzq5w" Oct 31 04:21:22.763841 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:22.763738 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92fkn\" (UniqueName: \"kubernetes.io/projected/830c4180-0a49-43d9-bfba-ed9f0cadc4ec-kube-api-access-92fkn\") pod \"payload-pre-processing-e2e-api-b-a43a70-85d6b96975-wzq5w\" (UID: \"830c4180-0a49-43d9-bfba-ed9f0cadc4ec\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a43a70-85d6b96975-wzq5w" Oct 31 04:21:22.764341 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:22.743446 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/830c4180-0a49-43d9-bfba-ed9f0cadc4ec-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-a43a70-85d6b96975-wzq5w\" (UID: \"830c4180-0a49-43d9-bfba-ed9f0cadc4ec\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a43a70-85d6b96975-wzq5w" Oct 31 04:21:22.888370 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:22.877495 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92fkn\" (UniqueName: \"kubernetes.io/projected/830c4180-0a49-43d9-bfba-ed9f0cadc4ec-kube-api-access-92fkn\") pod \"payload-pre-processing-e2e-api-b-a43a70-85d6b96975-wzq5w\" (UID: \"830c4180-0a49-43d9-bfba-ed9f0cadc4ec\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a43a70-85d6b96975-wzq5w" Oct 31 04:21:23.044402 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:23.043973 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-8c0086-openshift-default-58b446d58c-cdgtp" event={"ID":"5ab2d121-68d1-4cc1-9f0a-96755f3782f4","Type":"ContainerStarted","Data":"ceff6d0d5438bcf960823438a5d305ba8bbeef8ca1d142810f2ae29203e28e13"} Oct 31 04:21:23.053350 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:23.048174 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9883e4-openshift-default-5666f48c8c-g9v2t" event={"ID":"5c611e14-54b1-42de-b10d-faf45c6f6c04","Type":"ContainerStarted","Data":"977457ac069fad712b7ab6c7b61449c2b284a0f1b2f05c07e244bf9cfe15130a"} Oct 31 04:21:23.189576 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:23.181740 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a43a70-85d6b96975-wzq5w" Oct 31 04:21:23.418872 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:23.418829 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-8c0086-openshift-default-58b446d58c-cdgtp" Oct 31 04:21:23.419071 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:23.418940 4376 patch_prober.go:29] interesting pod/e2e-api-a-8c0086-openshift-default-58b446d58c-cdgtp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Oct 31 04:21:23.419071 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:23.419002 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-8c0086-openshift-default-58b446d58c-cdgtp" podUID="5ab2d121-68d1-4cc1-9f0a-96755f3782f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Oct 31 04:21:23.506406 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:21:23.506377 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod830c4180_0a49_43d9_bfba_ed9f0cadc4ec.slice/crio-e895ee89a75dacb1625ba7b2f71e821f2c1906224bbd4da1eb1f33c5c230db68 WatchSource:0}: Error finding container e895ee89a75dacb1625ba7b2f71e821f2c1906224bbd4da1eb1f33c5c230db68: Status 404 returned error can't find the container with id e895ee89a75dacb1625ba7b2f71e821f2c1906224bbd4da1eb1f33c5c230db68 Oct 31 04:21:23.509829 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:23.509807 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a43a70-85d6b96975-wzq5w"] Oct 31 04:21:23.932774 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:23.932734 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-9883e4-openshift-default-5666f48c8c-g9v2t" Oct 31 04:21:23.932954 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:23.932912 4376 patch_prober.go:29] interesting pod/e2e-api-a-9883e4-openshift-default-5666f48c8c-g9v2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Oct 31 04:21:23.933012 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:23.932969 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9883e4-openshift-default-5666f48c8c-g9v2t" podUID="5c611e14-54b1-42de-b10d-faf45c6f6c04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Oct 31 04:21:24.090637 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:24.090091 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a43a70-85d6b96975-wzq5w" event={"ID":"830c4180-0a49-43d9-bfba-ed9f0cadc4ec","Type":"ContainerStarted","Data":"32730010562cfd5e7cb8d26be447868d0f9537b6a44b4f2a346b6f0d9eca0986"} Oct 31 04:21:24.090637 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:24.090168 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a43a70-85d6b96975-wzq5w" event={"ID":"830c4180-0a49-43d9-bfba-ed9f0cadc4ec","Type":"ContainerStarted","Data":"e895ee89a75dacb1625ba7b2f71e821f2c1906224bbd4da1eb1f33c5c230db68"} Oct 31 04:21:24.096368 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:24.091701 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a43a70-85d6b96975-wzq5w" Oct 31 04:21:24.135123 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:24.135067 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a43a70-85d6b96975-wzq5w" podStartSLOduration=2.135051972 podStartE2EDuration="2.135051972s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:21:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:21:24.132468133 +0000 UTC m=+1031.846802773" watchObservedRunningTime="2026-10-31 04:21:24.135051972 +0000 UTC m=+1031.849386589" Oct 31 04:21:24.422348 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:24.421527 4376 patch_prober.go:29] interesting pod/e2e-api-a-8c0086-openshift-default-58b446d58c-cdgtp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Oct 31 04:21:24.422348 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:24.421592 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-8c0086-openshift-default-58b446d58c-cdgtp" podUID="5ab2d121-68d1-4cc1-9f0a-96755f3782f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Oct 31 04:21:24.936786 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:24.936749 4376 patch_prober.go:29] interesting pod/e2e-api-a-9883e4-openshift-default-5666f48c8c-g9v2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Oct 31 04:21:24.937060 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:24.937033 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9883e4-openshift-default-5666f48c8c-g9v2t" podUID="5c611e14-54b1-42de-b10d-faf45c6f6c04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Oct 31 04:21:25.419458 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:25.419399 4376 patch_prober.go:29] interesting pod/e2e-api-a-8c0086-openshift-default-58b446d58c-cdgtp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Oct 31 04:21:25.419458 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:25.419450 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-8c0086-openshift-default-58b446d58c-cdgtp" podUID="5ab2d121-68d1-4cc1-9f0a-96755f3782f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Oct 31 04:21:25.933028 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:25.932996 4376 patch_prober.go:29] interesting pod/e2e-api-a-9883e4-openshift-default-5666f48c8c-g9v2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Oct 31 04:21:25.933206 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:25.933050 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9883e4-openshift-default-5666f48c8c-g9v2t" podUID="5c611e14-54b1-42de-b10d-faf45c6f6c04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Oct 31 04:21:26.425362 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:26.423703 4376 patch_prober.go:29] interesting pod/e2e-api-a-8c0086-openshift-default-58b446d58c-cdgtp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Oct 31 04:21:26.425362 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:26.423766 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-8c0086-openshift-default-58b446d58c-cdgtp" podUID="5ab2d121-68d1-4cc1-9f0a-96755f3782f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Oct 31 04:21:26.933347 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:26.933302 4376 patch_prober.go:29] interesting pod/e2e-api-a-9883e4-openshift-default-5666f48c8c-g9v2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Oct 31 04:21:26.933528 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:26.933374 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9883e4-openshift-default-5666f48c8c-g9v2t" podUID="5c611e14-54b1-42de-b10d-faf45c6f6c04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Oct 31 04:21:27.419986 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:27.419943 4376 patch_prober.go:29] interesting pod/e2e-api-a-8c0086-openshift-default-58b446d58c-cdgtp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Oct 31 04:21:27.420201 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:27.419998 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-8c0086-openshift-default-58b446d58c-cdgtp" podUID="5ab2d121-68d1-4cc1-9f0a-96755f3782f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Oct 31 04:21:27.951366 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:27.951308 4376 patch_prober.go:29] interesting pod/e2e-api-a-9883e4-openshift-default-5666f48c8c-g9v2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Oct 31 04:21:27.952342 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:27.951921 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9883e4-openshift-default-5666f48c8c-g9v2t" podUID="5c611e14-54b1-42de-b10d-faf45c6f6c04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Oct 31 04:21:28.527804 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:28.527748 4376 patch_prober.go:29] interesting pod/e2e-api-a-8c0086-openshift-default-58b446d58c-cdgtp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Oct 31 04:21:28.528070 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:28.528038 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-8c0086-openshift-default-58b446d58c-cdgtp" podUID="5ab2d121-68d1-4cc1-9f0a-96755f3782f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Oct 31 04:21:28.932794 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:28.932757 4376 patch_prober.go:29] interesting pod/e2e-api-a-9883e4-openshift-default-5666f48c8c-g9v2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Oct 31 04:21:28.932977 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:28.932831 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9883e4-openshift-default-5666f48c8c-g9v2t" podUID="5c611e14-54b1-42de-b10d-faf45c6f6c04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Oct 31 04:21:29.172377 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:29.172345 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-8c0086-openshift-default-58b446d58c-cdgtp" Oct 31 04:21:29.434562 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:29.434525 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-8c0086-openshift-default-58b446d58c-cdgtp" Oct 31 04:21:29.436477 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:29.436442 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-8c0086-openshift-default-58b446d58c-cdgtp" Oct 31 04:21:29.937285 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:29.937255 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-9883e4-openshift-default-5666f48c8c-g9v2t" Oct 31 04:21:29.937702 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:29.937680 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-9883e4-openshift-default-5666f48c8c-g9v2t" Oct 31 04:21:29.938400 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:29.938375 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-9883e4-openshift-default-5666f48c8c-g9v2t" Oct 31 04:21:34.873631 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:34.873599 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-0fc77f-ffb9b967-2fq8h"] Oct 31 04:21:34.877902 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:34.877886 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0fc77f-ffb9b967-2fq8h" Oct 31 04:21:34.896841 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:34.896817 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-0fc77f-ffb9b967-2fq8h"] Oct 31 04:21:34.907374 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:34.904996 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-0fc77f-serving-cert\"" Oct 31 04:21:34.979924 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:34.979885 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4655062f-1974-4c0d-ac7a-ac0abe32991d-metrics-certs\") pod \"maas-api-e2e-api-b-0fc77f-ffb9b967-2fq8h\" (UID: \"4655062f-1974-4c0d-ac7a-ac0abe32991d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0fc77f-ffb9b967-2fq8h" Oct 31 04:21:34.980108 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:34.980004 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scgzq\" (UniqueName: \"kubernetes.io/projected/4655062f-1974-4c0d-ac7a-ac0abe32991d-kube-api-access-scgzq\") pod \"maas-api-e2e-api-b-0fc77f-ffb9b967-2fq8h\" (UID: \"4655062f-1974-4c0d-ac7a-ac0abe32991d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0fc77f-ffb9b967-2fq8h" Oct 31 04:21:34.980279 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:34.980259 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4655062f-1974-4c0d-ac7a-ac0abe32991d-maas-api-tls\") pod \"maas-api-e2e-api-b-0fc77f-ffb9b967-2fq8h\" (UID: \"4655062f-1974-4c0d-ac7a-ac0abe32991d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0fc77f-ffb9b967-2fq8h" Oct 31 04:21:34.980381 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:34.980305 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4655062f-1974-4c0d-ac7a-ac0abe32991d-openshift-service-ca\") pod \"maas-api-e2e-api-b-0fc77f-ffb9b967-2fq8h\" (UID: \"4655062f-1974-4c0d-ac7a-ac0abe32991d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0fc77f-ffb9b967-2fq8h" Oct 31 04:21:35.087317 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:35.087250 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4655062f-1974-4c0d-ac7a-ac0abe32991d-metrics-certs\") pod \"maas-api-e2e-api-b-0fc77f-ffb9b967-2fq8h\" (UID: \"4655062f-1974-4c0d-ac7a-ac0abe32991d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0fc77f-ffb9b967-2fq8h" Oct 31 04:21:35.088255 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:35.088229 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scgzq\" (UniqueName: \"kubernetes.io/projected/4655062f-1974-4c0d-ac7a-ac0abe32991d-kube-api-access-scgzq\") pod \"maas-api-e2e-api-b-0fc77f-ffb9b967-2fq8h\" (UID: \"4655062f-1974-4c0d-ac7a-ac0abe32991d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0fc77f-ffb9b967-2fq8h" Oct 31 04:21:35.089068 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:35.089044 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4655062f-1974-4c0d-ac7a-ac0abe32991d-maas-api-tls\") pod \"maas-api-e2e-api-b-0fc77f-ffb9b967-2fq8h\" (UID: \"4655062f-1974-4c0d-ac7a-ac0abe32991d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0fc77f-ffb9b967-2fq8h" Oct 31 04:21:35.089247 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:35.089227 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4655062f-1974-4c0d-ac7a-ac0abe32991d-openshift-service-ca\") pod \"maas-api-e2e-api-b-0fc77f-ffb9b967-2fq8h\" (UID: \"4655062f-1974-4c0d-ac7a-ac0abe32991d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0fc77f-ffb9b967-2fq8h" Oct 31 04:21:35.092552 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:35.090533 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4655062f-1974-4c0d-ac7a-ac0abe32991d-openshift-service-ca\") pod \"maas-api-e2e-api-b-0fc77f-ffb9b967-2fq8h\" (UID: \"4655062f-1974-4c0d-ac7a-ac0abe32991d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0fc77f-ffb9b967-2fq8h" Oct 31 04:21:35.092552 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:21:35.090637 4376 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-0fc77f-serving-cert: secret "maas-api-e2e-api-b-0fc77f-serving-cert" not found Oct 31 04:21:35.095853 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:35.095827 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4655062f-1974-4c0d-ac7a-ac0abe32991d-metrics-certs\") pod \"maas-api-e2e-api-b-0fc77f-ffb9b967-2fq8h\" (UID: \"4655062f-1974-4c0d-ac7a-ac0abe32991d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0fc77f-ffb9b967-2fq8h" Oct 31 04:21:35.112354 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:21:35.106889 4376 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4655062f-1974-4c0d-ac7a-ac0abe32991d-maas-api-tls podName:4655062f-1974-4c0d-ac7a-ac0abe32991d nodeName:}" failed. No retries permitted until 2026-10-31 04:21:35.606854628 +0000 UTC m=+1043.321189240 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4655062f-1974-4c0d-ac7a-ac0abe32991d-maas-api-tls") pod "maas-api-e2e-api-b-0fc77f-ffb9b967-2fq8h" (UID: "4655062f-1974-4c0d-ac7a-ac0abe32991d") : secret "maas-api-e2e-api-b-0fc77f-serving-cert" not found Oct 31 04:21:35.133131 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:35.123615 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-scgzq\" (UniqueName: \"kubernetes.io/projected/4655062f-1974-4c0d-ac7a-ac0abe32991d-kube-api-access-scgzq\") pod \"maas-api-e2e-api-b-0fc77f-ffb9b967-2fq8h\" (UID: \"4655062f-1974-4c0d-ac7a-ac0abe32991d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0fc77f-ffb9b967-2fq8h" Oct 31 04:21:35.614554 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:35.614475 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4655062f-1974-4c0d-ac7a-ac0abe32991d-maas-api-tls\") pod \"maas-api-e2e-api-b-0fc77f-ffb9b967-2fq8h\" (UID: \"4655062f-1974-4c0d-ac7a-ac0abe32991d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0fc77f-ffb9b967-2fq8h" Oct 31 04:21:35.616883 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:35.616855 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4655062f-1974-4c0d-ac7a-ac0abe32991d-maas-api-tls\") pod \"maas-api-e2e-api-b-0fc77f-ffb9b967-2fq8h\" (UID: \"4655062f-1974-4c0d-ac7a-ac0abe32991d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0fc77f-ffb9b967-2fq8h" Oct 31 04:21:35.797608 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:35.797544 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0fc77f-ffb9b967-2fq8h" Oct 31 04:21:36.109042 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:36.109013 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a43a70-85d6b96975-wzq5w" Oct 31 04:21:36.342163 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:36.342124 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-0fc77f-ffb9b967-2fq8h"] Oct 31 04:21:37.278377 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:37.278313 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0fc77f-ffb9b967-2fq8h" event={"ID":"4655062f-1974-4c0d-ac7a-ac0abe32991d","Type":"ContainerStarted","Data":"9bb348a99dfc8453fa69059d5f833d600b154d5411f4a3b601ca8b3c74d4f1de"} Oct 31 04:21:37.278761 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:37.278396 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0fc77f-ffb9b967-2fq8h" event={"ID":"4655062f-1974-4c0d-ac7a-ac0abe32991d","Type":"ContainerStarted","Data":"e6ba015e0ed5e34990a7cfc92934038a94baf6736524df736052aa09bdcf174d"} Oct 31 04:21:37.278761 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:37.278474 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0fc77f-ffb9b967-2fq8h" Oct 31 04:21:37.323931 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:37.323885 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0fc77f-ffb9b967-2fq8h" podStartSLOduration=2.847054192 podStartE2EDuration="3.323871267s" totalImagesPullingTime="476.817075ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:21:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:21:37.32098808 +0000 UTC m=+1045.035322697" watchObservedRunningTime="2026-10-31 04:21:37.323871267 +0000 UTC m=+1045.038205884" Oct 31 04:21:38.972522 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:38.972481 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-n6t9x"] Oct 31 04:21:38.979357 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:38.978555 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-n6t9x" Oct 31 04:21:39.007188 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:39.007153 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31c92206-bebe-492e-be35-a008408e1e86-istio-token\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-n6t9x\" (UID: \"31c92206-bebe-492e-be35-a008408e1e86\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-n6t9x" Oct 31 04:21:39.007408 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:39.007394 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31c92206-bebe-492e-be35-a008408e1e86-istio-podinfo\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-n6t9x\" (UID: \"31c92206-bebe-492e-be35-a008408e1e86\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-n6t9x" Oct 31 04:21:39.007577 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:39.007564 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31c92206-bebe-492e-be35-a008408e1e86-workload-certs\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-n6t9x\" (UID: \"31c92206-bebe-492e-be35-a008408e1e86\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-n6t9x" Oct 31 04:21:39.007666 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:39.007655 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31c92206-bebe-492e-be35-a008408e1e86-credential-socket\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-n6t9x\" (UID: \"31c92206-bebe-492e-be35-a008408e1e86\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-n6t9x" Oct 31 04:21:39.007826 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:39.007814 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31c92206-bebe-492e-be35-a008408e1e86-istio-data\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-n6t9x\" (UID: \"31c92206-bebe-492e-be35-a008408e1e86\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-n6t9x" Oct 31 04:21:39.007971 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:39.007962 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4dt9\" (UniqueName: \"kubernetes.io/projected/31c92206-bebe-492e-be35-a008408e1e86-kube-api-access-v4dt9\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-n6t9x\" (UID: \"31c92206-bebe-492e-be35-a008408e1e86\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-n6t9x" Oct 31 04:21:39.008040 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:39.008032 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31c92206-bebe-492e-be35-a008408e1e86-workload-socket\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-n6t9x\" (UID: \"31c92206-bebe-492e-be35-a008408e1e86\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-n6t9x" Oct 31 04:21:39.008113 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:39.008105 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31c92206-bebe-492e-be35-a008408e1e86-istiod-ca-cert\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-n6t9x\" (UID: \"31c92206-bebe-492e-be35-a008408e1e86\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-n6t9x" Oct 31 04:21:39.008183 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:39.008174 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31c92206-bebe-492e-be35-a008408e1e86-istio-envoy\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-n6t9x\" (UID: \"31c92206-bebe-492e-be35-a008408e1e86\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-n6t9x" Oct 31 04:21:39.015766 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:39.015735 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-n6t9x"] Oct 31 04:21:39.109555 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:39.109526 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31c92206-bebe-492e-be35-a008408e1e86-istio-envoy\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-n6t9x\" (UID: \"31c92206-bebe-492e-be35-a008408e1e86\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-n6t9x" Oct 31 04:21:39.109738 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:39.109613 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31c92206-bebe-492e-be35-a008408e1e86-istio-token\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-n6t9x\" (UID: \"31c92206-bebe-492e-be35-a008408e1e86\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-n6t9x" Oct 31 04:21:39.109738 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:39.109699 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31c92206-bebe-492e-be35-a008408e1e86-istio-podinfo\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-n6t9x\" (UID: \"31c92206-bebe-492e-be35-a008408e1e86\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-n6t9x" Oct 31 04:21:39.109738 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:39.109725 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31c92206-bebe-492e-be35-a008408e1e86-workload-certs\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-n6t9x\" (UID: \"31c92206-bebe-492e-be35-a008408e1e86\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-n6t9x" Oct 31 04:21:39.109918 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:39.109746 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31c92206-bebe-492e-be35-a008408e1e86-credential-socket\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-n6t9x\" (UID: \"31c92206-bebe-492e-be35-a008408e1e86\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-n6t9x" Oct 31 04:21:39.109918 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:39.109829 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31c92206-bebe-492e-be35-a008408e1e86-istio-data\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-n6t9x\" (UID: \"31c92206-bebe-492e-be35-a008408e1e86\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-n6t9x" Oct 31 04:21:39.110015 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:39.109968 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4dt9\" (UniqueName: \"kubernetes.io/projected/31c92206-bebe-492e-be35-a008408e1e86-kube-api-access-v4dt9\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-n6t9x\" (UID: \"31c92206-bebe-492e-be35-a008408e1e86\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-n6t9x" Oct 31 04:21:39.110015 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:39.110002 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31c92206-bebe-492e-be35-a008408e1e86-workload-socket\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-n6t9x\" (UID: \"31c92206-bebe-492e-be35-a008408e1e86\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-n6t9x" Oct 31 04:21:39.110117 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:39.110043 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31c92206-bebe-492e-be35-a008408e1e86-istiod-ca-cert\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-n6t9x\" (UID: \"31c92206-bebe-492e-be35-a008408e1e86\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-n6t9x" Oct 31 04:21:39.110117 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:39.110099 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31c92206-bebe-492e-be35-a008408e1e86-workload-certs\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-n6t9x\" (UID: \"31c92206-bebe-492e-be35-a008408e1e86\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-n6t9x" Oct 31 04:21:39.110244 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:39.110219 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31c92206-bebe-492e-be35-a008408e1e86-istio-data\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-n6t9x\" (UID: \"31c92206-bebe-492e-be35-a008408e1e86\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-n6t9x" Oct 31 04:21:39.110316 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:39.110294 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31c92206-bebe-492e-be35-a008408e1e86-workload-socket\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-n6t9x\" (UID: \"31c92206-bebe-492e-be35-a008408e1e86\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-n6t9x" Oct 31 04:21:39.110500 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:39.110481 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31c92206-bebe-492e-be35-a008408e1e86-credential-socket\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-n6t9x\" (UID: \"31c92206-bebe-492e-be35-a008408e1e86\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-n6t9x" Oct 31 04:21:39.110716 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:39.110698 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31c92206-bebe-492e-be35-a008408e1e86-istiod-ca-cert\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-n6t9x\" (UID: \"31c92206-bebe-492e-be35-a008408e1e86\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-n6t9x" Oct 31 04:21:39.111823 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:39.111802 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31c92206-bebe-492e-be35-a008408e1e86-istio-envoy\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-n6t9x\" (UID: \"31c92206-bebe-492e-be35-a008408e1e86\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-n6t9x" Oct 31 04:21:39.112388 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:39.112371 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31c92206-bebe-492e-be35-a008408e1e86-istio-podinfo\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-n6t9x\" (UID: \"31c92206-bebe-492e-be35-a008408e1e86\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-n6t9x" Oct 31 04:21:39.133384 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:39.133357 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4dt9\" (UniqueName: \"kubernetes.io/projected/31c92206-bebe-492e-be35-a008408e1e86-kube-api-access-v4dt9\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-n6t9x\" (UID: \"31c92206-bebe-492e-be35-a008408e1e86\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-n6t9x" Oct 31 04:21:39.133598 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:39.133565 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31c92206-bebe-492e-be35-a008408e1e86-istio-token\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-n6t9x\" (UID: \"31c92206-bebe-492e-be35-a008408e1e86\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-n6t9x" Oct 31 04:21:39.307140 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:39.307064 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-n6t9x" Oct 31 04:21:39.622154 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:21:39.622124 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31c92206_bebe_492e_be35_a008408e1e86.slice/crio-2281b255574b50e80fa3bfef1d986e2019efb566bd8a3e7cb76c4191bbb8c3c5 WatchSource:0}: Error finding container 2281b255574b50e80fa3bfef1d986e2019efb566bd8a3e7cb76c4191bbb8c3c5: Status 404 returned error can't find the container with id 2281b255574b50e80fa3bfef1d986e2019efb566bd8a3e7cb76c4191bbb8c3c5 Oct 31 04:21:39.624506 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:39.624484 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-n6t9x"] Oct 31 04:21:39.624774 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:39.624743 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:21:39.624847 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:39.624835 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:21:39.624882 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:39.624867 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:21:40.298478 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:40.298443 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-n6t9x" event={"ID":"31c92206-bebe-492e-be35-a008408e1e86","Type":"ContainerStarted","Data":"379565cdf4008e1cecc097ff692ae676392277493bbda4f557b86a7536232897"} Oct 31 04:21:40.298869 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:40.298496 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-n6t9x" event={"ID":"31c92206-bebe-492e-be35-a008408e1e86","Type":"ContainerStarted","Data":"2281b255574b50e80fa3bfef1d986e2019efb566bd8a3e7cb76c4191bbb8c3c5"} Oct 31 04:21:40.308251 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:40.308227 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-n6t9x" Oct 31 04:21:40.310051 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:40.310026 4376 patch_prober.go:29] interesting pod/e2e-worker-w3-4c2387-openshift-default-5b68d44445-n6t9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Oct 31 04:21:40.310163 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:40.310081 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-n6t9x" podUID="31c92206-bebe-492e-be35-a008408e1e86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Oct 31 04:21:40.386431 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:40.379146 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-n6t9x" podStartSLOduration=2.379124058 podStartE2EDuration="2.379124058s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:21:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:21:40.358908214 +0000 UTC m=+1048.073242850" watchObservedRunningTime="2026-10-31 04:21:40.379124058 +0000 UTC m=+1048.093458676" Oct 31 04:21:41.003093 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:21:41.003058 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 04:21:41.308140 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:41.308061 4376 patch_prober.go:29] interesting pod/e2e-worker-w3-4c2387-openshift-default-5b68d44445-n6t9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Oct 31 04:21:41.308140 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:41.308109 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-n6t9x" podUID="31c92206-bebe-492e-be35-a008408e1e86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Oct 31 04:21:42.307463 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:42.307430 4376 patch_prober.go:29] interesting pod/e2e-worker-w3-4c2387-openshift-default-5b68d44445-n6t9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Oct 31 04:21:42.307682 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:42.307501 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-n6t9x" podUID="31c92206-bebe-492e-be35-a008408e1e86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Oct 31 04:21:43.289906 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:43.289875 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0fc77f-ffb9b967-2fq8h" Oct 31 04:21:43.307433 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:43.307397 4376 patch_prober.go:29] interesting pod/e2e-worker-w3-4c2387-openshift-default-5b68d44445-n6t9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Oct 31 04:21:43.307580 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:43.307460 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-n6t9x" podUID="31c92206-bebe-492e-be35-a008408e1e86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Oct 31 04:21:44.307413 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:44.307377 4376 patch_prober.go:29] interesting pod/e2e-worker-w3-4c2387-openshift-default-5b68d44445-n6t9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Oct 31 04:21:44.307833 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:44.307463 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-n6t9x" podUID="31c92206-bebe-492e-be35-a008408e1e86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Oct 31 04:21:45.308404 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:45.308368 4376 patch_prober.go:29] interesting pod/e2e-worker-w3-4c2387-openshift-default-5b68d44445-n6t9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Oct 31 04:21:45.308962 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:45.308426 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-n6t9x" podUID="31c92206-bebe-492e-be35-a008408e1e86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Oct 31 04:21:46.307872 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:46.307836 4376 patch_prober.go:29] interesting pod/e2e-worker-w3-4c2387-openshift-default-5b68d44445-n6t9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Oct 31 04:21:46.308051 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:46.307888 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-n6t9x" podUID="31c92206-bebe-492e-be35-a008408e1e86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Oct 31 04:21:47.311947 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:47.311911 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-n6t9x" Oct 31 04:21:47.312364 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:47.312306 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-n6t9x" Oct 31 04:21:47.313001 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:47.312976 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-n6t9x" Oct 31 04:21:47.775813 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:21:47.775776 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 04:22:10.203354 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:10.200620 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-8c0086-fd5c48549-k5ltz"] Oct 31 04:22:11.091741 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:22:11.091703 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 04:22:14.394042 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:14.394005 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 04:22:15.205352 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:15.204449 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-8c0086-fd5c48549-k5ltz" podUID="9de59006-1b3d-48f4-a990-a51968da6a23" containerName="payload-processing" containerID="cri-o://14a6dbaf540610df7d6709f2591b0f2830255ba50682869ea00d3a5d58840c4a" gracePeriod=25 Oct 31 04:22:15.838349 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:15.836577 4376 generic.go:356] "Generic (PLEG): container finished" podID="9de59006-1b3d-48f4-a990-a51968da6a23" containerID="14a6dbaf540610df7d6709f2591b0f2830255ba50682869ea00d3a5d58840c4a" exitCode=0 Oct 31 04:22:15.838349 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:15.836640 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-8c0086-fd5c48549-k5ltz" event={"ID":"9de59006-1b3d-48f4-a990-a51968da6a23","Type":"ContainerDied","Data":"14a6dbaf540610df7d6709f2591b0f2830255ba50682869ea00d3a5d58840c4a"} Oct 31 04:22:15.852070 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:15.851517 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-8c0086-fd5c48549-k5ltz" Oct 31 04:22:15.977960 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:15.974814 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9de59006-1b3d-48f4-a990-a51968da6a23-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9de59006-1b3d-48f4-a990-a51968da6a23-plugins-config-volume\") pod \"9de59006-1b3d-48f4-a990-a51968da6a23\" (UID: \"9de59006-1b3d-48f4-a990-a51968da6a23\") " Oct 31 04:22:15.977960 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:15.975039 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9de59006-1b3d-48f4-a990-a51968da6a23-kube-api-access-jgw85\" (UniqueName: \"kubernetes.io/projected/9de59006-1b3d-48f4-a990-a51968da6a23-kube-api-access-jgw85\") pod \"9de59006-1b3d-48f4-a990-a51968da6a23\" (UID: \"9de59006-1b3d-48f4-a990-a51968da6a23\") " Oct 31 04:22:15.985618 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:15.985578 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9de59006-1b3d-48f4-a990-a51968da6a23-plugins-config-volume" pod "9de59006-1b3d-48f4-a990-a51968da6a23" (UID: "9de59006-1b3d-48f4-a990-a51968da6a23"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:22:16.223698 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:16.216367 4376 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9de59006-1b3d-48f4-a990-a51968da6a23-plugins-config-volume\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:22:16.224663 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:16.224624 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9de59006-1b3d-48f4-a990-a51968da6a23-kube-api-access-jgw85" pod "9de59006-1b3d-48f4-a990-a51968da6a23" (UID: "9de59006-1b3d-48f4-a990-a51968da6a23"). InnerVolumeSpecName "kube-api-access-jgw85". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:22:16.326775 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:16.326682 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jgw85\" (UniqueName: \"kubernetes.io/projected/9de59006-1b3d-48f4-a990-a51968da6a23-kube-api-access-jgw85\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:22:16.893352 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:16.881967 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-8c0086-fd5c48549-k5ltz" event={"ID":"9de59006-1b3d-48f4-a990-a51968da6a23","Type":"ContainerDied","Data":"878dea6b7da3ee2620e2e6f08baa981267863a333bb5462d9ed744b0b6c3d2c5"} Oct 31 04:22:16.893352 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:16.882042 4376 scope.go:122] "RemoveContainer" containerID="14a6dbaf540610df7d6709f2591b0f2830255ba50682869ea00d3a5d58840c4a" Oct 31 04:22:16.893352 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:16.882219 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-8c0086-fd5c48549-k5ltz" Oct 31 04:22:16.992035 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:16.992003 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-8c0086-fd5c48549-k5ltz"] Oct 31 04:22:17.047279 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:17.047241 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-8c0086-fd5c48549-k5ltz"] Oct 31 04:22:18.470768 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:18.470734 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9de59006-1b3d-48f4-a990-a51968da6a23" path="/var/lib/kubelet/pods/9de59006-1b3d-48f4-a990-a51968da6a23/volumes" Oct 31 04:22:27.027619 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:27.027587 4376 generic.go:356] "Generic (PLEG): container finished" podID="3c4d4b3d-2929-4563-ac6a-18d5f01c0db6" containerID="0bad7043dcb2db1128e5638cd787770f06bb54cbe2f6ed979fd9b483785e4360" exitCode=0 Oct 31 04:22:27.028065 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:27.027660 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a43a70-openshift-default-59d5b5dcf-rlkth" event={"ID":"3c4d4b3d-2929-4563-ac6a-18d5f01c0db6","Type":"ContainerDied","Data":"0bad7043dcb2db1128e5638cd787770f06bb54cbe2f6ed979fd9b483785e4360"} Oct 31 04:22:27.028129 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:27.028112 4376 scope.go:122] "RemoveContainer" containerID="0bad7043dcb2db1128e5638cd787770f06bb54cbe2f6ed979fd9b483785e4360" Oct 31 04:22:27.029724 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:27.029694 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:22:27.029789 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:27.029764 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:22:27.029829 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:27.029798 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:22:28.038169 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:28.038131 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a43a70-openshift-default-59d5b5dcf-rlkth" event={"ID":"3c4d4b3d-2929-4563-ac6a-18d5f01c0db6","Type":"ContainerStarted","Data":"fc901d90a0557a98804f0fa1ac967a1768e5b41c868d4f1ffe70a472e03095d9"} Oct 31 04:22:28.316623 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:28.316536 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-a43a70-openshift-default-59d5b5dcf-rlkth" Oct 31 04:22:28.316826 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:28.316799 4376 patch_prober.go:29] interesting pod/e2e-api-b-a43a70-openshift-default-59d5b5dcf-rlkth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Oct 31 04:22:28.316943 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:28.316915 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a43a70-openshift-default-59d5b5dcf-rlkth" podUID="3c4d4b3d-2929-4563-ac6a-18d5f01c0db6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Oct 31 04:22:29.316222 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:29.316182 4376 patch_prober.go:29] interesting pod/e2e-api-b-a43a70-openshift-default-59d5b5dcf-rlkth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Oct 31 04:22:29.316671 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:29.316236 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a43a70-openshift-default-59d5b5dcf-rlkth" podUID="3c4d4b3d-2929-4563-ac6a-18d5f01c0db6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Oct 31 04:22:30.316784 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:30.316748 4376 patch_prober.go:29] interesting pod/e2e-api-b-a43a70-openshift-default-59d5b5dcf-rlkth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Oct 31 04:22:30.317200 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:30.316811 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a43a70-openshift-default-59d5b5dcf-rlkth" podUID="3c4d4b3d-2929-4563-ac6a-18d5f01c0db6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Oct 31 04:22:31.324353 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:31.323753 4376 patch_prober.go:29] interesting pod/e2e-api-b-a43a70-openshift-default-59d5b5dcf-rlkth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Oct 31 04:22:31.324353 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:31.323829 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a43a70-openshift-default-59d5b5dcf-rlkth" podUID="3c4d4b3d-2929-4563-ac6a-18d5f01c0db6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Oct 31 04:22:32.316208 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:32.316177 4376 patch_prober.go:29] interesting pod/e2e-api-b-a43a70-openshift-default-59d5b5dcf-rlkth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Oct 31 04:22:32.316407 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:32.316228 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a43a70-openshift-default-59d5b5dcf-rlkth" podUID="3c4d4b3d-2929-4563-ac6a-18d5f01c0db6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Oct 31 04:22:33.354181 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:33.354147 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-a43a70-openshift-default-59d5b5dcf-rlkth" Oct 31 04:22:33.354962 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:33.354942 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-a43a70-openshift-default-59d5b5dcf-rlkth" Oct 31 04:22:33.356789 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:33.356757 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-a43a70-openshift-default-59d5b5dcf-rlkth" Oct 31 04:22:34.080315 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:34.080280 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-8c0086-openshift-default-58b446d58c-cdgtp"] Oct 31 04:22:34.080796 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:34.080766 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-8c0086-openshift-default-58b446d58c-cdgtp" podUID="5ab2d121-68d1-4cc1-9f0a-96755f3782f4" containerName="istio-proxy" containerID="cri-o://ceff6d0d5438bcf960823438a5d305ba8bbeef8ca1d142810f2ae29203e28e13" gracePeriod=30 Oct 31 04:22:35.219242 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:35.219210 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-9883e4-7f5fbccbbd-tv8v6"] Oct 31 04:22:35.219706 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:35.219482 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9883e4-7f5fbccbbd-tv8v6" podUID="47ce5c05-1ed0-471c-8a53-7d6f581e5f48" containerName="maas-api" containerID="cri-o://e45f86710c598eb9820798d43a15f9a01d91eaa40453b889891157095d91aa8b" gracePeriod=30 Oct 31 04:22:35.306558 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:35.306519 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9883e4-7f5fbccbbd-tv8v6" podUID="47ce5c05-1ed0-471c-8a53-7d6f581e5f48" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.86:8443/health\": dial tcp 10.133.0.86:8443: connect: connection refused" Oct 31 04:22:35.655596 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:35.655566 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9883e4-7f5fbccbbd-tv8v6" Oct 31 04:22:35.781080 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:35.778833 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/47ce5c05-1ed0-471c-8a53-7d6f581e5f48-metrics-certs\" (UniqueName: \"kubernetes.io/secret/47ce5c05-1ed0-471c-8a53-7d6f581e5f48-metrics-certs\") pod \"47ce5c05-1ed0-471c-8a53-7d6f581e5f48\" (UID: \"47ce5c05-1ed0-471c-8a53-7d6f581e5f48\") " Oct 31 04:22:35.781080 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:35.778975 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47ce5c05-1ed0-471c-8a53-7d6f581e5f48-kube-api-access-ppcsd\" (UniqueName: \"kubernetes.io/projected/47ce5c05-1ed0-471c-8a53-7d6f581e5f48-kube-api-access-ppcsd\") pod \"47ce5c05-1ed0-471c-8a53-7d6f581e5f48\" (UID: \"47ce5c05-1ed0-471c-8a53-7d6f581e5f48\") " Oct 31 04:22:35.781080 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:35.779176 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/47ce5c05-1ed0-471c-8a53-7d6f581e5f48-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47ce5c05-1ed0-471c-8a53-7d6f581e5f48-maas-api-tls\") pod \"47ce5c05-1ed0-471c-8a53-7d6f581e5f48\" (UID: \"47ce5c05-1ed0-471c-8a53-7d6f581e5f48\") " Oct 31 04:22:35.781080 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:35.779243 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47ce5c05-1ed0-471c-8a53-7d6f581e5f48-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/47ce5c05-1ed0-471c-8a53-7d6f581e5f48-openshift-service-ca\") pod \"47ce5c05-1ed0-471c-8a53-7d6f581e5f48\" (UID: \"47ce5c05-1ed0-471c-8a53-7d6f581e5f48\") " Oct 31 04:22:35.781971 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:35.781625 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47ce5c05-1ed0-471c-8a53-7d6f581e5f48-openshift-service-ca" pod "47ce5c05-1ed0-471c-8a53-7d6f581e5f48" (UID: "47ce5c05-1ed0-471c-8a53-7d6f581e5f48"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:22:35.807905 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:35.807596 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47ce5c05-1ed0-471c-8a53-7d6f581e5f48-metrics-certs" pod "47ce5c05-1ed0-471c-8a53-7d6f581e5f48" (UID: "47ce5c05-1ed0-471c-8a53-7d6f581e5f48"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:22:35.826189 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:35.826104 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47ce5c05-1ed0-471c-8a53-7d6f581e5f48-kube-api-access-ppcsd" pod "47ce5c05-1ed0-471c-8a53-7d6f581e5f48" (UID: "47ce5c05-1ed0-471c-8a53-7d6f581e5f48"). InnerVolumeSpecName "kube-api-access-ppcsd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:22:35.831576 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:35.831506 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47ce5c05-1ed0-471c-8a53-7d6f581e5f48-maas-api-tls" pod "47ce5c05-1ed0-471c-8a53-7d6f581e5f48" (UID: "47ce5c05-1ed0-471c-8a53-7d6f581e5f48"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:22:35.915353 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:35.884905 4376 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47ce5c05-1ed0-471c-8a53-7d6f581e5f48-maas-api-tls\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:22:35.915353 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:35.884939 4376 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/47ce5c05-1ed0-471c-8a53-7d6f581e5f48-openshift-service-ca\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:22:35.915353 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:35.884953 4376 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47ce5c05-1ed0-471c-8a53-7d6f581e5f48-metrics-certs\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:22:35.915353 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:35.884968 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ppcsd\" (UniqueName: \"kubernetes.io/projected/47ce5c05-1ed0-471c-8a53-7d6f581e5f48-kube-api-access-ppcsd\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:22:36.271921 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:36.271881 4376 generic.go:356] "Generic (PLEG): container finished" podID="47ce5c05-1ed0-471c-8a53-7d6f581e5f48" containerID="e45f86710c598eb9820798d43a15f9a01d91eaa40453b889891157095d91aa8b" exitCode=0 Oct 31 04:22:36.272324 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:36.271945 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9883e4-7f5fbccbbd-tv8v6" Oct 31 04:22:36.272324 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:36.271959 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9883e4-7f5fbccbbd-tv8v6" event={"ID":"47ce5c05-1ed0-471c-8a53-7d6f581e5f48","Type":"ContainerDied","Data":"e45f86710c598eb9820798d43a15f9a01d91eaa40453b889891157095d91aa8b"} Oct 31 04:22:36.272324 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:36.272043 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9883e4-7f5fbccbbd-tv8v6" event={"ID":"47ce5c05-1ed0-471c-8a53-7d6f581e5f48","Type":"ContainerDied","Data":"89010ab1d2b8f3bfaca477df0009bd4d9ecf9878aac38c3f7fb2d581e18885e7"} Oct 31 04:22:36.272324 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:36.272056 4376 scope.go:122] "RemoveContainer" containerID="e45f86710c598eb9820798d43a15f9a01d91eaa40453b889891157095d91aa8b" Oct 31 04:22:36.282682 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:36.282665 4376 scope.go:122] "RemoveContainer" containerID="e45f86710c598eb9820798d43a15f9a01d91eaa40453b889891157095d91aa8b" Oct 31 04:22:36.282921 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:22:36.282901 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e45f86710c598eb9820798d43a15f9a01d91eaa40453b889891157095d91aa8b\": container with ID starting with e45f86710c598eb9820798d43a15f9a01d91eaa40453b889891157095d91aa8b not found: ID does not exist" containerID="e45f86710c598eb9820798d43a15f9a01d91eaa40453b889891157095d91aa8b" Oct 31 04:22:36.282976 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:36.282928 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e45f86710c598eb9820798d43a15f9a01d91eaa40453b889891157095d91aa8b"} err="failed to get container status \"e45f86710c598eb9820798d43a15f9a01d91eaa40453b889891157095d91aa8b\": rpc error: code = NotFound desc = could not find container \"e45f86710c598eb9820798d43a15f9a01d91eaa40453b889891157095d91aa8b\": container with ID starting with e45f86710c598eb9820798d43a15f9a01d91eaa40453b889891157095d91aa8b not found: ID does not exist" Oct 31 04:22:36.325646 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:36.325615 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-9883e4-7f5fbccbbd-tv8v6"] Oct 31 04:22:36.338452 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:36.338407 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-9883e4-7f5fbccbbd-tv8v6"] Oct 31 04:22:36.465472 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:36.465428 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47ce5c05-1ed0-471c-8a53-7d6f581e5f48" path="/var/lib/kubelet/pods/47ce5c05-1ed0-471c-8a53-7d6f581e5f48/volumes" Oct 31 04:22:39.321769 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:39.321590 4376 generic.go:356] "Generic (PLEG): container finished" podID="5ab2d121-68d1-4cc1-9f0a-96755f3782f4" containerID="ceff6d0d5438bcf960823438a5d305ba8bbeef8ca1d142810f2ae29203e28e13" exitCode=0 Oct 31 04:22:39.321769 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:39.321679 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-8c0086-openshift-default-58b446d58c-cdgtp" event={"ID":"5ab2d121-68d1-4cc1-9f0a-96755f3782f4","Type":"ContainerDied","Data":"ceff6d0d5438bcf960823438a5d305ba8bbeef8ca1d142810f2ae29203e28e13"} Oct 31 04:22:39.321769 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:39.321743 4376 scope.go:122] "RemoveContainer" containerID="98e53ff19986f018955a4600a3874f577718448d591b945669c50655a96d8981" Oct 31 04:22:39.393096 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:39.393059 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-8c0086-openshift-default-58b446d58c-cdgtp" Oct 31 04:22:39.542232 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:39.542197 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ab2d121-68d1-4cc1-9f0a-96755f3782f4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ab2d121-68d1-4cc1-9f0a-96755f3782f4-credential-socket\") pod \"5ab2d121-68d1-4cc1-9f0a-96755f3782f4\" (UID: \"5ab2d121-68d1-4cc1-9f0a-96755f3782f4\") " Oct 31 04:22:39.542471 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:39.542267 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ab2d121-68d1-4cc1-9f0a-96755f3782f4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ab2d121-68d1-4cc1-9f0a-96755f3782f4-workload-certs\") pod \"5ab2d121-68d1-4cc1-9f0a-96755f3782f4\" (UID: \"5ab2d121-68d1-4cc1-9f0a-96755f3782f4\") " Oct 31 04:22:39.542471 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:39.542321 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ab2d121-68d1-4cc1-9f0a-96755f3782f4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ab2d121-68d1-4cc1-9f0a-96755f3782f4-workload-socket\") pod \"5ab2d121-68d1-4cc1-9f0a-96755f3782f4\" (UID: \"5ab2d121-68d1-4cc1-9f0a-96755f3782f4\") " Oct 31 04:22:39.542471 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:39.542445 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ab2d121-68d1-4cc1-9f0a-96755f3782f4-kube-api-access-7rvtx\" (UniqueName: \"kubernetes.io/projected/5ab2d121-68d1-4cc1-9f0a-96755f3782f4-kube-api-access-7rvtx\") pod \"5ab2d121-68d1-4cc1-9f0a-96755f3782f4\" (UID: \"5ab2d121-68d1-4cc1-9f0a-96755f3782f4\") " Oct 31 04:22:39.542644 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:39.542556 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ab2d121-68d1-4cc1-9f0a-96755f3782f4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ab2d121-68d1-4cc1-9f0a-96755f3782f4-istio-envoy\") pod \"5ab2d121-68d1-4cc1-9f0a-96755f3782f4\" (UID: \"5ab2d121-68d1-4cc1-9f0a-96755f3782f4\") " Oct 31 04:22:39.542644 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:39.542613 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ab2d121-68d1-4cc1-9f0a-96755f3782f4-credential-socket" pod "5ab2d121-68d1-4cc1-9f0a-96755f3782f4" (UID: "5ab2d121-68d1-4cc1-9f0a-96755f3782f4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:22:39.542644 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:39.542621 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5ab2d121-68d1-4cc1-9f0a-96755f3782f4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ab2d121-68d1-4cc1-9f0a-96755f3782f4-istio-podinfo\") pod \"5ab2d121-68d1-4cc1-9f0a-96755f3782f4\" (UID: \"5ab2d121-68d1-4cc1-9f0a-96755f3782f4\") " Oct 31 04:22:39.542813 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:39.542643 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ab2d121-68d1-4cc1-9f0a-96755f3782f4-workload-certs" pod "5ab2d121-68d1-4cc1-9f0a-96755f3782f4" (UID: "5ab2d121-68d1-4cc1-9f0a-96755f3782f4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:22:39.542813 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:39.542691 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ab2d121-68d1-4cc1-9f0a-96755f3782f4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ab2d121-68d1-4cc1-9f0a-96755f3782f4-istiod-ca-cert\") pod \"5ab2d121-68d1-4cc1-9f0a-96755f3782f4\" (UID: \"5ab2d121-68d1-4cc1-9f0a-96755f3782f4\") " Oct 31 04:22:39.542926 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:39.542827 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ab2d121-68d1-4cc1-9f0a-96755f3782f4-istio-token\" (UniqueName: \"kubernetes.io/projected/5ab2d121-68d1-4cc1-9f0a-96755f3782f4-istio-token\") pod \"5ab2d121-68d1-4cc1-9f0a-96755f3782f4\" (UID: \"5ab2d121-68d1-4cc1-9f0a-96755f3782f4\") " Oct 31 04:22:39.543053 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:39.543034 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ab2d121-68d1-4cc1-9f0a-96755f3782f4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ab2d121-68d1-4cc1-9f0a-96755f3782f4-istio-data\") pod \"5ab2d121-68d1-4cc1-9f0a-96755f3782f4\" (UID: \"5ab2d121-68d1-4cc1-9f0a-96755f3782f4\") " Oct 31 04:22:39.543200 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:39.543177 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ab2d121-68d1-4cc1-9f0a-96755f3782f4-istiod-ca-cert" pod "5ab2d121-68d1-4cc1-9f0a-96755f3782f4" (UID: "5ab2d121-68d1-4cc1-9f0a-96755f3782f4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:22:39.543484 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:39.543456 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ab2d121-68d1-4cc1-9f0a-96755f3782f4-workload-socket" pod "5ab2d121-68d1-4cc1-9f0a-96755f3782f4" (UID: "5ab2d121-68d1-4cc1-9f0a-96755f3782f4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:22:39.543610 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:39.543510 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ab2d121-68d1-4cc1-9f0a-96755f3782f4-istio-data" pod "5ab2d121-68d1-4cc1-9f0a-96755f3782f4" (UID: "5ab2d121-68d1-4cc1-9f0a-96755f3782f4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:22:39.544243 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:39.543986 4376 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ab2d121-68d1-4cc1-9f0a-96755f3782f4-istiod-ca-cert\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:22:39.544243 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:39.544011 4376 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ab2d121-68d1-4cc1-9f0a-96755f3782f4-istio-data\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:22:39.544243 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:39.544024 4376 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ab2d121-68d1-4cc1-9f0a-96755f3782f4-credential-socket\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:22:39.544243 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:39.544037 4376 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ab2d121-68d1-4cc1-9f0a-96755f3782f4-workload-certs\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:22:39.544243 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:39.544050 4376 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ab2d121-68d1-4cc1-9f0a-96755f3782f4-workload-socket\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:22:39.545527 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:39.545498 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ab2d121-68d1-4cc1-9f0a-96755f3782f4-istio-envoy" pod "5ab2d121-68d1-4cc1-9f0a-96755f3782f4" (UID: "5ab2d121-68d1-4cc1-9f0a-96755f3782f4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:22:39.545673 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:39.545651 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ab2d121-68d1-4cc1-9f0a-96755f3782f4-istio-token" pod "5ab2d121-68d1-4cc1-9f0a-96755f3782f4" (UID: "5ab2d121-68d1-4cc1-9f0a-96755f3782f4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:22:39.545982 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:39.545957 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5ab2d121-68d1-4cc1-9f0a-96755f3782f4-istio-podinfo" pod "5ab2d121-68d1-4cc1-9f0a-96755f3782f4" (UID: "5ab2d121-68d1-4cc1-9f0a-96755f3782f4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 04:22:39.546982 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:39.546960 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ab2d121-68d1-4cc1-9f0a-96755f3782f4-kube-api-access-7rvtx" pod "5ab2d121-68d1-4cc1-9f0a-96755f3782f4" (UID: "5ab2d121-68d1-4cc1-9f0a-96755f3782f4"). InnerVolumeSpecName "kube-api-access-7rvtx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:22:39.645698 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:39.645664 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rvtx\" (UniqueName: \"kubernetes.io/projected/5ab2d121-68d1-4cc1-9f0a-96755f3782f4-kube-api-access-7rvtx\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:22:39.645698 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:39.645692 4376 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ab2d121-68d1-4cc1-9f0a-96755f3782f4-istio-envoy\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:22:39.645698 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:39.645703 4376 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ab2d121-68d1-4cc1-9f0a-96755f3782f4-istio-podinfo\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:22:39.645953 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:39.645712 4376 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ab2d121-68d1-4cc1-9f0a-96755f3782f4-istio-token\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:22:40.329833 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:40.329798 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-8c0086-openshift-default-58b446d58c-cdgtp" Oct 31 04:22:40.329833 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:40.329823 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-8c0086-openshift-default-58b446d58c-cdgtp" event={"ID":"5ab2d121-68d1-4cc1-9f0a-96755f3782f4","Type":"ContainerDied","Data":"da13116685351b425ee141cd219b0b05a848647a269527e758f64e4e61d78c1a"} Oct 31 04:22:40.330323 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:40.329890 4376 scope.go:122] "RemoveContainer" containerID="ceff6d0d5438bcf960823438a5d305ba8bbeef8ca1d142810f2ae29203e28e13" Oct 31 04:22:40.397426 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:40.397390 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-8c0086-openshift-default-58b446d58c-cdgtp"] Oct 31 04:22:40.413599 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:40.411300 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-8c0086-openshift-default-58b446d58c-cdgtp"] Oct 31 04:22:40.445711 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:40.445681 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ab2d121-68d1-4cc1-9f0a-96755f3782f4" path="/var/lib/kubelet/pods/5ab2d121-68d1-4cc1-9f0a-96755f3782f4/volumes" Oct 31 04:22:41.166352 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:22:41.165625 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 04:22:46.529268 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:46.529237 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-9883e4-openshift-default-5666f48c8c-g9v2t"] Oct 31 04:22:46.529962 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:46.529930 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-9883e4-openshift-default-5666f48c8c-g9v2t" podUID="5c611e14-54b1-42de-b10d-faf45c6f6c04" containerName="istio-proxy" containerID="cri-o://977457ac069fad712b7ab6c7b61449c2b284a0f1b2f05c07e244bf9cfe15130a" gracePeriod=30 Oct 31 04:22:51.927353 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:51.913566 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-9883e4-openshift-default-5666f48c8c-g9v2t" Oct 31 04:22:52.008520 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:52.007477 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c611e14-54b1-42de-b10d-faf45c6f6c04-istio-token\" (UniqueName: \"kubernetes.io/projected/5c611e14-54b1-42de-b10d-faf45c6f6c04-istio-token\") pod \"5c611e14-54b1-42de-b10d-faf45c6f6c04\" (UID: \"5c611e14-54b1-42de-b10d-faf45c6f6c04\") " Oct 31 04:22:52.008520 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:52.007523 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c611e14-54b1-42de-b10d-faf45c6f6c04-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c611e14-54b1-42de-b10d-faf45c6f6c04-credential-socket\") pod \"5c611e14-54b1-42de-b10d-faf45c6f6c04\" (UID: \"5c611e14-54b1-42de-b10d-faf45c6f6c04\") " Oct 31 04:22:52.008520 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:52.007607 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c611e14-54b1-42de-b10d-faf45c6f6c04-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c611e14-54b1-42de-b10d-faf45c6f6c04-workload-socket\") pod \"5c611e14-54b1-42de-b10d-faf45c6f6c04\" (UID: \"5c611e14-54b1-42de-b10d-faf45c6f6c04\") " Oct 31 04:22:52.008520 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:52.007668 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c611e14-54b1-42de-b10d-faf45c6f6c04-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c611e14-54b1-42de-b10d-faf45c6f6c04-istio-envoy\") pod \"5c611e14-54b1-42de-b10d-faf45c6f6c04\" (UID: \"5c611e14-54b1-42de-b10d-faf45c6f6c04\") " Oct 31 04:22:52.008520 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:52.007719 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c611e14-54b1-42de-b10d-faf45c6f6c04-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c611e14-54b1-42de-b10d-faf45c6f6c04-workload-certs\") pod \"5c611e14-54b1-42de-b10d-faf45c6f6c04\" (UID: \"5c611e14-54b1-42de-b10d-faf45c6f6c04\") " Oct 31 04:22:52.008520 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:52.007747 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5c611e14-54b1-42de-b10d-faf45c6f6c04-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c611e14-54b1-42de-b10d-faf45c6f6c04-istio-podinfo\") pod \"5c611e14-54b1-42de-b10d-faf45c6f6c04\" (UID: \"5c611e14-54b1-42de-b10d-faf45c6f6c04\") " Oct 31 04:22:52.008520 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:52.007819 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c611e14-54b1-42de-b10d-faf45c6f6c04-kube-api-access-dc8f6\" (UniqueName: \"kubernetes.io/projected/5c611e14-54b1-42de-b10d-faf45c6f6c04-kube-api-access-dc8f6\") pod \"5c611e14-54b1-42de-b10d-faf45c6f6c04\" (UID: \"5c611e14-54b1-42de-b10d-faf45c6f6c04\") " Oct 31 04:22:52.008520 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:52.007961 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c611e14-54b1-42de-b10d-faf45c6f6c04-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c611e14-54b1-42de-b10d-faf45c6f6c04-istiod-ca-cert\") pod \"5c611e14-54b1-42de-b10d-faf45c6f6c04\" (UID: \"5c611e14-54b1-42de-b10d-faf45c6f6c04\") " Oct 31 04:22:52.008520 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:52.007995 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c611e14-54b1-42de-b10d-faf45c6f6c04-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c611e14-54b1-42de-b10d-faf45c6f6c04-istio-data\") pod \"5c611e14-54b1-42de-b10d-faf45c6f6c04\" (UID: \"5c611e14-54b1-42de-b10d-faf45c6f6c04\") " Oct 31 04:22:52.008520 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:52.008085 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c611e14-54b1-42de-b10d-faf45c6f6c04-workload-socket" pod "5c611e14-54b1-42de-b10d-faf45c6f6c04" (UID: "5c611e14-54b1-42de-b10d-faf45c6f6c04"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:22:52.008520 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:52.008130 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-0fc77f-ffb9b967-2fq8h"] Oct 31 04:22:52.008520 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:52.008373 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0fc77f-ffb9b967-2fq8h" podUID="4655062f-1974-4c0d-ac7a-ac0abe32991d" containerName="maas-api" containerID="cri-o://9bb348a99dfc8453fa69059d5f833d600b154d5411f4a3b601ca8b3c74d4f1de" gracePeriod=30 Oct 31 04:22:52.009304 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:52.008647 4376 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c611e14-54b1-42de-b10d-faf45c6f6c04-workload-socket\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:22:52.009304 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:52.008907 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c611e14-54b1-42de-b10d-faf45c6f6c04-istio-data" pod "5c611e14-54b1-42de-b10d-faf45c6f6c04" (UID: "5c611e14-54b1-42de-b10d-faf45c6f6c04"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:22:52.010901 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:52.010869 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c611e14-54b1-42de-b10d-faf45c6f6c04-istiod-ca-cert" pod "5c611e14-54b1-42de-b10d-faf45c6f6c04" (UID: "5c611e14-54b1-42de-b10d-faf45c6f6c04"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:22:52.011543 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:52.011514 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c611e14-54b1-42de-b10d-faf45c6f6c04-credential-socket" pod "5c611e14-54b1-42de-b10d-faf45c6f6c04" (UID: "5c611e14-54b1-42de-b10d-faf45c6f6c04"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:22:52.012005 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:52.011979 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c611e14-54b1-42de-b10d-faf45c6f6c04-workload-certs" pod "5c611e14-54b1-42de-b10d-faf45c6f6c04" (UID: "5c611e14-54b1-42de-b10d-faf45c6f6c04"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:22:52.012817 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:52.012791 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c611e14-54b1-42de-b10d-faf45c6f6c04-istio-envoy" pod "5c611e14-54b1-42de-b10d-faf45c6f6c04" (UID: "5c611e14-54b1-42de-b10d-faf45c6f6c04"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:22:52.013720 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:52.013696 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c611e14-54b1-42de-b10d-faf45c6f6c04-kube-api-access-dc8f6" pod "5c611e14-54b1-42de-b10d-faf45c6f6c04" (UID: "5c611e14-54b1-42de-b10d-faf45c6f6c04"). InnerVolumeSpecName "kube-api-access-dc8f6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:22:52.014062 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:52.014046 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5c611e14-54b1-42de-b10d-faf45c6f6c04-istio-podinfo" pod "5c611e14-54b1-42de-b10d-faf45c6f6c04" (UID: "5c611e14-54b1-42de-b10d-faf45c6f6c04"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 04:22:52.017557 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:52.017529 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c611e14-54b1-42de-b10d-faf45c6f6c04-istio-token" pod "5c611e14-54b1-42de-b10d-faf45c6f6c04" (UID: "5c611e14-54b1-42de-b10d-faf45c6f6c04"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:22:52.109992 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:52.109957 4376 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c611e14-54b1-42de-b10d-faf45c6f6c04-istio-token\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:22:52.109992 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:52.109985 4376 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c611e14-54b1-42de-b10d-faf45c6f6c04-credential-socket\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:22:52.109992 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:52.109998 4376 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c611e14-54b1-42de-b10d-faf45c6f6c04-istio-envoy\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:22:52.110215 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:52.110013 4376 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c611e14-54b1-42de-b10d-faf45c6f6c04-workload-certs\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:22:52.110215 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:52.110024 4376 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c611e14-54b1-42de-b10d-faf45c6f6c04-istio-podinfo\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:22:52.110215 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:52.110034 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dc8f6\" (UniqueName: \"kubernetes.io/projected/5c611e14-54b1-42de-b10d-faf45c6f6c04-kube-api-access-dc8f6\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:22:52.110215 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:52.110045 4376 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c611e14-54b1-42de-b10d-faf45c6f6c04-istiod-ca-cert\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:22:52.110215 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:52.110056 4376 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c611e14-54b1-42de-b10d-faf45c6f6c04-istio-data\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:22:52.266152 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:52.266126 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0fc77f-ffb9b967-2fq8h" Oct 31 04:22:52.313005 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:52.312969 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4655062f-1974-4c0d-ac7a-ac0abe32991d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4655062f-1974-4c0d-ac7a-ac0abe32991d-metrics-certs\") pod \"4655062f-1974-4c0d-ac7a-ac0abe32991d\" (UID: \"4655062f-1974-4c0d-ac7a-ac0abe32991d\") " Oct 31 04:22:52.313186 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:52.313111 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4655062f-1974-4c0d-ac7a-ac0abe32991d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4655062f-1974-4c0d-ac7a-ac0abe32991d-openshift-service-ca\") pod \"4655062f-1974-4c0d-ac7a-ac0abe32991d\" (UID: \"4655062f-1974-4c0d-ac7a-ac0abe32991d\") " Oct 31 04:22:52.313723 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:52.313220 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4655062f-1974-4c0d-ac7a-ac0abe32991d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4655062f-1974-4c0d-ac7a-ac0abe32991d-maas-api-tls\") pod \"4655062f-1974-4c0d-ac7a-ac0abe32991d\" (UID: \"4655062f-1974-4c0d-ac7a-ac0abe32991d\") " Oct 31 04:22:52.313723 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:52.313669 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4655062f-1974-4c0d-ac7a-ac0abe32991d-openshift-service-ca" pod "4655062f-1974-4c0d-ac7a-ac0abe32991d" (UID: "4655062f-1974-4c0d-ac7a-ac0abe32991d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:22:52.313890 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:52.313823 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4655062f-1974-4c0d-ac7a-ac0abe32991d-kube-api-access-scgzq\" (UniqueName: \"kubernetes.io/projected/4655062f-1974-4c0d-ac7a-ac0abe32991d-kube-api-access-scgzq\") pod \"4655062f-1974-4c0d-ac7a-ac0abe32991d\" (UID: \"4655062f-1974-4c0d-ac7a-ac0abe32991d\") " Oct 31 04:22:52.315065 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:52.315036 4376 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4655062f-1974-4c0d-ac7a-ac0abe32991d-openshift-service-ca\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:22:52.316255 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:52.316226 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4655062f-1974-4c0d-ac7a-ac0abe32991d-maas-api-tls" pod "4655062f-1974-4c0d-ac7a-ac0abe32991d" (UID: "4655062f-1974-4c0d-ac7a-ac0abe32991d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:22:52.316447 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:52.316422 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4655062f-1974-4c0d-ac7a-ac0abe32991d-metrics-certs" pod "4655062f-1974-4c0d-ac7a-ac0abe32991d" (UID: "4655062f-1974-4c0d-ac7a-ac0abe32991d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:22:52.317263 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:52.317241 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4655062f-1974-4c0d-ac7a-ac0abe32991d-kube-api-access-scgzq" pod "4655062f-1974-4c0d-ac7a-ac0abe32991d" (UID: "4655062f-1974-4c0d-ac7a-ac0abe32991d"). InnerVolumeSpecName "kube-api-access-scgzq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:22:52.416512 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:52.416472 4376 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4655062f-1974-4c0d-ac7a-ac0abe32991d-maas-api-tls\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:22:52.416512 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:52.416500 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-scgzq\" (UniqueName: \"kubernetes.io/projected/4655062f-1974-4c0d-ac7a-ac0abe32991d-kube-api-access-scgzq\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:22:52.416512 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:52.416508 4376 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4655062f-1974-4c0d-ac7a-ac0abe32991d-metrics-certs\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:22:52.672352 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:52.670620 4376 generic.go:356] "Generic (PLEG): container finished" podID="5c611e14-54b1-42de-b10d-faf45c6f6c04" containerID="977457ac069fad712b7ab6c7b61449c2b284a0f1b2f05c07e244bf9cfe15130a" exitCode=0 Oct 31 04:22:52.672352 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:52.671170 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9883e4-openshift-default-5666f48c8c-g9v2t" event={"ID":"5c611e14-54b1-42de-b10d-faf45c6f6c04","Type":"ContainerDied","Data":"977457ac069fad712b7ab6c7b61449c2b284a0f1b2f05c07e244bf9cfe15130a"} Oct 31 04:22:52.672352 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:52.671238 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9883e4-openshift-default-5666f48c8c-g9v2t" event={"ID":"5c611e14-54b1-42de-b10d-faf45c6f6c04","Type":"ContainerDied","Data":"8efcb68892655affb7e852e917fb2cad3b66712aacebdd57072207b20e430614"} Oct 31 04:22:52.672352 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:52.671271 4376 scope.go:122] "RemoveContainer" containerID="977457ac069fad712b7ab6c7b61449c2b284a0f1b2f05c07e244bf9cfe15130a" Oct 31 04:22:52.672352 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:52.671469 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-9883e4-openshift-default-5666f48c8c-g9v2t" Oct 31 04:22:52.703493 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:52.699402 4376 generic.go:356] "Generic (PLEG): container finished" podID="4655062f-1974-4c0d-ac7a-ac0abe32991d" containerID="9bb348a99dfc8453fa69059d5f833d600b154d5411f4a3b601ca8b3c74d4f1de" exitCode=0 Oct 31 04:22:52.703493 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:52.699520 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0fc77f-ffb9b967-2fq8h" event={"ID":"4655062f-1974-4c0d-ac7a-ac0abe32991d","Type":"ContainerDied","Data":"9bb348a99dfc8453fa69059d5f833d600b154d5411f4a3b601ca8b3c74d4f1de"} Oct 31 04:22:52.703493 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:52.699565 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0fc77f-ffb9b967-2fq8h" event={"ID":"4655062f-1974-4c0d-ac7a-ac0abe32991d","Type":"ContainerDied","Data":"e6ba015e0ed5e34990a7cfc92934038a94baf6736524df736052aa09bdcf174d"} Oct 31 04:22:52.703493 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:52.699659 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0fc77f-ffb9b967-2fq8h" Oct 31 04:22:52.724414 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:52.724378 4376 scope.go:122] "RemoveContainer" containerID="a5402ceb186f4948fa5368278a004b1f1ae679eea48803c4f687e9fba6d03f26" Oct 31 04:22:52.735597 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:52.735316 4376 scope.go:122] "RemoveContainer" containerID="977457ac069fad712b7ab6c7b61449c2b284a0f1b2f05c07e244bf9cfe15130a" Oct 31 04:22:52.735738 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:22:52.735641 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"977457ac069fad712b7ab6c7b61449c2b284a0f1b2f05c07e244bf9cfe15130a\": container with ID starting with 977457ac069fad712b7ab6c7b61449c2b284a0f1b2f05c07e244bf9cfe15130a not found: ID does not exist" containerID="977457ac069fad712b7ab6c7b61449c2b284a0f1b2f05c07e244bf9cfe15130a" Oct 31 04:22:52.735738 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:52.735677 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"977457ac069fad712b7ab6c7b61449c2b284a0f1b2f05c07e244bf9cfe15130a"} err="failed to get container status \"977457ac069fad712b7ab6c7b61449c2b284a0f1b2f05c07e244bf9cfe15130a\": rpc error: code = NotFound desc = could not find container \"977457ac069fad712b7ab6c7b61449c2b284a0f1b2f05c07e244bf9cfe15130a\": container with ID starting with 977457ac069fad712b7ab6c7b61449c2b284a0f1b2f05c07e244bf9cfe15130a not found: ID does not exist" Oct 31 04:22:52.735738 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:52.735699 4376 scope.go:122] "RemoveContainer" containerID="a5402ceb186f4948fa5368278a004b1f1ae679eea48803c4f687e9fba6d03f26" Oct 31 04:22:52.736015 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:22:52.735976 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5402ceb186f4948fa5368278a004b1f1ae679eea48803c4f687e9fba6d03f26\": container with ID starting with a5402ceb186f4948fa5368278a004b1f1ae679eea48803c4f687e9fba6d03f26 not found: ID does not exist" containerID="a5402ceb186f4948fa5368278a004b1f1ae679eea48803c4f687e9fba6d03f26" Oct 31 04:22:52.736097 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:52.736016 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5402ceb186f4948fa5368278a004b1f1ae679eea48803c4f687e9fba6d03f26"} err="failed to get container status \"a5402ceb186f4948fa5368278a004b1f1ae679eea48803c4f687e9fba6d03f26\": rpc error: code = NotFound desc = could not find container \"a5402ceb186f4948fa5368278a004b1f1ae679eea48803c4f687e9fba6d03f26\": container with ID starting with a5402ceb186f4948fa5368278a004b1f1ae679eea48803c4f687e9fba6d03f26 not found: ID does not exist" Oct 31 04:22:52.736097 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:52.736041 4376 scope.go:122] "RemoveContainer" containerID="9bb348a99dfc8453fa69059d5f833d600b154d5411f4a3b601ca8b3c74d4f1de" Oct 31 04:22:52.745362 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:52.745266 4376 scope.go:122] "RemoveContainer" containerID="9bb348a99dfc8453fa69059d5f833d600b154d5411f4a3b601ca8b3c74d4f1de" Oct 31 04:22:52.745598 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:22:52.745564 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9bb348a99dfc8453fa69059d5f833d600b154d5411f4a3b601ca8b3c74d4f1de\": container with ID starting with 9bb348a99dfc8453fa69059d5f833d600b154d5411f4a3b601ca8b3c74d4f1de not found: ID does not exist" containerID="9bb348a99dfc8453fa69059d5f833d600b154d5411f4a3b601ca8b3c74d4f1de" Oct 31 04:22:52.745649 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:52.745613 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bb348a99dfc8453fa69059d5f833d600b154d5411f4a3b601ca8b3c74d4f1de"} err="failed to get container status \"9bb348a99dfc8453fa69059d5f833d600b154d5411f4a3b601ca8b3c74d4f1de\": rpc error: code = NotFound desc = could not find container \"9bb348a99dfc8453fa69059d5f833d600b154d5411f4a3b601ca8b3c74d4f1de\": container with ID starting with 9bb348a99dfc8453fa69059d5f833d600b154d5411f4a3b601ca8b3c74d4f1de not found: ID does not exist" Oct 31 04:22:52.765852 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:52.765794 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-9883e4-openshift-default-5666f48c8c-g9v2t"] Oct 31 04:22:52.778039 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:52.778013 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-9883e4-openshift-default-5666f48c8c-g9v2t"] Oct 31 04:22:52.806213 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:52.806183 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-0fc77f-ffb9b967-2fq8h"] Oct 31 04:22:52.809695 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:52.809673 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-0fc77f-ffb9b967-2fq8h"] Oct 31 04:22:53.945606 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:53.945569 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a43a70-85d6b96975-wzq5w"] Oct 31 04:22:54.444298 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:54.444266 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4655062f-1974-4c0d-ac7a-ac0abe32991d" path="/var/lib/kubelet/pods/4655062f-1974-4c0d-ac7a-ac0abe32991d/volumes" Oct 31 04:22:54.444776 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:54.444761 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c611e14-54b1-42de-b10d-faf45c6f6c04" path="/var/lib/kubelet/pods/5c611e14-54b1-42de-b10d-faf45c6f6c04/volumes" Oct 31 04:22:56.956070 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:56.956036 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-3430f6-openshift-default-595d4f6b8c-ltm6g"] Oct 31 04:22:56.957048 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:56.956908 4376 cpu_manager.go:414] "containerMap: removing container" podUID="47ce5c05-1ed0-471c-8a53-7d6f581e5f48" containerName="maas-api" Oct 31 04:22:56.957048 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:56.956932 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47ce5c05-1ed0-471c-8a53-7d6f581e5f48" containerName="maas-api" Oct 31 04:22:56.957048 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:56.956947 4376 cpu_manager.go:414] "containerMap: removing container" podUID="5c611e14-54b1-42de-b10d-faf45c6f6c04" containerName="istio-proxy" Oct 31 04:22:56.957048 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:56.956955 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c611e14-54b1-42de-b10d-faf45c6f6c04" containerName="istio-proxy" Oct 31 04:22:56.957048 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:56.956984 4376 cpu_manager.go:414] "containerMap: removing container" podUID="5c611e14-54b1-42de-b10d-faf45c6f6c04" containerName="istio-proxy" Oct 31 04:22:56.957048 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:56.956990 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c611e14-54b1-42de-b10d-faf45c6f6c04" containerName="istio-proxy" Oct 31 04:22:56.957048 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:56.957022 4376 cpu_manager.go:414] "containerMap: removing container" podUID="9de59006-1b3d-48f4-a990-a51968da6a23" containerName="payload-processing" Oct 31 04:22:56.957048 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:56.957027 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9de59006-1b3d-48f4-a990-a51968da6a23" containerName="payload-processing" Oct 31 04:22:56.957048 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:56.957052 4376 cpu_manager.go:414] "containerMap: removing container" podUID="5ab2d121-68d1-4cc1-9f0a-96755f3782f4" containerName="istio-proxy" Oct 31 04:22:56.957048 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:56.957057 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ab2d121-68d1-4cc1-9f0a-96755f3782f4" containerName="istio-proxy" Oct 31 04:22:56.957558 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:56.957067 4376 cpu_manager.go:414] "containerMap: removing container" podUID="4655062f-1974-4c0d-ac7a-ac0abe32991d" containerName="maas-api" Oct 31 04:22:56.957558 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:56.957071 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4655062f-1974-4c0d-ac7a-ac0abe32991d" containerName="maas-api" Oct 31 04:22:56.957558 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:56.957088 4376 cpu_manager.go:414] "containerMap: removing container" podUID="5ab2d121-68d1-4cc1-9f0a-96755f3782f4" containerName="istio-proxy" Oct 31 04:22:56.957558 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:56.957096 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ab2d121-68d1-4cc1-9f0a-96755f3782f4" containerName="istio-proxy" Oct 31 04:22:56.957558 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:56.957209 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="9de59006-1b3d-48f4-a990-a51968da6a23" containerName="payload-processing" Oct 31 04:22:56.957558 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:56.957219 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ab2d121-68d1-4cc1-9f0a-96755f3782f4" containerName="istio-proxy" Oct 31 04:22:56.957558 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:56.957226 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="4655062f-1974-4c0d-ac7a-ac0abe32991d" containerName="maas-api" Oct 31 04:22:56.957558 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:56.957233 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ab2d121-68d1-4cc1-9f0a-96755f3782f4" containerName="istio-proxy" Oct 31 04:22:56.957558 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:56.957243 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="47ce5c05-1ed0-471c-8a53-7d6f581e5f48" containerName="maas-api" Oct 31 04:22:56.957558 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:56.957251 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c611e14-54b1-42de-b10d-faf45c6f6c04" containerName="istio-proxy" Oct 31 04:22:56.961917 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:56.961898 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3430f6-openshift-default-595d4f6b8c-ltm6g" Oct 31 04:22:56.992283 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:56.992251 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-3430f6-openshift-default-dockercfg-l77nt\"" Oct 31 04:22:56.998594 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:56.998563 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-3430f6-openshift-default-595d4f6b8c-ltm6g"] Oct 31 04:22:57.119354 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:57.115719 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ade7461-87d8-42ea-a721-512bae1f5f65-workload-certs\") pod \"e2e-ipp-a-3430f6-openshift-default-595d4f6b8c-ltm6g\" (UID: \"3ade7461-87d8-42ea-a721-512bae1f5f65\") " pod="openshift-ingress/e2e-ipp-a-3430f6-openshift-default-595d4f6b8c-ltm6g" Oct 31 04:22:57.119724 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:57.119698 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ade7461-87d8-42ea-a721-512bae1f5f65-workload-socket\") pod \"e2e-ipp-a-3430f6-openshift-default-595d4f6b8c-ltm6g\" (UID: \"3ade7461-87d8-42ea-a721-512bae1f5f65\") " pod="openshift-ingress/e2e-ipp-a-3430f6-openshift-default-595d4f6b8c-ltm6g" Oct 31 04:22:57.119938 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:57.119918 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ade7461-87d8-42ea-a721-512bae1f5f65-istio-envoy\") pod \"e2e-ipp-a-3430f6-openshift-default-595d4f6b8c-ltm6g\" (UID: \"3ade7461-87d8-42ea-a721-512bae1f5f65\") " pod="openshift-ingress/e2e-ipp-a-3430f6-openshift-default-595d4f6b8c-ltm6g" Oct 31 04:22:57.120166 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:57.120149 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ade7461-87d8-42ea-a721-512bae1f5f65-istiod-ca-cert\") pod \"e2e-ipp-a-3430f6-openshift-default-595d4f6b8c-ltm6g\" (UID: \"3ade7461-87d8-42ea-a721-512bae1f5f65\") " pod="openshift-ingress/e2e-ipp-a-3430f6-openshift-default-595d4f6b8c-ltm6g" Oct 31 04:22:57.120307 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:57.120291 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ade7461-87d8-42ea-a721-512bae1f5f65-credential-socket\") pod \"e2e-ipp-a-3430f6-openshift-default-595d4f6b8c-ltm6g\" (UID: \"3ade7461-87d8-42ea-a721-512bae1f5f65\") " pod="openshift-ingress/e2e-ipp-a-3430f6-openshift-default-595d4f6b8c-ltm6g" Oct 31 04:22:57.144352 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:57.120473 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxmq8\" (UniqueName: \"kubernetes.io/projected/3ade7461-87d8-42ea-a721-512bae1f5f65-kube-api-access-bxmq8\") pod \"e2e-ipp-a-3430f6-openshift-default-595d4f6b8c-ltm6g\" (UID: \"3ade7461-87d8-42ea-a721-512bae1f5f65\") " pod="openshift-ingress/e2e-ipp-a-3430f6-openshift-default-595d4f6b8c-ltm6g" Oct 31 04:22:57.145948 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:57.145894 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ade7461-87d8-42ea-a721-512bae1f5f65-istio-data\") pod \"e2e-ipp-a-3430f6-openshift-default-595d4f6b8c-ltm6g\" (UID: \"3ade7461-87d8-42ea-a721-512bae1f5f65\") " pod="openshift-ingress/e2e-ipp-a-3430f6-openshift-default-595d4f6b8c-ltm6g" Oct 31 04:22:57.146273 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:57.146254 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ade7461-87d8-42ea-a721-512bae1f5f65-istio-token\") pod \"e2e-ipp-a-3430f6-openshift-default-595d4f6b8c-ltm6g\" (UID: \"3ade7461-87d8-42ea-a721-512bae1f5f65\") " pod="openshift-ingress/e2e-ipp-a-3430f6-openshift-default-595d4f6b8c-ltm6g" Oct 31 04:22:57.146574 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:57.146553 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ade7461-87d8-42ea-a721-512bae1f5f65-istio-podinfo\") pod \"e2e-ipp-a-3430f6-openshift-default-595d4f6b8c-ltm6g\" (UID: \"3ade7461-87d8-42ea-a721-512bae1f5f65\") " pod="openshift-ingress/e2e-ipp-a-3430f6-openshift-default-595d4f6b8c-ltm6g" Oct 31 04:22:57.254003 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:57.253892 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ade7461-87d8-42ea-a721-512bae1f5f65-istiod-ca-cert\") pod \"e2e-ipp-a-3430f6-openshift-default-595d4f6b8c-ltm6g\" (UID: \"3ade7461-87d8-42ea-a721-512bae1f5f65\") " pod="openshift-ingress/e2e-ipp-a-3430f6-openshift-default-595d4f6b8c-ltm6g" Oct 31 04:22:57.254255 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:57.254236 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ade7461-87d8-42ea-a721-512bae1f5f65-credential-socket\") pod \"e2e-ipp-a-3430f6-openshift-default-595d4f6b8c-ltm6g\" (UID: \"3ade7461-87d8-42ea-a721-512bae1f5f65\") " pod="openshift-ingress/e2e-ipp-a-3430f6-openshift-default-595d4f6b8c-ltm6g" Oct 31 04:22:57.254682 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:57.254658 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxmq8\" (UniqueName: \"kubernetes.io/projected/3ade7461-87d8-42ea-a721-512bae1f5f65-kube-api-access-bxmq8\") pod \"e2e-ipp-a-3430f6-openshift-default-595d4f6b8c-ltm6g\" (UID: \"3ade7461-87d8-42ea-a721-512bae1f5f65\") " pod="openshift-ingress/e2e-ipp-a-3430f6-openshift-default-595d4f6b8c-ltm6g" Oct 31 04:22:57.254861 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:57.254843 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ade7461-87d8-42ea-a721-512bae1f5f65-istio-data\") pod \"e2e-ipp-a-3430f6-openshift-default-595d4f6b8c-ltm6g\" (UID: \"3ade7461-87d8-42ea-a721-512bae1f5f65\") " pod="openshift-ingress/e2e-ipp-a-3430f6-openshift-default-595d4f6b8c-ltm6g" Oct 31 04:22:57.254986 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:57.254970 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ade7461-87d8-42ea-a721-512bae1f5f65-istio-token\") pod \"e2e-ipp-a-3430f6-openshift-default-595d4f6b8c-ltm6g\" (UID: \"3ade7461-87d8-42ea-a721-512bae1f5f65\") " pod="openshift-ingress/e2e-ipp-a-3430f6-openshift-default-595d4f6b8c-ltm6g" Oct 31 04:22:57.255107 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:57.255094 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ade7461-87d8-42ea-a721-512bae1f5f65-istio-podinfo\") pod \"e2e-ipp-a-3430f6-openshift-default-595d4f6b8c-ltm6g\" (UID: \"3ade7461-87d8-42ea-a721-512bae1f5f65\") " pod="openshift-ingress/e2e-ipp-a-3430f6-openshift-default-595d4f6b8c-ltm6g" Oct 31 04:22:57.255259 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:57.255246 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ade7461-87d8-42ea-a721-512bae1f5f65-workload-certs\") pod \"e2e-ipp-a-3430f6-openshift-default-595d4f6b8c-ltm6g\" (UID: \"3ade7461-87d8-42ea-a721-512bae1f5f65\") " pod="openshift-ingress/e2e-ipp-a-3430f6-openshift-default-595d4f6b8c-ltm6g" Oct 31 04:22:57.255428 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:57.255414 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ade7461-87d8-42ea-a721-512bae1f5f65-workload-socket\") pod \"e2e-ipp-a-3430f6-openshift-default-595d4f6b8c-ltm6g\" (UID: \"3ade7461-87d8-42ea-a721-512bae1f5f65\") " pod="openshift-ingress/e2e-ipp-a-3430f6-openshift-default-595d4f6b8c-ltm6g" Oct 31 04:22:57.255553 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:57.255539 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ade7461-87d8-42ea-a721-512bae1f5f65-istio-envoy\") pod \"e2e-ipp-a-3430f6-openshift-default-595d4f6b8c-ltm6g\" (UID: \"3ade7461-87d8-42ea-a721-512bae1f5f65\") " pod="openshift-ingress/e2e-ipp-a-3430f6-openshift-default-595d4f6b8c-ltm6g" Oct 31 04:22:57.256793 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:57.256765 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ade7461-87d8-42ea-a721-512bae1f5f65-istio-data\") pod \"e2e-ipp-a-3430f6-openshift-default-595d4f6b8c-ltm6g\" (UID: \"3ade7461-87d8-42ea-a721-512bae1f5f65\") " pod="openshift-ingress/e2e-ipp-a-3430f6-openshift-default-595d4f6b8c-ltm6g" Oct 31 04:22:57.272016 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:57.258304 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ade7461-87d8-42ea-a721-512bae1f5f65-credential-socket\") pod \"e2e-ipp-a-3430f6-openshift-default-595d4f6b8c-ltm6g\" (UID: \"3ade7461-87d8-42ea-a721-512bae1f5f65\") " pod="openshift-ingress/e2e-ipp-a-3430f6-openshift-default-595d4f6b8c-ltm6g" Oct 31 04:22:57.272244 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:57.259088 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ade7461-87d8-42ea-a721-512bae1f5f65-istiod-ca-cert\") pod \"e2e-ipp-a-3430f6-openshift-default-595d4f6b8c-ltm6g\" (UID: \"3ade7461-87d8-42ea-a721-512bae1f5f65\") " pod="openshift-ingress/e2e-ipp-a-3430f6-openshift-default-595d4f6b8c-ltm6g" Oct 31 04:22:57.272368 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:57.270355 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ade7461-87d8-42ea-a721-512bae1f5f65-workload-certs\") pod \"e2e-ipp-a-3430f6-openshift-default-595d4f6b8c-ltm6g\" (UID: \"3ade7461-87d8-42ea-a721-512bae1f5f65\") " pod="openshift-ingress/e2e-ipp-a-3430f6-openshift-default-595d4f6b8c-ltm6g" Oct 31 04:22:57.272497 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:57.270770 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ade7461-87d8-42ea-a721-512bae1f5f65-workload-socket\") pod \"e2e-ipp-a-3430f6-openshift-default-595d4f6b8c-ltm6g\" (UID: \"3ade7461-87d8-42ea-a721-512bae1f5f65\") " pod="openshift-ingress/e2e-ipp-a-3430f6-openshift-default-595d4f6b8c-ltm6g" Oct 31 04:22:57.280934 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:57.280521 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ade7461-87d8-42ea-a721-512bae1f5f65-istio-envoy\") pod \"e2e-ipp-a-3430f6-openshift-default-595d4f6b8c-ltm6g\" (UID: \"3ade7461-87d8-42ea-a721-512bae1f5f65\") " pod="openshift-ingress/e2e-ipp-a-3430f6-openshift-default-595d4f6b8c-ltm6g" Oct 31 04:22:57.305281 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:57.304907 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ade7461-87d8-42ea-a721-512bae1f5f65-istio-podinfo\") pod \"e2e-ipp-a-3430f6-openshift-default-595d4f6b8c-ltm6g\" (UID: \"3ade7461-87d8-42ea-a721-512bae1f5f65\") " pod="openshift-ingress/e2e-ipp-a-3430f6-openshift-default-595d4f6b8c-ltm6g" Oct 31 04:22:57.317993 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:57.317501 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxmq8\" (UniqueName: \"kubernetes.io/projected/3ade7461-87d8-42ea-a721-512bae1f5f65-kube-api-access-bxmq8\") pod \"e2e-ipp-a-3430f6-openshift-default-595d4f6b8c-ltm6g\" (UID: \"3ade7461-87d8-42ea-a721-512bae1f5f65\") " pod="openshift-ingress/e2e-ipp-a-3430f6-openshift-default-595d4f6b8c-ltm6g" Oct 31 04:22:57.319022 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:57.318994 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ade7461-87d8-42ea-a721-512bae1f5f65-istio-token\") pod \"e2e-ipp-a-3430f6-openshift-default-595d4f6b8c-ltm6g\" (UID: \"3ade7461-87d8-42ea-a721-512bae1f5f65\") " pod="openshift-ingress/e2e-ipp-a-3430f6-openshift-default-595d4f6b8c-ltm6g" Oct 31 04:22:57.577067 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:57.576977 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3430f6-openshift-default-595d4f6b8c-ltm6g" Oct 31 04:22:57.715551 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:57.714087 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-f91cd1-openshift-default-556cb85755-v94km"] Oct 31 04:22:57.715551 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:57.715319 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c611e14-54b1-42de-b10d-faf45c6f6c04" containerName="istio-proxy" Oct 31 04:22:57.720607 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:57.720582 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f91cd1-openshift-default-556cb85755-v94km" Oct 31 04:22:57.742902 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:57.742871 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-f91cd1-openshift-default-dockercfg-x4vpw\"" Oct 31 04:22:57.754873 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:57.754836 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-f91cd1-openshift-default-556cb85755-v94km"] Oct 31 04:22:57.779655 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:57.779579 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd4632c9-069d-466e-b00a-c75aa4976d0d-workload-certs\") pod \"e2e-ipp-a-f91cd1-openshift-default-556cb85755-v94km\" (UID: \"fd4632c9-069d-466e-b00a-c75aa4976d0d\") " pod="openshift-ingress/e2e-ipp-a-f91cd1-openshift-default-556cb85755-v94km" Oct 31 04:22:57.779655 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:57.779636 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd4632c9-069d-466e-b00a-c75aa4976d0d-credential-socket\") pod \"e2e-ipp-a-f91cd1-openshift-default-556cb85755-v94km\" (UID: \"fd4632c9-069d-466e-b00a-c75aa4976d0d\") " pod="openshift-ingress/e2e-ipp-a-f91cd1-openshift-default-556cb85755-v94km" Oct 31 04:22:57.779938 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:57.779812 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd4632c9-069d-466e-b00a-c75aa4976d0d-istiod-ca-cert\") pod \"e2e-ipp-a-f91cd1-openshift-default-556cb85755-v94km\" (UID: \"fd4632c9-069d-466e-b00a-c75aa4976d0d\") " pod="openshift-ingress/e2e-ipp-a-f91cd1-openshift-default-556cb85755-v94km" Oct 31 04:22:57.779938 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:57.779863 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd4632c9-069d-466e-b00a-c75aa4976d0d-istio-token\") pod \"e2e-ipp-a-f91cd1-openshift-default-556cb85755-v94km\" (UID: \"fd4632c9-069d-466e-b00a-c75aa4976d0d\") " pod="openshift-ingress/e2e-ipp-a-f91cd1-openshift-default-556cb85755-v94km" Oct 31 04:22:57.779938 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:57.779908 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd4632c9-069d-466e-b00a-c75aa4976d0d-istio-data\") pod \"e2e-ipp-a-f91cd1-openshift-default-556cb85755-v94km\" (UID: \"fd4632c9-069d-466e-b00a-c75aa4976d0d\") " pod="openshift-ingress/e2e-ipp-a-f91cd1-openshift-default-556cb85755-v94km" Oct 31 04:22:57.780112 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:57.779944 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd4632c9-069d-466e-b00a-c75aa4976d0d-istio-podinfo\") pod \"e2e-ipp-a-f91cd1-openshift-default-556cb85755-v94km\" (UID: \"fd4632c9-069d-466e-b00a-c75aa4976d0d\") " pod="openshift-ingress/e2e-ipp-a-f91cd1-openshift-default-556cb85755-v94km" Oct 31 04:22:57.780112 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:57.780002 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd4632c9-069d-466e-b00a-c75aa4976d0d-workload-socket\") pod \"e2e-ipp-a-f91cd1-openshift-default-556cb85755-v94km\" (UID: \"fd4632c9-069d-466e-b00a-c75aa4976d0d\") " pod="openshift-ingress/e2e-ipp-a-f91cd1-openshift-default-556cb85755-v94km" Oct 31 04:22:57.780112 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:57.780099 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9lmb\" (UniqueName: \"kubernetes.io/projected/fd4632c9-069d-466e-b00a-c75aa4976d0d-kube-api-access-n9lmb\") pod \"e2e-ipp-a-f91cd1-openshift-default-556cb85755-v94km\" (UID: \"fd4632c9-069d-466e-b00a-c75aa4976d0d\") " pod="openshift-ingress/e2e-ipp-a-f91cd1-openshift-default-556cb85755-v94km" Oct 31 04:22:57.780315 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:57.780283 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd4632c9-069d-466e-b00a-c75aa4976d0d-istio-envoy\") pod \"e2e-ipp-a-f91cd1-openshift-default-556cb85755-v94km\" (UID: \"fd4632c9-069d-466e-b00a-c75aa4976d0d\") " pod="openshift-ingress/e2e-ipp-a-f91cd1-openshift-default-556cb85755-v94km" Oct 31 04:22:57.882268 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:57.882225 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd4632c9-069d-466e-b00a-c75aa4976d0d-istio-envoy\") pod \"e2e-ipp-a-f91cd1-openshift-default-556cb85755-v94km\" (UID: \"fd4632c9-069d-466e-b00a-c75aa4976d0d\") " pod="openshift-ingress/e2e-ipp-a-f91cd1-openshift-default-556cb85755-v94km" Oct 31 04:22:57.882514 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:57.882289 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd4632c9-069d-466e-b00a-c75aa4976d0d-workload-certs\") pod \"e2e-ipp-a-f91cd1-openshift-default-556cb85755-v94km\" (UID: \"fd4632c9-069d-466e-b00a-c75aa4976d0d\") " pod="openshift-ingress/e2e-ipp-a-f91cd1-openshift-default-556cb85755-v94km" Oct 31 04:22:57.882514 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:57.882368 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd4632c9-069d-466e-b00a-c75aa4976d0d-credential-socket\") pod \"e2e-ipp-a-f91cd1-openshift-default-556cb85755-v94km\" (UID: \"fd4632c9-069d-466e-b00a-c75aa4976d0d\") " pod="openshift-ingress/e2e-ipp-a-f91cd1-openshift-default-556cb85755-v94km" Oct 31 04:22:57.882514 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:57.882440 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd4632c9-069d-466e-b00a-c75aa4976d0d-istiod-ca-cert\") pod \"e2e-ipp-a-f91cd1-openshift-default-556cb85755-v94km\" (UID: \"fd4632c9-069d-466e-b00a-c75aa4976d0d\") " pod="openshift-ingress/e2e-ipp-a-f91cd1-openshift-default-556cb85755-v94km" Oct 31 04:22:57.882514 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:57.882476 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd4632c9-069d-466e-b00a-c75aa4976d0d-istio-token\") pod \"e2e-ipp-a-f91cd1-openshift-default-556cb85755-v94km\" (UID: \"fd4632c9-069d-466e-b00a-c75aa4976d0d\") " pod="openshift-ingress/e2e-ipp-a-f91cd1-openshift-default-556cb85755-v94km" Oct 31 04:22:57.882768 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:57.882518 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd4632c9-069d-466e-b00a-c75aa4976d0d-istio-data\") pod \"e2e-ipp-a-f91cd1-openshift-default-556cb85755-v94km\" (UID: \"fd4632c9-069d-466e-b00a-c75aa4976d0d\") " pod="openshift-ingress/e2e-ipp-a-f91cd1-openshift-default-556cb85755-v94km" Oct 31 04:22:57.882768 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:57.882556 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd4632c9-069d-466e-b00a-c75aa4976d0d-istio-podinfo\") pod \"e2e-ipp-a-f91cd1-openshift-default-556cb85755-v94km\" (UID: \"fd4632c9-069d-466e-b00a-c75aa4976d0d\") " pod="openshift-ingress/e2e-ipp-a-f91cd1-openshift-default-556cb85755-v94km" Oct 31 04:22:57.882768 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:57.882614 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd4632c9-069d-466e-b00a-c75aa4976d0d-workload-socket\") pod \"e2e-ipp-a-f91cd1-openshift-default-556cb85755-v94km\" (UID: \"fd4632c9-069d-466e-b00a-c75aa4976d0d\") " pod="openshift-ingress/e2e-ipp-a-f91cd1-openshift-default-556cb85755-v94km" Oct 31 04:22:57.882768 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:57.882678 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9lmb\" (UniqueName: \"kubernetes.io/projected/fd4632c9-069d-466e-b00a-c75aa4976d0d-kube-api-access-n9lmb\") pod \"e2e-ipp-a-f91cd1-openshift-default-556cb85755-v94km\" (UID: \"fd4632c9-069d-466e-b00a-c75aa4976d0d\") " pod="openshift-ingress/e2e-ipp-a-f91cd1-openshift-default-556cb85755-v94km" Oct 31 04:22:57.882768 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:57.882734 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd4632c9-069d-466e-b00a-c75aa4976d0d-workload-certs\") pod \"e2e-ipp-a-f91cd1-openshift-default-556cb85755-v94km\" (UID: \"fd4632c9-069d-466e-b00a-c75aa4976d0d\") " pod="openshift-ingress/e2e-ipp-a-f91cd1-openshift-default-556cb85755-v94km" Oct 31 04:22:57.883094 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:57.883050 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd4632c9-069d-466e-b00a-c75aa4976d0d-credential-socket\") pod \"e2e-ipp-a-f91cd1-openshift-default-556cb85755-v94km\" (UID: \"fd4632c9-069d-466e-b00a-c75aa4976d0d\") " pod="openshift-ingress/e2e-ipp-a-f91cd1-openshift-default-556cb85755-v94km" Oct 31 04:22:57.883094 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:57.883060 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd4632c9-069d-466e-b00a-c75aa4976d0d-istio-data\") pod \"e2e-ipp-a-f91cd1-openshift-default-556cb85755-v94km\" (UID: \"fd4632c9-069d-466e-b00a-c75aa4976d0d\") " pod="openshift-ingress/e2e-ipp-a-f91cd1-openshift-default-556cb85755-v94km" Oct 31 04:22:57.883216 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:57.883095 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd4632c9-069d-466e-b00a-c75aa4976d0d-workload-socket\") pod \"e2e-ipp-a-f91cd1-openshift-default-556cb85755-v94km\" (UID: \"fd4632c9-069d-466e-b00a-c75aa4976d0d\") " pod="openshift-ingress/e2e-ipp-a-f91cd1-openshift-default-556cb85755-v94km" Oct 31 04:22:57.883216 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:57.883162 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd4632c9-069d-466e-b00a-c75aa4976d0d-istiod-ca-cert\") pod \"e2e-ipp-a-f91cd1-openshift-default-556cb85755-v94km\" (UID: \"fd4632c9-069d-466e-b00a-c75aa4976d0d\") " pod="openshift-ingress/e2e-ipp-a-f91cd1-openshift-default-556cb85755-v94km" Oct 31 04:22:57.884681 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:57.884660 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd4632c9-069d-466e-b00a-c75aa4976d0d-istio-envoy\") pod \"e2e-ipp-a-f91cd1-openshift-default-556cb85755-v94km\" (UID: \"fd4632c9-069d-466e-b00a-c75aa4976d0d\") " pod="openshift-ingress/e2e-ipp-a-f91cd1-openshift-default-556cb85755-v94km" Oct 31 04:22:57.885036 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:57.885000 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd4632c9-069d-466e-b00a-c75aa4976d0d-istio-podinfo\") pod \"e2e-ipp-a-f91cd1-openshift-default-556cb85755-v94km\" (UID: \"fd4632c9-069d-466e-b00a-c75aa4976d0d\") " pod="openshift-ingress/e2e-ipp-a-f91cd1-openshift-default-556cb85755-v94km" Oct 31 04:22:57.902404 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:22:57.902376 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ade7461_87d8_42ea_a721_512bae1f5f65.slice/crio-d39c85cdb688145cbe9d5d675d15b8f6207cc93c82f7da029d6255464fee18ce WatchSource:0}: Error finding container d39c85cdb688145cbe9d5d675d15b8f6207cc93c82f7da029d6255464fee18ce: Status 404 returned error can't find the container with id d39c85cdb688145cbe9d5d675d15b8f6207cc93c82f7da029d6255464fee18ce Oct 31 04:22:57.903247 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:57.903221 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-3430f6-openshift-default-595d4f6b8c-ltm6g"] Oct 31 04:22:57.905407 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:57.905295 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:22:57.905514 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:57.905475 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:22:57.905574 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:57.905528 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:22:57.907754 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:57.907724 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd4632c9-069d-466e-b00a-c75aa4976d0d-istio-token\") pod \"e2e-ipp-a-f91cd1-openshift-default-556cb85755-v94km\" (UID: \"fd4632c9-069d-466e-b00a-c75aa4976d0d\") " pod="openshift-ingress/e2e-ipp-a-f91cd1-openshift-default-556cb85755-v94km" Oct 31 04:22:57.907871 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:57.907857 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9lmb\" (UniqueName: \"kubernetes.io/projected/fd4632c9-069d-466e-b00a-c75aa4976d0d-kube-api-access-n9lmb\") pod \"e2e-ipp-a-f91cd1-openshift-default-556cb85755-v94km\" (UID: \"fd4632c9-069d-466e-b00a-c75aa4976d0d\") " pod="openshift-ingress/e2e-ipp-a-f91cd1-openshift-default-556cb85755-v94km" Oct 31 04:22:58.052138 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:58.051810 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f91cd1-openshift-default-556cb85755-v94km" Oct 31 04:22:58.549186 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:58.548868 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 04:22:58.726473 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:22:58.726435 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd4632c9_069d_466e_b00a_c75aa4976d0d.slice/crio-a767d9c686b06a4570d14a42ec6922a14979f7cd3b1eacd2f7fbcf32220c4791 WatchSource:0}: Error finding container a767d9c686b06a4570d14a42ec6922a14979f7cd3b1eacd2f7fbcf32220c4791: Status 404 returned error can't find the container with id a767d9c686b06a4570d14a42ec6922a14979f7cd3b1eacd2f7fbcf32220c4791 Oct 31 04:22:58.728176 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:58.728151 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-f91cd1-openshift-default-556cb85755-v94km"] Oct 31 04:22:58.729267 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:58.729232 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:22:58.729355 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:58.729300 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:22:58.729399 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:58.729363 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:22:58.778986 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:58.778950 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f91cd1-openshift-default-556cb85755-v94km" event={"ID":"fd4632c9-069d-466e-b00a-c75aa4976d0d","Type":"ContainerStarted","Data":"a767d9c686b06a4570d14a42ec6922a14979f7cd3b1eacd2f7fbcf32220c4791"} Oct 31 04:22:58.780494 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:58.780466 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3430f6-openshift-default-595d4f6b8c-ltm6g" event={"ID":"3ade7461-87d8-42ea-a721-512bae1f5f65","Type":"ContainerStarted","Data":"137eadc48924c8f862ed0e8bb4ad99217a5739ed1579e0b78d0f8cdfab1423d3"} Oct 31 04:22:58.780598 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:58.780514 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3430f6-openshift-default-595d4f6b8c-ltm6g" event={"ID":"3ade7461-87d8-42ea-a721-512bae1f5f65","Type":"ContainerStarted","Data":"d39c85cdb688145cbe9d5d675d15b8f6207cc93c82f7da029d6255464fee18ce"} Oct 31 04:22:58.829068 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:58.829003 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-3430f6-openshift-default-595d4f6b8c-ltm6g" podStartSLOduration=2.828983119 podStartE2EDuration="2.828983119s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:22:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:22:58.825369031 +0000 UTC m=+1126.539703647" watchObservedRunningTime="2026-10-31 04:22:58.828983119 +0000 UTC m=+1126.543317736" Oct 31 04:22:58.947937 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:58.947172 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a43a70-85d6b96975-wzq5w" podUID="830c4180-0a49-43d9-bfba-ed9f0cadc4ec" containerName="payload-pre-processing" containerID="cri-o://32730010562cfd5e7cb8d26be447868d0f9537b6a44b4f2a346b6f0d9eca0986" gracePeriod=25 Oct 31 04:22:59.289483 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:59.289455 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a43a70-85d6b96975-wzq5w" Oct 31 04:22:59.462365 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:59.461851 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/830c4180-0a49-43d9-bfba-ed9f0cadc4ec-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/830c4180-0a49-43d9-bfba-ed9f0cadc4ec-plugins-config-volume\") pod \"830c4180-0a49-43d9-bfba-ed9f0cadc4ec\" (UID: \"830c4180-0a49-43d9-bfba-ed9f0cadc4ec\") " Oct 31 04:22:59.462365 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:59.462015 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/830c4180-0a49-43d9-bfba-ed9f0cadc4ec-kube-api-access-92fkn\" (UniqueName: \"kubernetes.io/projected/830c4180-0a49-43d9-bfba-ed9f0cadc4ec-kube-api-access-92fkn\") pod \"830c4180-0a49-43d9-bfba-ed9f0cadc4ec\" (UID: \"830c4180-0a49-43d9-bfba-ed9f0cadc4ec\") " Oct 31 04:22:59.482349 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:59.480058 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/830c4180-0a49-43d9-bfba-ed9f0cadc4ec-plugins-config-volume" pod "830c4180-0a49-43d9-bfba-ed9f0cadc4ec" (UID: "830c4180-0a49-43d9-bfba-ed9f0cadc4ec"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:22:59.491579 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:59.486951 4376 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/830c4180-0a49-43d9-bfba-ed9f0cadc4ec-plugins-config-volume\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:22:59.576615 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:59.576531 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/830c4180-0a49-43d9-bfba-ed9f0cadc4ec-kube-api-access-92fkn" pod "830c4180-0a49-43d9-bfba-ed9f0cadc4ec" (UID: "830c4180-0a49-43d9-bfba-ed9f0cadc4ec"). InnerVolumeSpecName "kube-api-access-92fkn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:22:59.586176 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:59.585704 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-3430f6-openshift-default-595d4f6b8c-ltm6g" Oct 31 04:22:59.599552 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:59.589110 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-92fkn\" (UniqueName: \"kubernetes.io/projected/830c4180-0a49-43d9-bfba-ed9f0cadc4ec-kube-api-access-92fkn\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:22:59.599552 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:59.594243 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-3430f6-openshift-default-595d4f6b8c-ltm6g" Oct 31 04:22:59.789853 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:59.789822 4376 generic.go:356] "Generic (PLEG): container finished" podID="830c4180-0a49-43d9-bfba-ed9f0cadc4ec" containerID="32730010562cfd5e7cb8d26be447868d0f9537b6a44b4f2a346b6f0d9eca0986" exitCode=0 Oct 31 04:22:59.790065 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:59.789875 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a43a70-85d6b96975-wzq5w" Oct 31 04:22:59.790065 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:59.789905 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a43a70-85d6b96975-wzq5w" event={"ID":"830c4180-0a49-43d9-bfba-ed9f0cadc4ec","Type":"ContainerDied","Data":"32730010562cfd5e7cb8d26be447868d0f9537b6a44b4f2a346b6f0d9eca0986"} Oct 31 04:22:59.790065 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:59.789977 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a43a70-85d6b96975-wzq5w" event={"ID":"830c4180-0a49-43d9-bfba-ed9f0cadc4ec","Type":"ContainerDied","Data":"e895ee89a75dacb1625ba7b2f71e821f2c1906224bbd4da1eb1f33c5c230db68"} Oct 31 04:22:59.790065 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:59.790005 4376 scope.go:122] "RemoveContainer" containerID="32730010562cfd5e7cb8d26be447868d0f9537b6a44b4f2a346b6f0d9eca0986" Oct 31 04:22:59.792395 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:59.792361 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f91cd1-openshift-default-556cb85755-v94km" event={"ID":"fd4632c9-069d-466e-b00a-c75aa4976d0d","Type":"ContainerStarted","Data":"3dd4da46f8a97e21d8f1fa204272a8ef9df80bb2368eb2ad619cd0b2063cfbb8"} Oct 31 04:22:59.793077 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:59.793042 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-3430f6-openshift-default-595d4f6b8c-ltm6g" Oct 31 04:22:59.797088 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:59.797063 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-3430f6-openshift-default-595d4f6b8c-ltm6g" Oct 31 04:22:59.800953 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:59.800935 4376 scope.go:122] "RemoveContainer" containerID="32730010562cfd5e7cb8d26be447868d0f9537b6a44b4f2a346b6f0d9eca0986" Oct 31 04:22:59.801261 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:22:59.801239 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32730010562cfd5e7cb8d26be447868d0f9537b6a44b4f2a346b6f0d9eca0986\": container with ID starting with 32730010562cfd5e7cb8d26be447868d0f9537b6a44b4f2a346b6f0d9eca0986 not found: ID does not exist" containerID="32730010562cfd5e7cb8d26be447868d0f9537b6a44b4f2a346b6f0d9eca0986" Oct 31 04:22:59.801320 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:59.801274 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32730010562cfd5e7cb8d26be447868d0f9537b6a44b4f2a346b6f0d9eca0986"} err="failed to get container status \"32730010562cfd5e7cb8d26be447868d0f9537b6a44b4f2a346b6f0d9eca0986\": rpc error: code = NotFound desc = could not find container \"32730010562cfd5e7cb8d26be447868d0f9537b6a44b4f2a346b6f0d9eca0986\": container with ID starting with 32730010562cfd5e7cb8d26be447868d0f9537b6a44b4f2a346b6f0d9eca0986 not found: ID does not exist" Oct 31 04:22:59.897859 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:59.897826 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a43a70-85d6b96975-wzq5w"] Oct 31 04:22:59.904996 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:59.904967 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a43a70-85d6b96975-wzq5w"] Oct 31 04:22:59.952413 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:22:59.952361 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-f91cd1-openshift-default-556cb85755-v94km" podStartSLOduration=2.952343198 podStartE2EDuration="2.952343198s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:22:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:22:59.9483728 +0000 UTC m=+1127.662707415" watchObservedRunningTime="2026-10-31 04:22:59.952343198 +0000 UTC m=+1127.666677815" Oct 31 04:23:00.052495 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:00.052465 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-f91cd1-openshift-default-556cb85755-v94km" Oct 31 04:23:00.061017 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:00.060957 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-f91cd1-openshift-default-556cb85755-v94km" Oct 31 04:23:00.444594 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:00.444561 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="830c4180-0a49-43d9-bfba-ed9f0cadc4ec" path="/var/lib/kubelet/pods/830c4180-0a49-43d9-bfba-ed9f0cadc4ec/volumes" Oct 31 04:23:00.800435 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:00.800322 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-f91cd1-openshift-default-556cb85755-v94km" Oct 31 04:23:00.801185 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:00.801163 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-f91cd1-openshift-default-556cb85755-v94km" Oct 31 04:23:02.810438 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:02.810405 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-a43a70-openshift-default-59d5b5dcf-rlkth"] Oct 31 04:23:02.810865 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:02.810626 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-a43a70-openshift-default-59d5b5dcf-rlkth" podUID="3c4d4b3d-2929-4563-ac6a-18d5f01c0db6" containerName="istio-proxy" containerID="cri-o://fc901d90a0557a98804f0fa1ac967a1768e5b41c868d4f1ffe70a472e03095d9" gracePeriod=30 Oct 31 04:23:03.356592 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:03.356558 4376 patch_prober.go:29] interesting pod/e2e-api-b-a43a70-openshift-default-59d5b5dcf-rlkth container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 04:23:03.356819 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:03.356620 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-a43a70-openshift-default-59d5b5dcf-rlkth" podUID="3c4d4b3d-2929-4563-ac6a-18d5f01c0db6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 04:23:05.760710 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:05.760670 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f91cd1-54fbfd87bc-4m89w"] Oct 31 04:23:05.761299 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:05.761285 4376 cpu_manager.go:414] "containerMap: removing container" podUID="830c4180-0a49-43d9-bfba-ed9f0cadc4ec" containerName="payload-pre-processing" Oct 31 04:23:05.761353 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:05.761300 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="830c4180-0a49-43d9-bfba-ed9f0cadc4ec" containerName="payload-pre-processing" Oct 31 04:23:05.761412 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:05.761402 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="830c4180-0a49-43d9-bfba-ed9f0cadc4ec" containerName="payload-pre-processing" Oct 31 04:23:05.765145 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:05.765128 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f91cd1-54fbfd87bc-4m89w" Oct 31 04:23:05.781129 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:05.781102 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f91cd1-54fbfd87bc-4m89w"] Oct 31 04:23:05.790218 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:05.790196 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-f91cd1\"" Oct 31 04:23:05.790218 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:05.790200 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-f91cd1-dockercfg-88hzz\"" Oct 31 04:23:06.075485 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:06.045111 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kplnr\" (UniqueName: \"kubernetes.io/projected/0b6efcad-2313-41dd-8f67-03bde20dcd34-kube-api-access-kplnr\") pod \"payload-pre-processing-e2e-ipp-a-f91cd1-54fbfd87bc-4m89w\" (UID: \"0b6efcad-2313-41dd-8f67-03bde20dcd34\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f91cd1-54fbfd87bc-4m89w" Oct 31 04:23:06.095683 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:06.095625 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b6efcad-2313-41dd-8f67-03bde20dcd34-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f91cd1-54fbfd87bc-4m89w\" (UID: \"0b6efcad-2313-41dd-8f67-03bde20dcd34\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f91cd1-54fbfd87bc-4m89w" Oct 31 04:23:06.128547 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:06.127360 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f91cd1-6688fc5c7-x47zn"] Oct 31 04:23:06.165735 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:06.165701 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f91cd1-6688fc5c7-x47zn"] Oct 31 04:23:06.165923 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:06.165826 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f91cd1-6688fc5c7-x47zn" Oct 31 04:23:06.197950 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:06.197920 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kplnr\" (UniqueName: \"kubernetes.io/projected/0b6efcad-2313-41dd-8f67-03bde20dcd34-kube-api-access-kplnr\") pod \"payload-pre-processing-e2e-ipp-a-f91cd1-54fbfd87bc-4m89w\" (UID: \"0b6efcad-2313-41dd-8f67-03bde20dcd34\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f91cd1-54fbfd87bc-4m89w" Oct 31 04:23:06.198120 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:06.198097 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b6efcad-2313-41dd-8f67-03bde20dcd34-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f91cd1-54fbfd87bc-4m89w\" (UID: \"0b6efcad-2313-41dd-8f67-03bde20dcd34\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f91cd1-54fbfd87bc-4m89w" Oct 31 04:23:06.198729 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:06.198712 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b6efcad-2313-41dd-8f67-03bde20dcd34-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f91cd1-54fbfd87bc-4m89w\" (UID: \"0b6efcad-2313-41dd-8f67-03bde20dcd34\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f91cd1-54fbfd87bc-4m89w" Oct 31 04:23:06.221604 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:06.221574 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kplnr\" (UniqueName: \"kubernetes.io/projected/0b6efcad-2313-41dd-8f67-03bde20dcd34-kube-api-access-kplnr\") pod \"payload-pre-processing-e2e-ipp-a-f91cd1-54fbfd87bc-4m89w\" (UID: \"0b6efcad-2313-41dd-8f67-03bde20dcd34\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f91cd1-54fbfd87bc-4m89w" Oct 31 04:23:06.300200 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:06.300164 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfx2c\" (UniqueName: \"kubernetes.io/projected/a82b667c-713c-4cc1-88fd-07742c8fc3b8-kube-api-access-vfx2c\") pod \"payload-processing-e2e-ipp-a-f91cd1-6688fc5c7-x47zn\" (UID: \"a82b667c-713c-4cc1-88fd-07742c8fc3b8\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f91cd1-6688fc5c7-x47zn" Oct 31 04:23:06.300391 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:06.300288 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a82b667c-713c-4cc1-88fd-07742c8fc3b8-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f91cd1-6688fc5c7-x47zn\" (UID: \"a82b667c-713c-4cc1-88fd-07742c8fc3b8\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f91cd1-6688fc5c7-x47zn" Oct 31 04:23:06.380247 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:06.380215 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f91cd1-54fbfd87bc-4m89w" Oct 31 04:23:06.402221 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:06.402187 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfx2c\" (UniqueName: \"kubernetes.io/projected/a82b667c-713c-4cc1-88fd-07742c8fc3b8-kube-api-access-vfx2c\") pod \"payload-processing-e2e-ipp-a-f91cd1-6688fc5c7-x47zn\" (UID: \"a82b667c-713c-4cc1-88fd-07742c8fc3b8\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f91cd1-6688fc5c7-x47zn" Oct 31 04:23:06.402389 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:06.402298 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a82b667c-713c-4cc1-88fd-07742c8fc3b8-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f91cd1-6688fc5c7-x47zn\" (UID: \"a82b667c-713c-4cc1-88fd-07742c8fc3b8\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f91cd1-6688fc5c7-x47zn" Oct 31 04:23:06.402990 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:06.402969 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a82b667c-713c-4cc1-88fd-07742c8fc3b8-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f91cd1-6688fc5c7-x47zn\" (UID: \"a82b667c-713c-4cc1-88fd-07742c8fc3b8\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f91cd1-6688fc5c7-x47zn" Oct 31 04:23:06.430060 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:06.430030 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfx2c\" (UniqueName: \"kubernetes.io/projected/a82b667c-713c-4cc1-88fd-07742c8fc3b8-kube-api-access-vfx2c\") pod \"payload-processing-e2e-ipp-a-f91cd1-6688fc5c7-x47zn\" (UID: \"a82b667c-713c-4cc1-88fd-07742c8fc3b8\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f91cd1-6688fc5c7-x47zn" Oct 31 04:23:06.483942 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:06.480769 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f91cd1-6688fc5c7-x47zn" Oct 31 04:23:06.681945 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:23:06.681914 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b6efcad_2313_41dd_8f67_03bde20dcd34.slice/crio-f44b56d0d02f480ae506ad2603aadad8c1c98fb3309f92ebbdf609a967b18ee0 WatchSource:0}: Error finding container f44b56d0d02f480ae506ad2603aadad8c1c98fb3309f92ebbdf609a967b18ee0: Status 404 returned error can't find the container with id f44b56d0d02f480ae506ad2603aadad8c1c98fb3309f92ebbdf609a967b18ee0 Oct 31 04:23:06.683440 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:06.683419 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f91cd1-54fbfd87bc-4m89w"] Oct 31 04:23:06.845810 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:06.845755 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f91cd1-6688fc5c7-x47zn"] Oct 31 04:23:07.052749 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:07.052684 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f91cd1-54fbfd87bc-4m89w" event={"ID":"0b6efcad-2313-41dd-8f67-03bde20dcd34","Type":"ContainerStarted","Data":"21210ff755518b3d4a873c5acd780cd29d2224b628113ec9ea4a005dc366c9d5"} Oct 31 04:23:07.052957 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:07.052771 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f91cd1-54fbfd87bc-4m89w" event={"ID":"0b6efcad-2313-41dd-8f67-03bde20dcd34","Type":"ContainerStarted","Data":"f44b56d0d02f480ae506ad2603aadad8c1c98fb3309f92ebbdf609a967b18ee0"} Oct 31 04:23:07.052957 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:07.052798 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f91cd1-54fbfd87bc-4m89w" Oct 31 04:23:07.054538 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:07.054511 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f91cd1-6688fc5c7-x47zn" event={"ID":"a82b667c-713c-4cc1-88fd-07742c8fc3b8","Type":"ContainerStarted","Data":"9a893490a679f1fba3910222551571feaa20673f7e7cbdbe2f1fa9cac19b9ddb"} Oct 31 04:23:07.054676 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:07.054554 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f91cd1-6688fc5c7-x47zn" event={"ID":"a82b667c-713c-4cc1-88fd-07742c8fc3b8","Type":"ContainerStarted","Data":"d43e4a1430888b87704d79c2c533d006a6a466001d26cc1d99fdb5427406c93e"} Oct 31 04:23:07.054676 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:07.054590 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-f91cd1-6688fc5c7-x47zn" Oct 31 04:23:07.097166 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:07.097107 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f91cd1-54fbfd87bc-4m89w" podStartSLOduration=2.097092151 podStartE2EDuration="2.097092151s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:23:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:23:07.094081716 +0000 UTC m=+1134.808416351" watchObservedRunningTime="2026-10-31 04:23:07.097092151 +0000 UTC m=+1134.811426767" Oct 31 04:23:07.138203 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:07.138145 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-f91cd1-6688fc5c7-x47zn" podStartSLOduration=2.138130007 podStartE2EDuration="2.138130007s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:23:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:23:07.135839358 +0000 UTC m=+1134.850173973" watchObservedRunningTime="2026-10-31 04:23:07.138130007 +0000 UTC m=+1134.852464623" Oct 31 04:23:08.063694 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:08.063658 4376 generic.go:356] "Generic (PLEG): container finished" podID="3c4d4b3d-2929-4563-ac6a-18d5f01c0db6" containerID="fc901d90a0557a98804f0fa1ac967a1768e5b41c868d4f1ffe70a472e03095d9" exitCode=0 Oct 31 04:23:08.064117 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:08.064095 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a43a70-openshift-default-59d5b5dcf-rlkth" event={"ID":"3c4d4b3d-2929-4563-ac6a-18d5f01c0db6","Type":"ContainerDied","Data":"fc901d90a0557a98804f0fa1ac967a1768e5b41c868d4f1ffe70a472e03095d9"} Oct 31 04:23:08.064225 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:08.064212 4376 scope.go:122] "RemoveContainer" containerID="0bad7043dcb2db1128e5638cd787770f06bb54cbe2f6ed979fd9b483785e4360" Oct 31 04:23:08.181236 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:08.181208 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-a43a70-openshift-default-59d5b5dcf-rlkth" Oct 31 04:23:08.265706 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:08.265668 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c4d4b3d-2929-4563-ac6a-18d5f01c0db6-kube-api-access-79zsd\" (UniqueName: \"kubernetes.io/projected/3c4d4b3d-2929-4563-ac6a-18d5f01c0db6-kube-api-access-79zsd\") pod \"3c4d4b3d-2929-4563-ac6a-18d5f01c0db6\" (UID: \"3c4d4b3d-2929-4563-ac6a-18d5f01c0db6\") " Oct 31 04:23:08.265900 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:08.265735 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c4d4b3d-2929-4563-ac6a-18d5f01c0db6-istio-token\" (UniqueName: \"kubernetes.io/projected/3c4d4b3d-2929-4563-ac6a-18d5f01c0db6-istio-token\") pod \"3c4d4b3d-2929-4563-ac6a-18d5f01c0db6\" (UID: \"3c4d4b3d-2929-4563-ac6a-18d5f01c0db6\") " Oct 31 04:23:08.265900 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:08.265793 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c4d4b3d-2929-4563-ac6a-18d5f01c0db6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c4d4b3d-2929-4563-ac6a-18d5f01c0db6-workload-socket\") pod \"3c4d4b3d-2929-4563-ac6a-18d5f01c0db6\" (UID: \"3c4d4b3d-2929-4563-ac6a-18d5f01c0db6\") " Oct 31 04:23:08.266031 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:08.265911 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c4d4b3d-2929-4563-ac6a-18d5f01c0db6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c4d4b3d-2929-4563-ac6a-18d5f01c0db6-credential-socket\") pod \"3c4d4b3d-2929-4563-ac6a-18d5f01c0db6\" (UID: \"3c4d4b3d-2929-4563-ac6a-18d5f01c0db6\") " Oct 31 04:23:08.266031 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:08.265995 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c4d4b3d-2929-4563-ac6a-18d5f01c0db6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c4d4b3d-2929-4563-ac6a-18d5f01c0db6-istio-data\") pod \"3c4d4b3d-2929-4563-ac6a-18d5f01c0db6\" (UID: \"3c4d4b3d-2929-4563-ac6a-18d5f01c0db6\") " Oct 31 04:23:08.266031 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:08.266020 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c4d4b3d-2929-4563-ac6a-18d5f01c0db6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c4d4b3d-2929-4563-ac6a-18d5f01c0db6-workload-certs\") pod \"3c4d4b3d-2929-4563-ac6a-18d5f01c0db6\" (UID: \"3c4d4b3d-2929-4563-ac6a-18d5f01c0db6\") " Oct 31 04:23:08.266188 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:08.266155 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3c4d4b3d-2929-4563-ac6a-18d5f01c0db6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c4d4b3d-2929-4563-ac6a-18d5f01c0db6-istio-podinfo\") pod \"3c4d4b3d-2929-4563-ac6a-18d5f01c0db6\" (UID: \"3c4d4b3d-2929-4563-ac6a-18d5f01c0db6\") " Oct 31 04:23:08.266259 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:08.266230 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c4d4b3d-2929-4563-ac6a-18d5f01c0db6-credential-socket" pod "3c4d4b3d-2929-4563-ac6a-18d5f01c0db6" (UID: "3c4d4b3d-2929-4563-ac6a-18d5f01c0db6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:23:08.266317 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:08.266243 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c4d4b3d-2929-4563-ac6a-18d5f01c0db6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c4d4b3d-2929-4563-ac6a-18d5f01c0db6-istio-envoy\") pod \"3c4d4b3d-2929-4563-ac6a-18d5f01c0db6\" (UID: \"3c4d4b3d-2929-4563-ac6a-18d5f01c0db6\") " Oct 31 04:23:08.268877 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:08.266498 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c4d4b3d-2929-4563-ac6a-18d5f01c0db6-workload-certs" pod "3c4d4b3d-2929-4563-ac6a-18d5f01c0db6" (UID: "3c4d4b3d-2929-4563-ac6a-18d5f01c0db6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:23:08.268877 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:08.266514 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c4d4b3d-2929-4563-ac6a-18d5f01c0db6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c4d4b3d-2929-4563-ac6a-18d5f01c0db6-istiod-ca-cert\") pod \"3c4d4b3d-2929-4563-ac6a-18d5f01c0db6\" (UID: \"3c4d4b3d-2929-4563-ac6a-18d5f01c0db6\") " Oct 31 04:23:08.268877 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:08.266924 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c4d4b3d-2929-4563-ac6a-18d5f01c0db6-istiod-ca-cert" pod "3c4d4b3d-2929-4563-ac6a-18d5f01c0db6" (UID: "3c4d4b3d-2929-4563-ac6a-18d5f01c0db6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:23:08.268877 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:08.267286 4376 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c4d4b3d-2929-4563-ac6a-18d5f01c0db6-istiod-ca-cert\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:23:08.268877 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:08.267309 4376 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c4d4b3d-2929-4563-ac6a-18d5f01c0db6-credential-socket\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:23:08.268877 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:08.267323 4376 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c4d4b3d-2929-4563-ac6a-18d5f01c0db6-workload-certs\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:23:08.268877 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:08.267668 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c4d4b3d-2929-4563-ac6a-18d5f01c0db6-istio-data" pod "3c4d4b3d-2929-4563-ac6a-18d5f01c0db6" (UID: "3c4d4b3d-2929-4563-ac6a-18d5f01c0db6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:23:08.268877 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:08.267724 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c4d4b3d-2929-4563-ac6a-18d5f01c0db6-workload-socket" pod "3c4d4b3d-2929-4563-ac6a-18d5f01c0db6" (UID: "3c4d4b3d-2929-4563-ac6a-18d5f01c0db6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:23:08.268877 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:08.268463 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c4d4b3d-2929-4563-ac6a-18d5f01c0db6-kube-api-access-79zsd" pod "3c4d4b3d-2929-4563-ac6a-18d5f01c0db6" (UID: "3c4d4b3d-2929-4563-ac6a-18d5f01c0db6"). InnerVolumeSpecName "kube-api-access-79zsd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:23:08.268877 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:08.268527 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c4d4b3d-2929-4563-ac6a-18d5f01c0db6-istio-token" pod "3c4d4b3d-2929-4563-ac6a-18d5f01c0db6" (UID: "3c4d4b3d-2929-4563-ac6a-18d5f01c0db6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:23:08.272047 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:08.271992 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3c4d4b3d-2929-4563-ac6a-18d5f01c0db6-istio-podinfo" pod "3c4d4b3d-2929-4563-ac6a-18d5f01c0db6" (UID: "3c4d4b3d-2929-4563-ac6a-18d5f01c0db6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 04:23:08.272189 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:08.272139 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c4d4b3d-2929-4563-ac6a-18d5f01c0db6-istio-envoy" pod "3c4d4b3d-2929-4563-ac6a-18d5f01c0db6" (UID: "3c4d4b3d-2929-4563-ac6a-18d5f01c0db6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:23:08.369162 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:08.369123 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79zsd\" (UniqueName: \"kubernetes.io/projected/3c4d4b3d-2929-4563-ac6a-18d5f01c0db6-kube-api-access-79zsd\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:23:08.369162 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:08.369159 4376 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c4d4b3d-2929-4563-ac6a-18d5f01c0db6-istio-token\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:23:08.369403 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:08.369169 4376 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c4d4b3d-2929-4563-ac6a-18d5f01c0db6-workload-socket\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:23:08.369403 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:08.369178 4376 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c4d4b3d-2929-4563-ac6a-18d5f01c0db6-istio-data\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:23:08.369403 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:08.369186 4376 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c4d4b3d-2929-4563-ac6a-18d5f01c0db6-istio-podinfo\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:23:08.369403 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:08.369194 4376 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c4d4b3d-2929-4563-ac6a-18d5f01c0db6-istio-envoy\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:23:09.098962 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:09.098930 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-a43a70-openshift-default-59d5b5dcf-rlkth" Oct 31 04:23:09.099446 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:09.098935 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a43a70-openshift-default-59d5b5dcf-rlkth" event={"ID":"3c4d4b3d-2929-4563-ac6a-18d5f01c0db6","Type":"ContainerDied","Data":"497d35c488a8cee933855c318627762b77d2fb083d9791b6497041826c19b741"} Oct 31 04:23:09.099446 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:09.099075 4376 scope.go:122] "RemoveContainer" containerID="fc901d90a0557a98804f0fa1ac967a1768e5b41c868d4f1ffe70a472e03095d9" Oct 31 04:23:09.170012 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:09.169943 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-a43a70-openshift-default-59d5b5dcf-rlkth"] Oct 31 04:23:09.184507 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:09.184472 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-a43a70-openshift-default-59d5b5dcf-rlkth"] Oct 31 04:23:10.443864 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:10.443834 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c4d4b3d-2929-4563-ac6a-18d5f01c0db6" path="/var/lib/kubelet/pods/3c4d4b3d-2929-4563-ac6a-18d5f01c0db6/volumes" Oct 31 04:23:11.001805 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:23:11.001757 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 04:23:11.010343 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:11.010302 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ceba51-bb5fb7d99-nbdsj"] Oct 31 04:23:11.010888 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:11.010873 4376 cpu_manager.go:414] "containerMap: removing container" podUID="3c4d4b3d-2929-4563-ac6a-18d5f01c0db6" containerName="istio-proxy" Oct 31 04:23:11.010936 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:11.010889 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c4d4b3d-2929-4563-ac6a-18d5f01c0db6" containerName="istio-proxy" Oct 31 04:23:11.011007 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:11.010999 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c4d4b3d-2929-4563-ac6a-18d5f01c0db6" containerName="istio-proxy" Oct 31 04:23:11.011044 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:11.011009 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c4d4b3d-2929-4563-ac6a-18d5f01c0db6" containerName="istio-proxy" Oct 31 04:23:11.015969 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:11.015947 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-ceba51-bb5fb7d99-nbdsj" Oct 31 04:23:11.031881 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:11.031853 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ceba51-bb5fb7d99-nbdsj"] Oct 31 04:23:11.040980 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:11.040952 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-ceba51\"" Oct 31 04:23:11.040980 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:11.040970 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-ceba51-dockercfg-q5zlr\"" Oct 31 04:23:11.077269 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:11.077237 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ceba51-5f4d757d95-ss4kp"] Oct 31 04:23:11.077897 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:11.077880 4376 cpu_manager.go:414] "containerMap: removing container" podUID="3c4d4b3d-2929-4563-ac6a-18d5f01c0db6" containerName="istio-proxy" Oct 31 04:23:11.077897 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:11.077899 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c4d4b3d-2929-4563-ac6a-18d5f01c0db6" containerName="istio-proxy" Oct 31 04:23:11.081433 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:11.081414 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ceba51-5f4d757d95-ss4kp" Oct 31 04:23:11.097827 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:11.097798 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ceba51-5f4d757d95-ss4kp"] Oct 31 04:23:11.335048 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:11.334934 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/407cfc33-fd88-4236-a178-52cfddd54cdc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-ceba51-5f4d757d95-ss4kp\" (UID: \"407cfc33-fd88-4236-a178-52cfddd54cdc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ceba51-5f4d757d95-ss4kp" Oct 31 04:23:11.335342 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:11.335302 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-768mf\" (UniqueName: \"kubernetes.io/projected/567e1919-44a9-4c76-ab90-bfdcfb1aef01-kube-api-access-768mf\") pod \"payload-processing-e2e-ipp-a-ceba51-bb5fb7d99-nbdsj\" (UID: \"567e1919-44a9-4c76-ab90-bfdcfb1aef01\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ceba51-bb5fb7d99-nbdsj" Oct 31 04:23:11.338788 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:11.335715 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6nml\" (UniqueName: \"kubernetes.io/projected/407cfc33-fd88-4236-a178-52cfddd54cdc-kube-api-access-j6nml\") pod \"payload-pre-processing-e2e-ipp-a-ceba51-5f4d757d95-ss4kp\" (UID: \"407cfc33-fd88-4236-a178-52cfddd54cdc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ceba51-5f4d757d95-ss4kp" Oct 31 04:23:11.364351 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:11.354736 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/567e1919-44a9-4c76-ab90-bfdcfb1aef01-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-ceba51-bb5fb7d99-nbdsj\" (UID: \"567e1919-44a9-4c76-ab90-bfdcfb1aef01\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ceba51-bb5fb7d99-nbdsj" Oct 31 04:23:11.456431 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:11.456387 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/567e1919-44a9-4c76-ab90-bfdcfb1aef01-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-ceba51-bb5fb7d99-nbdsj\" (UID: \"567e1919-44a9-4c76-ab90-bfdcfb1aef01\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ceba51-bb5fb7d99-nbdsj" Oct 31 04:23:11.456856 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:11.456495 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/407cfc33-fd88-4236-a178-52cfddd54cdc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-ceba51-5f4d757d95-ss4kp\" (UID: \"407cfc33-fd88-4236-a178-52cfddd54cdc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ceba51-5f4d757d95-ss4kp" Oct 31 04:23:11.456856 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:11.456541 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-768mf\" (UniqueName: \"kubernetes.io/projected/567e1919-44a9-4c76-ab90-bfdcfb1aef01-kube-api-access-768mf\") pod \"payload-processing-e2e-ipp-a-ceba51-bb5fb7d99-nbdsj\" (UID: \"567e1919-44a9-4c76-ab90-bfdcfb1aef01\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ceba51-bb5fb7d99-nbdsj" Oct 31 04:23:11.456856 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:11.456617 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6nml\" (UniqueName: \"kubernetes.io/projected/407cfc33-fd88-4236-a178-52cfddd54cdc-kube-api-access-j6nml\") pod \"payload-pre-processing-e2e-ipp-a-ceba51-5f4d757d95-ss4kp\" (UID: \"407cfc33-fd88-4236-a178-52cfddd54cdc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ceba51-5f4d757d95-ss4kp" Oct 31 04:23:11.457385 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:11.457363 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/407cfc33-fd88-4236-a178-52cfddd54cdc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-ceba51-5f4d757d95-ss4kp\" (UID: \"407cfc33-fd88-4236-a178-52cfddd54cdc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ceba51-5f4d757d95-ss4kp" Oct 31 04:23:11.457429 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:11.457397 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/567e1919-44a9-4c76-ab90-bfdcfb1aef01-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-ceba51-bb5fb7d99-nbdsj\" (UID: \"567e1919-44a9-4c76-ab90-bfdcfb1aef01\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ceba51-bb5fb7d99-nbdsj" Oct 31 04:23:11.480429 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:11.480397 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6nml\" (UniqueName: \"kubernetes.io/projected/407cfc33-fd88-4236-a178-52cfddd54cdc-kube-api-access-j6nml\") pod \"payload-pre-processing-e2e-ipp-a-ceba51-5f4d757d95-ss4kp\" (UID: \"407cfc33-fd88-4236-a178-52cfddd54cdc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ceba51-5f4d757d95-ss4kp" Oct 31 04:23:11.480613 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:11.480404 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-768mf\" (UniqueName: \"kubernetes.io/projected/567e1919-44a9-4c76-ab90-bfdcfb1aef01-kube-api-access-768mf\") pod \"payload-processing-e2e-ipp-a-ceba51-bb5fb7d99-nbdsj\" (UID: \"567e1919-44a9-4c76-ab90-bfdcfb1aef01\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ceba51-bb5fb7d99-nbdsj" Oct 31 04:23:11.629927 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:11.629898 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-ceba51-bb5fb7d99-nbdsj" Oct 31 04:23:11.706974 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:11.705495 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ceba51-5f4d757d95-ss4kp" Oct 31 04:23:11.931416 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:23:11.931381 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod567e1919_44a9_4c76_ab90_bfdcfb1aef01.slice/crio-6796d3971d9e36c1dc8c9ca2ae914de8aba64f537b904c0943e06ed35d48b641 WatchSource:0}: Error finding container 6796d3971d9e36c1dc8c9ca2ae914de8aba64f537b904c0943e06ed35d48b641: Status 404 returned error can't find the container with id 6796d3971d9e36c1dc8c9ca2ae914de8aba64f537b904c0943e06ed35d48b641 Oct 31 04:23:11.932898 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:11.932876 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ceba51-bb5fb7d99-nbdsj"] Oct 31 04:23:12.023060 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:23:12.023032 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod407cfc33_fd88_4236_a178_52cfddd54cdc.slice/crio-7c4ff67290e37a425dba21b28fc78019a7281d5abe2d51a9a7083964adae915e WatchSource:0}: Error finding container 7c4ff67290e37a425dba21b28fc78019a7281d5abe2d51a9a7083964adae915e: Status 404 returned error can't find the container with id 7c4ff67290e37a425dba21b28fc78019a7281d5abe2d51a9a7083964adae915e Oct 31 04:23:12.024926 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:12.024907 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ceba51-5f4d757d95-ss4kp"] Oct 31 04:23:12.241777 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:12.241677 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-ceba51-bb5fb7d99-nbdsj" event={"ID":"567e1919-44a9-4c76-ab90-bfdcfb1aef01","Type":"ContainerStarted","Data":"fcba94b185a7b6b69399cbed9c0440ecc0ffda1e0a653ee564d17041175a1395"} Oct 31 04:23:12.241777 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:12.241741 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-ceba51-bb5fb7d99-nbdsj" event={"ID":"567e1919-44a9-4c76-ab90-bfdcfb1aef01","Type":"ContainerStarted","Data":"6796d3971d9e36c1dc8c9ca2ae914de8aba64f537b904c0943e06ed35d48b641"} Oct 31 04:23:12.242019 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:12.241854 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-ceba51-bb5fb7d99-nbdsj" Oct 31 04:23:12.243500 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:12.243474 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ceba51-5f4d757d95-ss4kp" event={"ID":"407cfc33-fd88-4236-a178-52cfddd54cdc","Type":"ContainerStarted","Data":"c2f9b74936743bf0bd3e5c56851b49d8dbfd13c7e15dfdec16da754661138afd"} Oct 31 04:23:12.243641 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:12.243520 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ceba51-5f4d757d95-ss4kp" event={"ID":"407cfc33-fd88-4236-a178-52cfddd54cdc","Type":"ContainerStarted","Data":"7c4ff67290e37a425dba21b28fc78019a7281d5abe2d51a9a7083964adae915e"} Oct 31 04:23:12.243641 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:12.243580 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ceba51-5f4d757d95-ss4kp" Oct 31 04:23:12.284407 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:12.284362 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-ceba51-bb5fb7d99-nbdsj" podStartSLOduration=2.284350253 podStartE2EDuration="2.284350253s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:23:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:23:12.282226303 +0000 UTC m=+1139.996560935" watchObservedRunningTime="2026-10-31 04:23:12.284350253 +0000 UTC m=+1139.998684869" Oct 31 04:23:12.329837 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:12.329773 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ceba51-5f4d757d95-ss4kp" podStartSLOduration=1.329754788 podStartE2EDuration="1.329754788s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:23:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:23:12.32361904 +0000 UTC m=+1140.037953675" watchObservedRunningTime="2026-10-31 04:23:12.329754788 +0000 UTC m=+1140.044089407" Oct 31 04:23:15.657541 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:15.657508 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3430f6-774fd99d88-fqpg2"] Oct 31 04:23:15.662833 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:15.662811 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3430f6-774fd99d88-fqpg2" Oct 31 04:23:15.679927 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:15.679896 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3430f6-774fd99d88-fqpg2"] Oct 31 04:23:15.688244 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:15.688218 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-3430f6-serving-cert\"" Oct 31 04:23:15.748631 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:15.748600 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a62e709-216f-42e4-ac81-0c6712e61f31-maas-api-tls\") pod \"maas-api-e2e-ipp-a-3430f6-774fd99d88-fqpg2\" (UID: \"6a62e709-216f-42e4-ac81-0c6712e61f31\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3430f6-774fd99d88-fqpg2" Oct 31 04:23:15.748631 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:15.748633 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a62e709-216f-42e4-ac81-0c6712e61f31-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-3430f6-774fd99d88-fqpg2\" (UID: \"6a62e709-216f-42e4-ac81-0c6712e61f31\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3430f6-774fd99d88-fqpg2" Oct 31 04:23:15.748853 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:15.748780 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a62e709-216f-42e4-ac81-0c6712e61f31-metrics-certs\") pod \"maas-api-e2e-ipp-a-3430f6-774fd99d88-fqpg2\" (UID: \"6a62e709-216f-42e4-ac81-0c6712e61f31\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3430f6-774fd99d88-fqpg2" Oct 31 04:23:15.748932 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:15.748911 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8pbn\" (UniqueName: \"kubernetes.io/projected/6a62e709-216f-42e4-ac81-0c6712e61f31-kube-api-access-f8pbn\") pod \"maas-api-e2e-ipp-a-3430f6-774fd99d88-fqpg2\" (UID: \"6a62e709-216f-42e4-ac81-0c6712e61f31\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3430f6-774fd99d88-fqpg2" Oct 31 04:23:15.895366 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:15.892506 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a62e709-216f-42e4-ac81-0c6712e61f31-maas-api-tls\") pod \"maas-api-e2e-ipp-a-3430f6-774fd99d88-fqpg2\" (UID: \"6a62e709-216f-42e4-ac81-0c6712e61f31\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3430f6-774fd99d88-fqpg2" Oct 31 04:23:15.895366 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:15.892567 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a62e709-216f-42e4-ac81-0c6712e61f31-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-3430f6-774fd99d88-fqpg2\" (UID: \"6a62e709-216f-42e4-ac81-0c6712e61f31\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3430f6-774fd99d88-fqpg2" Oct 31 04:23:15.895366 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:15.892700 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a62e709-216f-42e4-ac81-0c6712e61f31-metrics-certs\") pod \"maas-api-e2e-ipp-a-3430f6-774fd99d88-fqpg2\" (UID: \"6a62e709-216f-42e4-ac81-0c6712e61f31\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3430f6-774fd99d88-fqpg2" Oct 31 04:23:15.895366 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:15.892871 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8pbn\" (UniqueName: \"kubernetes.io/projected/6a62e709-216f-42e4-ac81-0c6712e61f31-kube-api-access-f8pbn\") pod \"maas-api-e2e-ipp-a-3430f6-774fd99d88-fqpg2\" (UID: \"6a62e709-216f-42e4-ac81-0c6712e61f31\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3430f6-774fd99d88-fqpg2" Oct 31 04:23:15.897068 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:15.897033 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a62e709-216f-42e4-ac81-0c6712e61f31-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-3430f6-774fd99d88-fqpg2\" (UID: \"6a62e709-216f-42e4-ac81-0c6712e61f31\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3430f6-774fd99d88-fqpg2" Oct 31 04:23:15.963971 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:15.939471 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a62e709-216f-42e4-ac81-0c6712e61f31-maas-api-tls\") pod \"maas-api-e2e-ipp-a-3430f6-774fd99d88-fqpg2\" (UID: \"6a62e709-216f-42e4-ac81-0c6712e61f31\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3430f6-774fd99d88-fqpg2" Oct 31 04:23:15.963971 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:15.943435 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a62e709-216f-42e4-ac81-0c6712e61f31-metrics-certs\") pod \"maas-api-e2e-ipp-a-3430f6-774fd99d88-fqpg2\" (UID: \"6a62e709-216f-42e4-ac81-0c6712e61f31\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3430f6-774fd99d88-fqpg2" Oct 31 04:23:15.963971 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:15.943449 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8pbn\" (UniqueName: \"kubernetes.io/projected/6a62e709-216f-42e4-ac81-0c6712e61f31-kube-api-access-f8pbn\") pod \"maas-api-e2e-ipp-a-3430f6-774fd99d88-fqpg2\" (UID: \"6a62e709-216f-42e4-ac81-0c6712e61f31\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3430f6-774fd99d88-fqpg2" Oct 31 04:23:15.987007 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:15.985615 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3430f6-774fd99d88-fqpg2" Oct 31 04:23:16.323518 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:23:16.323484 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a62e709_216f_42e4_ac81_0c6712e61f31.slice/crio-c7dc1de2bd85d9e990c84275778b894cd04161739fe0ea2e0d108c8584872654 WatchSource:0}: Error finding container c7dc1de2bd85d9e990c84275778b894cd04161739fe0ea2e0d108c8584872654: Status 404 returned error can't find the container with id c7dc1de2bd85d9e990c84275778b894cd04161739fe0ea2e0d108c8584872654 Oct 31 04:23:16.325524 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:16.325498 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3430f6-774fd99d88-fqpg2"] Oct 31 04:23:17.276402 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:17.276368 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3430f6-774fd99d88-fqpg2" event={"ID":"6a62e709-216f-42e4-ac81-0c6712e61f31","Type":"ContainerStarted","Data":"ef3dcd284ba302f855f8e6377b6e79f8e4f82e34185b96e76b422847f4aa7e97"} Oct 31 04:23:17.276857 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:17.276425 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3430f6-774fd99d88-fqpg2" event={"ID":"6a62e709-216f-42e4-ac81-0c6712e61f31","Type":"ContainerStarted","Data":"c7dc1de2bd85d9e990c84275778b894cd04161739fe0ea2e0d108c8584872654"} Oct 31 04:23:17.276857 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:17.276584 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3430f6-774fd99d88-fqpg2" Oct 31 04:23:17.323228 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:17.323170 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3430f6-774fd99d88-fqpg2" podStartSLOduration=1.967268024 podStartE2EDuration="2.323155647s" totalImagesPullingTime="355.887623ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:23:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:23:17.321025993 +0000 UTC m=+1145.035360610" watchObservedRunningTime="2026-10-31 04:23:17.323155647 +0000 UTC m=+1145.037490270" Oct 31 04:23:18.067932 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:18.067906 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-f91cd1-6688fc5c7-x47zn" Oct 31 04:23:18.068105 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:18.067973 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f91cd1-54fbfd87bc-4m89w" Oct 31 04:23:23.252207 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:23.252178 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ceba51-5f4d757d95-ss4kp" Oct 31 04:23:23.252654 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:23.252243 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-ceba51-bb5fb7d99-nbdsj" Oct 31 04:23:23.292971 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:23.292944 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3430f6-774fd99d88-fqpg2" Oct 31 04:23:32.194666 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:32.194629 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-801218-openshift-default-58b4b8685c-8gpjq"] Oct 31 04:23:32.199445 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:32.199421 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-801218-openshift-default-58b4b8685c-8gpjq" Oct 31 04:23:32.218603 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:32.218576 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-801218-openshift-default-58b4b8685c-8gpjq"] Oct 31 04:23:32.226192 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:32.226170 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-801218-openshift-default-dockercfg-sxz4z\"" Oct 31 04:23:32.248706 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:32.248671 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3f3e12d-e06e-4b4e-a23a-784e49d5cbaf-istio-envoy\") pod \"e2e-api-a-801218-openshift-default-58b4b8685c-8gpjq\" (UID: \"e3f3e12d-e06e-4b4e-a23a-784e49d5cbaf\") " pod="openshift-ingress/e2e-api-a-801218-openshift-default-58b4b8685c-8gpjq" Oct 31 04:23:32.248839 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:32.248712 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vm88k\" (UniqueName: \"kubernetes.io/projected/e3f3e12d-e06e-4b4e-a23a-784e49d5cbaf-kube-api-access-vm88k\") pod \"e2e-api-a-801218-openshift-default-58b4b8685c-8gpjq\" (UID: \"e3f3e12d-e06e-4b4e-a23a-784e49d5cbaf\") " pod="openshift-ingress/e2e-api-a-801218-openshift-default-58b4b8685c-8gpjq" Oct 31 04:23:32.248920 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:32.248856 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3f3e12d-e06e-4b4e-a23a-784e49d5cbaf-credential-socket\") pod \"e2e-api-a-801218-openshift-default-58b4b8685c-8gpjq\" (UID: \"e3f3e12d-e06e-4b4e-a23a-784e49d5cbaf\") " pod="openshift-ingress/e2e-api-a-801218-openshift-default-58b4b8685c-8gpjq" Oct 31 04:23:32.248980 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:32.248931 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3f3e12d-e06e-4b4e-a23a-784e49d5cbaf-istio-data\") pod \"e2e-api-a-801218-openshift-default-58b4b8685c-8gpjq\" (UID: \"e3f3e12d-e06e-4b4e-a23a-784e49d5cbaf\") " pod="openshift-ingress/e2e-api-a-801218-openshift-default-58b4b8685c-8gpjq" Oct 31 04:23:32.249055 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:32.249038 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3f3e12d-e06e-4b4e-a23a-784e49d5cbaf-workload-certs\") pod \"e2e-api-a-801218-openshift-default-58b4b8685c-8gpjq\" (UID: \"e3f3e12d-e06e-4b4e-a23a-784e49d5cbaf\") " pod="openshift-ingress/e2e-api-a-801218-openshift-default-58b4b8685c-8gpjq" Oct 31 04:23:32.249163 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:32.249151 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3f3e12d-e06e-4b4e-a23a-784e49d5cbaf-workload-socket\") pod \"e2e-api-a-801218-openshift-default-58b4b8685c-8gpjq\" (UID: \"e3f3e12d-e06e-4b4e-a23a-784e49d5cbaf\") " pod="openshift-ingress/e2e-api-a-801218-openshift-default-58b4b8685c-8gpjq" Oct 31 04:23:32.249250 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:32.249215 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3f3e12d-e06e-4b4e-a23a-784e49d5cbaf-istio-token\") pod \"e2e-api-a-801218-openshift-default-58b4b8685c-8gpjq\" (UID: \"e3f3e12d-e06e-4b4e-a23a-784e49d5cbaf\") " pod="openshift-ingress/e2e-api-a-801218-openshift-default-58b4b8685c-8gpjq" Oct 31 04:23:32.249290 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:32.249279 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3f3e12d-e06e-4b4e-a23a-784e49d5cbaf-istiod-ca-cert\") pod \"e2e-api-a-801218-openshift-default-58b4b8685c-8gpjq\" (UID: \"e3f3e12d-e06e-4b4e-a23a-784e49d5cbaf\") " pod="openshift-ingress/e2e-api-a-801218-openshift-default-58b4b8685c-8gpjq" Oct 31 04:23:32.249324 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:32.249308 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3f3e12d-e06e-4b4e-a23a-784e49d5cbaf-istio-podinfo\") pod \"e2e-api-a-801218-openshift-default-58b4b8685c-8gpjq\" (UID: \"e3f3e12d-e06e-4b4e-a23a-784e49d5cbaf\") " pod="openshift-ingress/e2e-api-a-801218-openshift-default-58b4b8685c-8gpjq" Oct 31 04:23:32.370117 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:32.370059 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3f3e12d-e06e-4b4e-a23a-784e49d5cbaf-istio-envoy\") pod \"e2e-api-a-801218-openshift-default-58b4b8685c-8gpjq\" (UID: \"e3f3e12d-e06e-4b4e-a23a-784e49d5cbaf\") " pod="openshift-ingress/e2e-api-a-801218-openshift-default-58b4b8685c-8gpjq" Oct 31 04:23:32.370942 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:32.370917 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vm88k\" (UniqueName: \"kubernetes.io/projected/e3f3e12d-e06e-4b4e-a23a-784e49d5cbaf-kube-api-access-vm88k\") pod \"e2e-api-a-801218-openshift-default-58b4b8685c-8gpjq\" (UID: \"e3f3e12d-e06e-4b4e-a23a-784e49d5cbaf\") " pod="openshift-ingress/e2e-api-a-801218-openshift-default-58b4b8685c-8gpjq" Oct 31 04:23:32.371616 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:32.371593 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3f3e12d-e06e-4b4e-a23a-784e49d5cbaf-credential-socket\") pod \"e2e-api-a-801218-openshift-default-58b4b8685c-8gpjq\" (UID: \"e3f3e12d-e06e-4b4e-a23a-784e49d5cbaf\") " pod="openshift-ingress/e2e-api-a-801218-openshift-default-58b4b8685c-8gpjq" Oct 31 04:23:32.477490 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:32.472469 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3f3e12d-e06e-4b4e-a23a-784e49d5cbaf-istio-data\") pod \"e2e-api-a-801218-openshift-default-58b4b8685c-8gpjq\" (UID: \"e3f3e12d-e06e-4b4e-a23a-784e49d5cbaf\") " pod="openshift-ingress/e2e-api-a-801218-openshift-default-58b4b8685c-8gpjq" Oct 31 04:23:32.477490 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:32.472538 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3f3e12d-e06e-4b4e-a23a-784e49d5cbaf-workload-certs\") pod \"e2e-api-a-801218-openshift-default-58b4b8685c-8gpjq\" (UID: \"e3f3e12d-e06e-4b4e-a23a-784e49d5cbaf\") " pod="openshift-ingress/e2e-api-a-801218-openshift-default-58b4b8685c-8gpjq" Oct 31 04:23:32.477490 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:32.472683 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3f3e12d-e06e-4b4e-a23a-784e49d5cbaf-workload-socket\") pod \"e2e-api-a-801218-openshift-default-58b4b8685c-8gpjq\" (UID: \"e3f3e12d-e06e-4b4e-a23a-784e49d5cbaf\") " pod="openshift-ingress/e2e-api-a-801218-openshift-default-58b4b8685c-8gpjq" Oct 31 04:23:32.477490 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:32.472815 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3f3e12d-e06e-4b4e-a23a-784e49d5cbaf-istio-token\") pod \"e2e-api-a-801218-openshift-default-58b4b8685c-8gpjq\" (UID: \"e3f3e12d-e06e-4b4e-a23a-784e49d5cbaf\") " pod="openshift-ingress/e2e-api-a-801218-openshift-default-58b4b8685c-8gpjq" Oct 31 04:23:32.477490 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:32.472897 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3f3e12d-e06e-4b4e-a23a-784e49d5cbaf-istiod-ca-cert\") pod \"e2e-api-a-801218-openshift-default-58b4b8685c-8gpjq\" (UID: \"e3f3e12d-e06e-4b4e-a23a-784e49d5cbaf\") " pod="openshift-ingress/e2e-api-a-801218-openshift-default-58b4b8685c-8gpjq" Oct 31 04:23:32.477490 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:32.472940 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3f3e12d-e06e-4b4e-a23a-784e49d5cbaf-istio-podinfo\") pod \"e2e-api-a-801218-openshift-default-58b4b8685c-8gpjq\" (UID: \"e3f3e12d-e06e-4b4e-a23a-784e49d5cbaf\") " pod="openshift-ingress/e2e-api-a-801218-openshift-default-58b4b8685c-8gpjq" Oct 31 04:23:32.477490 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:32.473791 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3f3e12d-e06e-4b4e-a23a-784e49d5cbaf-workload-certs\") pod \"e2e-api-a-801218-openshift-default-58b4b8685c-8gpjq\" (UID: \"e3f3e12d-e06e-4b4e-a23a-784e49d5cbaf\") " pod="openshift-ingress/e2e-api-a-801218-openshift-default-58b4b8685c-8gpjq" Oct 31 04:23:32.477490 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:32.372100 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3f3e12d-e06e-4b4e-a23a-784e49d5cbaf-credential-socket\") pod \"e2e-api-a-801218-openshift-default-58b4b8685c-8gpjq\" (UID: \"e3f3e12d-e06e-4b4e-a23a-784e49d5cbaf\") " pod="openshift-ingress/e2e-api-a-801218-openshift-default-58b4b8685c-8gpjq" Oct 31 04:23:32.477490 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:32.474086 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3f3e12d-e06e-4b4e-a23a-784e49d5cbaf-istio-data\") pod \"e2e-api-a-801218-openshift-default-58b4b8685c-8gpjq\" (UID: \"e3f3e12d-e06e-4b4e-a23a-784e49d5cbaf\") " pod="openshift-ingress/e2e-api-a-801218-openshift-default-58b4b8685c-8gpjq" Oct 31 04:23:32.477490 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:32.465295 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vm88k\" (UniqueName: \"kubernetes.io/projected/e3f3e12d-e06e-4b4e-a23a-784e49d5cbaf-kube-api-access-vm88k\") pod \"e2e-api-a-801218-openshift-default-58b4b8685c-8gpjq\" (UID: \"e3f3e12d-e06e-4b4e-a23a-784e49d5cbaf\") " pod="openshift-ingress/e2e-api-a-801218-openshift-default-58b4b8685c-8gpjq" Oct 31 04:23:32.477490 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:32.474791 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3f3e12d-e06e-4b4e-a23a-784e49d5cbaf-workload-socket\") pod \"e2e-api-a-801218-openshift-default-58b4b8685c-8gpjq\" (UID: \"e3f3e12d-e06e-4b4e-a23a-784e49d5cbaf\") " pod="openshift-ingress/e2e-api-a-801218-openshift-default-58b4b8685c-8gpjq" Oct 31 04:23:32.477490 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:32.475462 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3f3e12d-e06e-4b4e-a23a-784e49d5cbaf-istiod-ca-cert\") pod \"e2e-api-a-801218-openshift-default-58b4b8685c-8gpjq\" (UID: \"e3f3e12d-e06e-4b4e-a23a-784e49d5cbaf\") " pod="openshift-ingress/e2e-api-a-801218-openshift-default-58b4b8685c-8gpjq" Oct 31 04:23:32.543356 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:32.541821 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3f3e12d-e06e-4b4e-a23a-784e49d5cbaf-istio-envoy\") pod \"e2e-api-a-801218-openshift-default-58b4b8685c-8gpjq\" (UID: \"e3f3e12d-e06e-4b4e-a23a-784e49d5cbaf\") " pod="openshift-ingress/e2e-api-a-801218-openshift-default-58b4b8685c-8gpjq" Oct 31 04:23:32.547351 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:32.543933 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3f3e12d-e06e-4b4e-a23a-784e49d5cbaf-istio-podinfo\") pod \"e2e-api-a-801218-openshift-default-58b4b8685c-8gpjq\" (UID: \"e3f3e12d-e06e-4b4e-a23a-784e49d5cbaf\") " pod="openshift-ingress/e2e-api-a-801218-openshift-default-58b4b8685c-8gpjq" Oct 31 04:23:32.599913 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:32.599878 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3f3e12d-e06e-4b4e-a23a-784e49d5cbaf-istio-token\") pod \"e2e-api-a-801218-openshift-default-58b4b8685c-8gpjq\" (UID: \"e3f3e12d-e06e-4b4e-a23a-784e49d5cbaf\") " pod="openshift-ingress/e2e-api-a-801218-openshift-default-58b4b8685c-8gpjq" Oct 31 04:23:32.815413 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:32.815312 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-801218-openshift-default-58b4b8685c-8gpjq" Oct 31 04:23:33.132215 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:23:33.132189 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3f3e12d_e06e_4b4e_a23a_784e49d5cbaf.slice/crio-8e349ce805ed9f8488d1261d495f1025b0c0afbe260b0d5933641c07fea5ce86 WatchSource:0}: Error finding container 8e349ce805ed9f8488d1261d495f1025b0c0afbe260b0d5933641c07fea5ce86: Status 404 returned error can't find the container with id 8e349ce805ed9f8488d1261d495f1025b0c0afbe260b0d5933641c07fea5ce86 Oct 31 04:23:33.134604 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:33.134580 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-801218-openshift-default-58b4b8685c-8gpjq"] Oct 31 04:23:33.135560 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:33.135528 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:23:33.135621 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:33.135603 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:23:33.135653 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:33.135633 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:23:33.564304 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:33.564220 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7102-0c87d7"] Oct 31 04:23:33.574843 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:33.574806 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-801218-openshift-default-58b4b8685c-8gpjq" event={"ID":"e3f3e12d-e06e-4b4e-a23a-784e49d5cbaf","Type":"ContainerStarted","Data":"f9cbfef45650c56699306fde4739e979d5b6d74a078474c21b5a6d2d92557af2"} Oct 31 04:23:33.575001 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:33.574862 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-801218-openshift-default-58b4b8685c-8gpjq" event={"ID":"e3f3e12d-e06e-4b4e-a23a-784e49d5cbaf","Type":"ContainerStarted","Data":"8e349ce805ed9f8488d1261d495f1025b0c0afbe260b0d5933641c07fea5ce86"} Oct 31 04:23:33.575001 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:33.574942 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7102-0c87d7" Oct 31 04:23:33.586883 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:33.586850 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7102-0c87d7"] Oct 31 04:23:33.590743 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:33.590726 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-prswc\"" Oct 31 04:23:33.619018 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:33.618967 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-801218-openshift-default-58b4b8685c-8gpjq" podStartSLOduration=1.61895428 podStartE2EDuration="1.61895428s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:23:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:23:33.616721587 +0000 UTC m=+1161.331056221" watchObservedRunningTime="2026-10-31 04:23:33.61895428 +0000 UTC m=+1161.333288896" Oct 31 04:23:33.780116 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:33.780080 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fc9hk\" (UniqueName: \"kubernetes.io/projected/c662d8a7-6eea-4a3c-bf0e-e5bd8b403364-kube-api-access-fc9hk\") pod \"test-curl-7102-0c87d7\" (UID: \"c662d8a7-6eea-4a3c-bf0e-e5bd8b403364\") " pod="opendatahub/test-curl-7102-0c87d7" Oct 31 04:23:33.821414 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:33.816069 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-801218-openshift-default-58b4b8685c-8gpjq" Oct 31 04:23:33.882480 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:33.882447 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fc9hk\" (UniqueName: \"kubernetes.io/projected/c662d8a7-6eea-4a3c-bf0e-e5bd8b403364-kube-api-access-fc9hk\") pod \"test-curl-7102-0c87d7\" (UID: \"c662d8a7-6eea-4a3c-bf0e-e5bd8b403364\") " pod="opendatahub/test-curl-7102-0c87d7" Oct 31 04:23:33.906263 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:33.906241 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fc9hk\" (UniqueName: \"kubernetes.io/projected/c662d8a7-6eea-4a3c-bf0e-e5bd8b403364-kube-api-access-fc9hk\") pod \"test-curl-7102-0c87d7\" (UID: \"c662d8a7-6eea-4a3c-bf0e-e5bd8b403364\") " pod="opendatahub/test-curl-7102-0c87d7" Oct 31 04:23:34.195216 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:34.195184 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7102-0c87d7" Oct 31 04:23:34.491202 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:23:34.491174 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc662d8a7_6eea_4a3c_bf0e_e5bd8b403364.slice/crio-996dad5571ccb6c4cddd6d62fbbfbdfe2f15f501a2a8f9963590f7665ae10eb8 WatchSource:0}: Error finding container 996dad5571ccb6c4cddd6d62fbbfbdfe2f15f501a2a8f9963590f7665ae10eb8: Status 404 returned error can't find the container with id 996dad5571ccb6c4cddd6d62fbbfbdfe2f15f501a2a8f9963590f7665ae10eb8 Oct 31 04:23:34.492713 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:34.492660 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7102-0c87d7"] Oct 31 04:23:34.579591 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:34.579556 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7102-0c87d7" event={"ID":"c662d8a7-6eea-4a3c-bf0e-e5bd8b403364","Type":"ContainerStarted","Data":"996dad5571ccb6c4cddd6d62fbbfbdfe2f15f501a2a8f9963590f7665ae10eb8"} Oct 31 04:23:34.820639 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:34.820565 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-801218-openshift-default-58b4b8685c-8gpjq" Oct 31 04:23:35.587150 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:35.587120 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-801218-openshift-default-58b4b8685c-8gpjq" Oct 31 04:23:35.588157 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:35.588137 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-801218-openshift-default-58b4b8685c-8gpjq" Oct 31 04:23:37.633847 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:37.633813 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7102-0c87d7" event={"ID":"c662d8a7-6eea-4a3c-bf0e-e5bd8b403364","Type":"ContainerStarted","Data":"04ecd6a4c24fff9ec9a836cfbd00458b9ebec04b0a0f84b9a2dbf92c1baa787b"} Oct 31 04:23:37.673323 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:37.673276 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7102-0c87d7" podStartSLOduration=2.348938545 podStartE2EDuration="4.673263352s" totalImagesPullingTime="2.324324807s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:23:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:23:37.671503819 +0000 UTC m=+1165.385838460" watchObservedRunningTime="2026-10-31 04:23:37.673263352 +0000 UTC m=+1165.387597968" Oct 31 04:23:38.570296 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:38.570261 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7102-0c87d7"] Oct 31 04:23:38.575994 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:38.575967 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7102-0c87d7"] Oct 31 04:23:38.957022 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:38.956973 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-801218-5c988b64cf-w2d4l"] Oct 31 04:23:38.957749 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:38.957731 4376 cpu_manager.go:414] "containerMap: removing container" podUID="c662d8a7-6eea-4a3c-bf0e-e5bd8b403364" containerName="test-curl-7102-0c87d7" Oct 31 04:23:38.957794 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:38.957749 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c662d8a7-6eea-4a3c-bf0e-e5bd8b403364" containerName="test-curl-7102-0c87d7" Oct 31 04:23:38.957844 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:38.957835 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="c662d8a7-6eea-4a3c-bf0e-e5bd8b403364" containerName="test-curl-7102-0c87d7" Oct 31 04:23:38.962582 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:38.962562 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-801218-5c988b64cf-w2d4l" Oct 31 04:23:38.980008 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:38.979980 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-801218\"" Oct 31 04:23:38.980421 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:38.980388 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-801218-5c988b64cf-w2d4l"] Oct 31 04:23:39.030970 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:39.030937 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-801218-7bd7574847-2fgt5"] Oct 31 04:23:39.035185 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:39.035165 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-801218-7bd7574847-2fgt5" Oct 31 04:23:39.050755 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:39.050725 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-801218-dockercfg-rwbmn\"" Oct 31 04:23:39.053251 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:39.053226 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-801218-7bd7574847-2fgt5"] Oct 31 04:23:39.130352 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:39.121321 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdw4m\" (UniqueName: \"kubernetes.io/projected/11f03146-3b0e-4cb0-b42c-0cb1d1952997-kube-api-access-mdw4m\") pod \"payload-processing-e2e-api-a-801218-5c988b64cf-w2d4l\" (UID: \"11f03146-3b0e-4cb0-b42c-0cb1d1952997\") " pod="openshift-ingress/payload-processing-e2e-api-a-801218-5c988b64cf-w2d4l" Oct 31 04:23:39.130352 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:39.121450 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11f03146-3b0e-4cb0-b42c-0cb1d1952997-plugins-config-volume\") pod \"payload-processing-e2e-api-a-801218-5c988b64cf-w2d4l\" (UID: \"11f03146-3b0e-4cb0-b42c-0cb1d1952997\") " pod="openshift-ingress/payload-processing-e2e-api-a-801218-5c988b64cf-w2d4l" Oct 31 04:23:39.130352 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:39.121542 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfj48\" (UniqueName: \"kubernetes.io/projected/862d48b9-2fae-4584-a58b-8dbd9d985857-kube-api-access-xfj48\") pod \"payload-pre-processing-e2e-api-a-801218-7bd7574847-2fgt5\" (UID: \"862d48b9-2fae-4584-a58b-8dbd9d985857\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-801218-7bd7574847-2fgt5" Oct 31 04:23:39.130352 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:39.121656 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/862d48b9-2fae-4584-a58b-8dbd9d985857-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-801218-7bd7574847-2fgt5\" (UID: \"862d48b9-2fae-4584-a58b-8dbd9d985857\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-801218-7bd7574847-2fgt5" Oct 31 04:23:39.244623 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:39.244446 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11f03146-3b0e-4cb0-b42c-0cb1d1952997-plugins-config-volume\") pod \"payload-processing-e2e-api-a-801218-5c988b64cf-w2d4l\" (UID: \"11f03146-3b0e-4cb0-b42c-0cb1d1952997\") " pod="openshift-ingress/payload-processing-e2e-api-a-801218-5c988b64cf-w2d4l" Oct 31 04:23:39.245870 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:39.245848 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfj48\" (UniqueName: \"kubernetes.io/projected/862d48b9-2fae-4584-a58b-8dbd9d985857-kube-api-access-xfj48\") pod \"payload-pre-processing-e2e-api-a-801218-7bd7574847-2fgt5\" (UID: \"862d48b9-2fae-4584-a58b-8dbd9d985857\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-801218-7bd7574847-2fgt5" Oct 31 04:23:39.246412 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:39.246390 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/862d48b9-2fae-4584-a58b-8dbd9d985857-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-801218-7bd7574847-2fgt5\" (UID: \"862d48b9-2fae-4584-a58b-8dbd9d985857\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-801218-7bd7574847-2fgt5" Oct 31 04:23:39.246734 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:39.246712 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdw4m\" (UniqueName: \"kubernetes.io/projected/11f03146-3b0e-4cb0-b42c-0cb1d1952997-kube-api-access-mdw4m\") pod \"payload-processing-e2e-api-a-801218-5c988b64cf-w2d4l\" (UID: \"11f03146-3b0e-4cb0-b42c-0cb1d1952997\") " pod="openshift-ingress/payload-processing-e2e-api-a-801218-5c988b64cf-w2d4l" Oct 31 04:23:39.251238 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:39.245721 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11f03146-3b0e-4cb0-b42c-0cb1d1952997-plugins-config-volume\") pod \"payload-processing-e2e-api-a-801218-5c988b64cf-w2d4l\" (UID: \"11f03146-3b0e-4cb0-b42c-0cb1d1952997\") " pod="openshift-ingress/payload-processing-e2e-api-a-801218-5c988b64cf-w2d4l" Oct 31 04:23:39.252181 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:39.252149 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/862d48b9-2fae-4584-a58b-8dbd9d985857-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-801218-7bd7574847-2fgt5\" (UID: \"862d48b9-2fae-4584-a58b-8dbd9d985857\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-801218-7bd7574847-2fgt5" Oct 31 04:23:39.377912 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:39.377753 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfj48\" (UniqueName: \"kubernetes.io/projected/862d48b9-2fae-4584-a58b-8dbd9d985857-kube-api-access-xfj48\") pod \"payload-pre-processing-e2e-api-a-801218-7bd7574847-2fgt5\" (UID: \"862d48b9-2fae-4584-a58b-8dbd9d985857\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-801218-7bd7574847-2fgt5" Oct 31 04:23:39.380751 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:39.380702 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdw4m\" (UniqueName: \"kubernetes.io/projected/11f03146-3b0e-4cb0-b42c-0cb1d1952997-kube-api-access-mdw4m\") pod \"payload-processing-e2e-api-a-801218-5c988b64cf-w2d4l\" (UID: \"11f03146-3b0e-4cb0-b42c-0cb1d1952997\") " pod="openshift-ingress/payload-processing-e2e-api-a-801218-5c988b64cf-w2d4l" Oct 31 04:23:39.577376 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:39.577281 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-801218-5c988b64cf-w2d4l" Oct 31 04:23:39.665611 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:39.649256 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7102-0c87d7" podUID="c662d8a7-6eea-4a3c-bf0e-e5bd8b403364" containerName="test-curl-7102-0c87d7" containerID="cri-o://04ecd6a4c24fff9ec9a836cfbd00458b9ebec04b0a0f84b9a2dbf92c1baa787b" gracePeriod=2 Oct 31 04:23:39.665611 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:39.655685 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-801218-7bd7574847-2fgt5" Oct 31 04:23:39.665611 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:23:39.663638 4376 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7102-0c87d7\" is forbidden: User \"system:node:ip-10-0-134-155.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-155.ec2.internal' and this object" podUID="c662d8a7-6eea-4a3c-bf0e-e5bd8b403364" pod="opendatahub/test-curl-7102-0c87d7" Oct 31 04:23:39.922323 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:39.922283 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-801218-5c988b64cf-w2d4l"] Oct 31 04:23:39.983538 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:23:39.983450 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod862d48b9_2fae_4584_a58b_8dbd9d985857.slice/crio-85d85d81caa126370ada75269fe800fe88d1d738d5cfa45202207c5fca5a7b41 WatchSource:0}: Error finding container 85d85d81caa126370ada75269fe800fe88d1d738d5cfa45202207c5fca5a7b41: Status 404 returned error can't find the container with id 85d85d81caa126370ada75269fe800fe88d1d738d5cfa45202207c5fca5a7b41 Oct 31 04:23:39.984502 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:39.984480 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-801218-7bd7574847-2fgt5"] Oct 31 04:23:40.658474 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:40.658435 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-801218-7bd7574847-2fgt5" event={"ID":"862d48b9-2fae-4584-a58b-8dbd9d985857","Type":"ContainerStarted","Data":"871924ac16310975ea9771c3314f78ccc7012c627b3380ca8ed70ad8e0e06458"} Oct 31 04:23:40.658683 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:40.658502 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-801218-7bd7574847-2fgt5" event={"ID":"862d48b9-2fae-4584-a58b-8dbd9d985857","Type":"ContainerStarted","Data":"85d85d81caa126370ada75269fe800fe88d1d738d5cfa45202207c5fca5a7b41"} Oct 31 04:23:40.658683 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:40.658646 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-801218-7bd7574847-2fgt5" Oct 31 04:23:40.660194 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:40.660166 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-801218-5c988b64cf-w2d4l" event={"ID":"11f03146-3b0e-4cb0-b42c-0cb1d1952997","Type":"ContainerStarted","Data":"f236204cd75a5a857b553c26d7df30ca7371a70b348c3fbb40483bb99ed98491"} Oct 31 04:23:40.660308 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:40.660212 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-801218-5c988b64cf-w2d4l" event={"ID":"11f03146-3b0e-4cb0-b42c-0cb1d1952997","Type":"ContainerStarted","Data":"aabb405cf91f5b79356f5cd081a3ccdb4d91b52cd370984b9a755fd03de034d3"} Oct 31 04:23:40.660308 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:40.660258 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-801218-5c988b64cf-w2d4l" Oct 31 04:23:40.700251 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:40.700209 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-801218-7bd7574847-2fgt5" podStartSLOduration=2.70019777 podStartE2EDuration="2.70019777s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:23:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:23:40.698881094 +0000 UTC m=+1168.413215709" watchObservedRunningTime="2026-10-31 04:23:40.70019777 +0000 UTC m=+1168.414532423" Oct 31 04:23:40.745183 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:40.745121 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-801218-5c988b64cf-w2d4l" podStartSLOduration=2.745104356 podStartE2EDuration="2.745104356s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:23:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:23:40.739867188 +0000 UTC m=+1168.454201803" watchObservedRunningTime="2026-10-31 04:23:40.745104356 +0000 UTC m=+1168.459438973" Oct 31 04:23:41.000293 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:23:41.000204 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 04:23:41.941544 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:41.941519 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7102-0c87d7" Oct 31 04:23:42.111776 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:42.111740 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c662d8a7-6eea-4a3c-bf0e-e5bd8b403364-kube-api-access-fc9hk\" (UniqueName: \"kubernetes.io/projected/c662d8a7-6eea-4a3c-bf0e-e5bd8b403364-kube-api-access-fc9hk\") pod \"c662d8a7-6eea-4a3c-bf0e-e5bd8b403364\" (UID: \"c662d8a7-6eea-4a3c-bf0e-e5bd8b403364\") " Oct 31 04:23:42.113854 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:42.113831 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c662d8a7-6eea-4a3c-bf0e-e5bd8b403364-kube-api-access-fc9hk" pod "c662d8a7-6eea-4a3c-bf0e-e5bd8b403364" (UID: "c662d8a7-6eea-4a3c-bf0e-e5bd8b403364"). InnerVolumeSpecName "kube-api-access-fc9hk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:23:42.213533 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:42.213450 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fc9hk\" (UniqueName: \"kubernetes.io/projected/c662d8a7-6eea-4a3c-bf0e-e5bd8b403364-kube-api-access-fc9hk\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:23:42.474884 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:42.474806 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c662d8a7-6eea-4a3c-bf0e-e5bd8b403364" path="/var/lib/kubelet/pods/c662d8a7-6eea-4a3c-bf0e-e5bd8b403364/volumes" Oct 31 04:23:42.676476 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:42.676441 4376 generic.go:356] "Generic (PLEG): container finished" podID="c662d8a7-6eea-4a3c-bf0e-e5bd8b403364" containerID="04ecd6a4c24fff9ec9a836cfbd00458b9ebec04b0a0f84b9a2dbf92c1baa787b" exitCode=137 Oct 31 04:23:42.676671 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:42.676484 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7102-0c87d7" Oct 31 04:23:42.676671 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:42.676512 4376 scope.go:122] "RemoveContainer" containerID="04ecd6a4c24fff9ec9a836cfbd00458b9ebec04b0a0f84b9a2dbf92c1baa787b" Oct 31 04:23:42.686698 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:42.686680 4376 scope.go:122] "RemoveContainer" containerID="04ecd6a4c24fff9ec9a836cfbd00458b9ebec04b0a0f84b9a2dbf92c1baa787b" Oct 31 04:23:42.686968 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:23:42.686944 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04ecd6a4c24fff9ec9a836cfbd00458b9ebec04b0a0f84b9a2dbf92c1baa787b\": container with ID starting with 04ecd6a4c24fff9ec9a836cfbd00458b9ebec04b0a0f84b9a2dbf92c1baa787b not found: ID does not exist" containerID="04ecd6a4c24fff9ec9a836cfbd00458b9ebec04b0a0f84b9a2dbf92c1baa787b" Oct 31 04:23:42.687021 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:42.686978 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04ecd6a4c24fff9ec9a836cfbd00458b9ebec04b0a0f84b9a2dbf92c1baa787b"} err="failed to get container status \"04ecd6a4c24fff9ec9a836cfbd00458b9ebec04b0a0f84b9a2dbf92c1baa787b\": rpc error: code = NotFound desc = could not find container \"04ecd6a4c24fff9ec9a836cfbd00458b9ebec04b0a0f84b9a2dbf92c1baa787b\": container with ID starting with 04ecd6a4c24fff9ec9a836cfbd00458b9ebec04b0a0f84b9a2dbf92c1baa787b not found: ID does not exist" Oct 31 04:23:43.563976 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:43.563943 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 04:23:43.849399 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:43.845216 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f91cd1-6688fc5c7-x47zn"] Oct 31 04:23:43.915395 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:43.913534 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f91cd1-54fbfd87bc-4m89w"] Oct 31 04:23:44.652607 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:44.652577 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-b8bb1e-openshift-default-557bbf897c-phbbm"] Oct 31 04:23:44.658764 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:44.658738 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-b8bb1e-openshift-default-557bbf897c-phbbm" Oct 31 04:23:44.685510 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:44.685482 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-b8bb1e-openshift-default-557bbf897c-phbbm"] Oct 31 04:23:44.737829 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:44.737794 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-b8bb1e-openshift-default-dockercfg-dwzjk\"" Oct 31 04:23:44.771740 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:44.771709 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61a59ba0-c79d-4942-8b8a-1493cb9e9182-credential-socket\") pod \"e2e-ipp-b-b8bb1e-openshift-default-557bbf897c-phbbm\" (UID: \"61a59ba0-c79d-4942-8b8a-1493cb9e9182\") " pod="openshift-ingress/e2e-ipp-b-b8bb1e-openshift-default-557bbf897c-phbbm" Oct 31 04:23:44.771899 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:44.771807 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61a59ba0-c79d-4942-8b8a-1493cb9e9182-istiod-ca-cert\") pod \"e2e-ipp-b-b8bb1e-openshift-default-557bbf897c-phbbm\" (UID: \"61a59ba0-c79d-4942-8b8a-1493cb9e9182\") " pod="openshift-ingress/e2e-ipp-b-b8bb1e-openshift-default-557bbf897c-phbbm" Oct 31 04:23:44.771899 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:44.771838 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48gpj\" (UniqueName: \"kubernetes.io/projected/61a59ba0-c79d-4942-8b8a-1493cb9e9182-kube-api-access-48gpj\") pod \"e2e-ipp-b-b8bb1e-openshift-default-557bbf897c-phbbm\" (UID: \"61a59ba0-c79d-4942-8b8a-1493cb9e9182\") " pod="openshift-ingress/e2e-ipp-b-b8bb1e-openshift-default-557bbf897c-phbbm" Oct 31 04:23:44.771996 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:44.771981 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61a59ba0-c79d-4942-8b8a-1493cb9e9182-workload-certs\") pod \"e2e-ipp-b-b8bb1e-openshift-default-557bbf897c-phbbm\" (UID: \"61a59ba0-c79d-4942-8b8a-1493cb9e9182\") " pod="openshift-ingress/e2e-ipp-b-b8bb1e-openshift-default-557bbf897c-phbbm" Oct 31 04:23:44.772030 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:44.772013 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61a59ba0-c79d-4942-8b8a-1493cb9e9182-istio-podinfo\") pod \"e2e-ipp-b-b8bb1e-openshift-default-557bbf897c-phbbm\" (UID: \"61a59ba0-c79d-4942-8b8a-1493cb9e9182\") " pod="openshift-ingress/e2e-ipp-b-b8bb1e-openshift-default-557bbf897c-phbbm" Oct 31 04:23:44.772062 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:44.772046 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61a59ba0-c79d-4942-8b8a-1493cb9e9182-workload-socket\") pod \"e2e-ipp-b-b8bb1e-openshift-default-557bbf897c-phbbm\" (UID: \"61a59ba0-c79d-4942-8b8a-1493cb9e9182\") " pod="openshift-ingress/e2e-ipp-b-b8bb1e-openshift-default-557bbf897c-phbbm" Oct 31 04:23:44.772163 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:44.772146 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61a59ba0-c79d-4942-8b8a-1493cb9e9182-istio-envoy\") pod \"e2e-ipp-b-b8bb1e-openshift-default-557bbf897c-phbbm\" (UID: \"61a59ba0-c79d-4942-8b8a-1493cb9e9182\") " pod="openshift-ingress/e2e-ipp-b-b8bb1e-openshift-default-557bbf897c-phbbm" Oct 31 04:23:44.772275 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:44.772259 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61a59ba0-c79d-4942-8b8a-1493cb9e9182-istio-token\") pod \"e2e-ipp-b-b8bb1e-openshift-default-557bbf897c-phbbm\" (UID: \"61a59ba0-c79d-4942-8b8a-1493cb9e9182\") " pod="openshift-ingress/e2e-ipp-b-b8bb1e-openshift-default-557bbf897c-phbbm" Oct 31 04:23:44.772443 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:44.772429 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61a59ba0-c79d-4942-8b8a-1493cb9e9182-istio-data\") pod \"e2e-ipp-b-b8bb1e-openshift-default-557bbf897c-phbbm\" (UID: \"61a59ba0-c79d-4942-8b8a-1493cb9e9182\") " pod="openshift-ingress/e2e-ipp-b-b8bb1e-openshift-default-557bbf897c-phbbm" Oct 31 04:23:44.876382 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:44.875640 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61a59ba0-c79d-4942-8b8a-1493cb9e9182-istio-token\") pod \"e2e-ipp-b-b8bb1e-openshift-default-557bbf897c-phbbm\" (UID: \"61a59ba0-c79d-4942-8b8a-1493cb9e9182\") " pod="openshift-ingress/e2e-ipp-b-b8bb1e-openshift-default-557bbf897c-phbbm" Oct 31 04:23:44.876382 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:44.875866 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61a59ba0-c79d-4942-8b8a-1493cb9e9182-istio-data\") pod \"e2e-ipp-b-b8bb1e-openshift-default-557bbf897c-phbbm\" (UID: \"61a59ba0-c79d-4942-8b8a-1493cb9e9182\") " pod="openshift-ingress/e2e-ipp-b-b8bb1e-openshift-default-557bbf897c-phbbm" Oct 31 04:23:44.876382 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:44.875903 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61a59ba0-c79d-4942-8b8a-1493cb9e9182-credential-socket\") pod \"e2e-ipp-b-b8bb1e-openshift-default-557bbf897c-phbbm\" (UID: \"61a59ba0-c79d-4942-8b8a-1493cb9e9182\") " pod="openshift-ingress/e2e-ipp-b-b8bb1e-openshift-default-557bbf897c-phbbm" Oct 31 04:23:44.876382 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:44.875953 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61a59ba0-c79d-4942-8b8a-1493cb9e9182-istiod-ca-cert\") pod \"e2e-ipp-b-b8bb1e-openshift-default-557bbf897c-phbbm\" (UID: \"61a59ba0-c79d-4942-8b8a-1493cb9e9182\") " pod="openshift-ingress/e2e-ipp-b-b8bb1e-openshift-default-557bbf897c-phbbm" Oct 31 04:23:44.876382 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:44.875978 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48gpj\" (UniqueName: \"kubernetes.io/projected/61a59ba0-c79d-4942-8b8a-1493cb9e9182-kube-api-access-48gpj\") pod \"e2e-ipp-b-b8bb1e-openshift-default-557bbf897c-phbbm\" (UID: \"61a59ba0-c79d-4942-8b8a-1493cb9e9182\") " pod="openshift-ingress/e2e-ipp-b-b8bb1e-openshift-default-557bbf897c-phbbm" Oct 31 04:23:44.876382 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:44.876067 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61a59ba0-c79d-4942-8b8a-1493cb9e9182-workload-certs\") pod \"e2e-ipp-b-b8bb1e-openshift-default-557bbf897c-phbbm\" (UID: \"61a59ba0-c79d-4942-8b8a-1493cb9e9182\") " pod="openshift-ingress/e2e-ipp-b-b8bb1e-openshift-default-557bbf897c-phbbm" Oct 31 04:23:44.876382 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:44.876096 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61a59ba0-c79d-4942-8b8a-1493cb9e9182-istio-podinfo\") pod \"e2e-ipp-b-b8bb1e-openshift-default-557bbf897c-phbbm\" (UID: \"61a59ba0-c79d-4942-8b8a-1493cb9e9182\") " pod="openshift-ingress/e2e-ipp-b-b8bb1e-openshift-default-557bbf897c-phbbm" Oct 31 04:23:44.876382 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:44.876140 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61a59ba0-c79d-4942-8b8a-1493cb9e9182-workload-socket\") pod \"e2e-ipp-b-b8bb1e-openshift-default-557bbf897c-phbbm\" (UID: \"61a59ba0-c79d-4942-8b8a-1493cb9e9182\") " pod="openshift-ingress/e2e-ipp-b-b8bb1e-openshift-default-557bbf897c-phbbm" Oct 31 04:23:44.876382 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:44.876173 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61a59ba0-c79d-4942-8b8a-1493cb9e9182-istio-envoy\") pod \"e2e-ipp-b-b8bb1e-openshift-default-557bbf897c-phbbm\" (UID: \"61a59ba0-c79d-4942-8b8a-1493cb9e9182\") " pod="openshift-ingress/e2e-ipp-b-b8bb1e-openshift-default-557bbf897c-phbbm" Oct 31 04:23:44.903398 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:44.882778 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61a59ba0-c79d-4942-8b8a-1493cb9e9182-workload-certs\") pod \"e2e-ipp-b-b8bb1e-openshift-default-557bbf897c-phbbm\" (UID: \"61a59ba0-c79d-4942-8b8a-1493cb9e9182\") " pod="openshift-ingress/e2e-ipp-b-b8bb1e-openshift-default-557bbf897c-phbbm" Oct 31 04:23:44.903398 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:44.898909 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61a59ba0-c79d-4942-8b8a-1493cb9e9182-istio-data\") pod \"e2e-ipp-b-b8bb1e-openshift-default-557bbf897c-phbbm\" (UID: \"61a59ba0-c79d-4942-8b8a-1493cb9e9182\") " pod="openshift-ingress/e2e-ipp-b-b8bb1e-openshift-default-557bbf897c-phbbm" Oct 31 04:23:44.903398 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:44.899220 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61a59ba0-c79d-4942-8b8a-1493cb9e9182-credential-socket\") pod \"e2e-ipp-b-b8bb1e-openshift-default-557bbf897c-phbbm\" (UID: \"61a59ba0-c79d-4942-8b8a-1493cb9e9182\") " pod="openshift-ingress/e2e-ipp-b-b8bb1e-openshift-default-557bbf897c-phbbm" Oct 31 04:23:44.903398 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:44.899548 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61a59ba0-c79d-4942-8b8a-1493cb9e9182-workload-socket\") pod \"e2e-ipp-b-b8bb1e-openshift-default-557bbf897c-phbbm\" (UID: \"61a59ba0-c79d-4942-8b8a-1493cb9e9182\") " pod="openshift-ingress/e2e-ipp-b-b8bb1e-openshift-default-557bbf897c-phbbm" Oct 31 04:23:44.903398 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:44.900260 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61a59ba0-c79d-4942-8b8a-1493cb9e9182-istiod-ca-cert\") pod \"e2e-ipp-b-b8bb1e-openshift-default-557bbf897c-phbbm\" (UID: \"61a59ba0-c79d-4942-8b8a-1493cb9e9182\") " pod="openshift-ingress/e2e-ipp-b-b8bb1e-openshift-default-557bbf897c-phbbm" Oct 31 04:23:44.923349 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:44.906761 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61a59ba0-c79d-4942-8b8a-1493cb9e9182-istio-envoy\") pod \"e2e-ipp-b-b8bb1e-openshift-default-557bbf897c-phbbm\" (UID: \"61a59ba0-c79d-4942-8b8a-1493cb9e9182\") " pod="openshift-ingress/e2e-ipp-b-b8bb1e-openshift-default-557bbf897c-phbbm" Oct 31 04:23:44.923349 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:44.923031 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48gpj\" (UniqueName: \"kubernetes.io/projected/61a59ba0-c79d-4942-8b8a-1493cb9e9182-kube-api-access-48gpj\") pod \"e2e-ipp-b-b8bb1e-openshift-default-557bbf897c-phbbm\" (UID: \"61a59ba0-c79d-4942-8b8a-1493cb9e9182\") " pod="openshift-ingress/e2e-ipp-b-b8bb1e-openshift-default-557bbf897c-phbbm" Oct 31 04:23:44.923581 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:44.923526 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61a59ba0-c79d-4942-8b8a-1493cb9e9182-istio-token\") pod \"e2e-ipp-b-b8bb1e-openshift-default-557bbf897c-phbbm\" (UID: \"61a59ba0-c79d-4942-8b8a-1493cb9e9182\") " pod="openshift-ingress/e2e-ipp-b-b8bb1e-openshift-default-557bbf897c-phbbm" Oct 31 04:23:44.934871 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:44.934835 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61a59ba0-c79d-4942-8b8a-1493cb9e9182-istio-podinfo\") pod \"e2e-ipp-b-b8bb1e-openshift-default-557bbf897c-phbbm\" (UID: \"61a59ba0-c79d-4942-8b8a-1493cb9e9182\") " pod="openshift-ingress/e2e-ipp-b-b8bb1e-openshift-default-557bbf897c-phbbm" Oct 31 04:23:44.982034 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:44.981696 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-b8bb1e-openshift-default-557bbf897c-phbbm" Oct 31 04:23:45.467688 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:45.467657 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-b8bb1e-openshift-default-557bbf897c-phbbm"] Oct 31 04:23:45.468142 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:23:45.468114 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61a59ba0_c79d_4942_8b8a_1493cb9e9182.slice/crio-0ab1a3671171bc76797fbb3ad195b86924ca53801677cd8e85ff90e5bdaf977d WatchSource:0}: Error finding container 0ab1a3671171bc76797fbb3ad195b86924ca53801677cd8e85ff90e5bdaf977d: Status 404 returned error can't find the container with id 0ab1a3671171bc76797fbb3ad195b86924ca53801677cd8e85ff90e5bdaf977d Oct 31 04:23:45.470780 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:45.470743 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:23:45.470887 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:45.470823 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:23:45.470887 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:45.470870 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:23:45.773836 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:45.773730 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b8bb1e-openshift-default-557bbf897c-phbbm" event={"ID":"61a59ba0-c79d-4942-8b8a-1493cb9e9182","Type":"ContainerStarted","Data":"4fd82931ebe743135b6a825d8f86ecdb8db0e895ee103a632b0318de61547947"} Oct 31 04:23:45.773836 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:45.773805 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b8bb1e-openshift-default-557bbf897c-phbbm" event={"ID":"61a59ba0-c79d-4942-8b8a-1493cb9e9182","Type":"ContainerStarted","Data":"0ab1a3671171bc76797fbb3ad195b86924ca53801677cd8e85ff90e5bdaf977d"} Oct 31 04:23:45.824235 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:45.824186 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-b8bb1e-openshift-default-557bbf897c-phbbm" podStartSLOduration=1.824172299 podStartE2EDuration="1.824172299s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:23:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:23:45.820436704 +0000 UTC m=+1173.534771338" watchObservedRunningTime="2026-10-31 04:23:45.824172299 +0000 UTC m=+1173.538506915" Oct 31 04:23:45.989472 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:45.989439 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-b8bb1e-openshift-default-557bbf897c-phbbm" Oct 31 04:23:46.993575 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:46.993540 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-b8bb1e-openshift-default-557bbf897c-phbbm" Oct 31 04:23:47.812296 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:47.812267 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-b8bb1e-openshift-default-557bbf897c-phbbm" Oct 31 04:23:47.813416 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:47.813353 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-b8bb1e-openshift-default-557bbf897c-phbbm" Oct 31 04:23:48.848440 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:48.848398 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-f91cd1-6688fc5c7-x47zn" podUID="a82b667c-713c-4cc1-88fd-07742c8fc3b8" containerName="payload-processing" containerID="cri-o://9a893490a679f1fba3910222551571feaa20673f7e7cbdbe2f1fa9cac19b9ddb" gracePeriod=25 Oct 31 04:23:48.914321 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:48.914280 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f91cd1-54fbfd87bc-4m89w" podUID="0b6efcad-2313-41dd-8f67-03bde20dcd34" containerName="payload-pre-processing" containerID="cri-o://21210ff755518b3d4a873c5acd780cd29d2224b628113ec9ea4a005dc366c9d5" gracePeriod=25 Oct 31 04:23:49.635490 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:49.635469 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f91cd1-6688fc5c7-x47zn" Oct 31 04:23:49.716812 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:49.716782 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a82b667c-713c-4cc1-88fd-07742c8fc3b8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a82b667c-713c-4cc1-88fd-07742c8fc3b8-plugins-config-volume\") pod \"a82b667c-713c-4cc1-88fd-07742c8fc3b8\" (UID: \"a82b667c-713c-4cc1-88fd-07742c8fc3b8\") " Oct 31 04:23:49.716953 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:49.716889 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a82b667c-713c-4cc1-88fd-07742c8fc3b8-kube-api-access-vfx2c\" (UniqueName: \"kubernetes.io/projected/a82b667c-713c-4cc1-88fd-07742c8fc3b8-kube-api-access-vfx2c\") pod \"a82b667c-713c-4cc1-88fd-07742c8fc3b8\" (UID: \"a82b667c-713c-4cc1-88fd-07742c8fc3b8\") " Oct 31 04:23:49.717646 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:49.717617 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a82b667c-713c-4cc1-88fd-07742c8fc3b8-plugins-config-volume" pod "a82b667c-713c-4cc1-88fd-07742c8fc3b8" (UID: "a82b667c-713c-4cc1-88fd-07742c8fc3b8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:23:49.719167 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:49.719144 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a82b667c-713c-4cc1-88fd-07742c8fc3b8-kube-api-access-vfx2c" pod "a82b667c-713c-4cc1-88fd-07742c8fc3b8" (UID: "a82b667c-713c-4cc1-88fd-07742c8fc3b8"). InnerVolumeSpecName "kube-api-access-vfx2c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:23:49.733774 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:49.733754 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f91cd1-54fbfd87bc-4m89w" Oct 31 04:23:49.818497 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:49.818422 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b6efcad-2313-41dd-8f67-03bde20dcd34-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b6efcad-2313-41dd-8f67-03bde20dcd34-plugins-config-volume\") pod \"0b6efcad-2313-41dd-8f67-03bde20dcd34\" (UID: \"0b6efcad-2313-41dd-8f67-03bde20dcd34\") " Oct 31 04:23:49.818497 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:49.818480 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b6efcad-2313-41dd-8f67-03bde20dcd34-kube-api-access-kplnr\" (UniqueName: \"kubernetes.io/projected/0b6efcad-2313-41dd-8f67-03bde20dcd34-kube-api-access-kplnr\") pod \"0b6efcad-2313-41dd-8f67-03bde20dcd34\" (UID: \"0b6efcad-2313-41dd-8f67-03bde20dcd34\") " Oct 31 04:23:49.818912 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:49.818887 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b6efcad-2313-41dd-8f67-03bde20dcd34-plugins-config-volume" pod "0b6efcad-2313-41dd-8f67-03bde20dcd34" (UID: "0b6efcad-2313-41dd-8f67-03bde20dcd34"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:23:49.819012 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:49.818913 4376 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a82b667c-713c-4cc1-88fd-07742c8fc3b8-plugins-config-volume\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:23:49.819012 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:49.818925 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vfx2c\" (UniqueName: \"kubernetes.io/projected/a82b667c-713c-4cc1-88fd-07742c8fc3b8-kube-api-access-vfx2c\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:23:49.821818 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:49.821776 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b6efcad-2313-41dd-8f67-03bde20dcd34-kube-api-access-kplnr" pod "0b6efcad-2313-41dd-8f67-03bde20dcd34" (UID: "0b6efcad-2313-41dd-8f67-03bde20dcd34"). InnerVolumeSpecName "kube-api-access-kplnr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:23:49.826494 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:49.826467 4376 generic.go:356] "Generic (PLEG): container finished" podID="0b6efcad-2313-41dd-8f67-03bde20dcd34" containerID="21210ff755518b3d4a873c5acd780cd29d2224b628113ec9ea4a005dc366c9d5" exitCode=0 Oct 31 04:23:49.826604 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:49.826588 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f91cd1-54fbfd87bc-4m89w" Oct 31 04:23:49.827041 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:49.827017 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f91cd1-54fbfd87bc-4m89w" event={"ID":"0b6efcad-2313-41dd-8f67-03bde20dcd34","Type":"ContainerDied","Data":"21210ff755518b3d4a873c5acd780cd29d2224b628113ec9ea4a005dc366c9d5"} Oct 31 04:23:49.827134 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:49.827074 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f91cd1-54fbfd87bc-4m89w" event={"ID":"0b6efcad-2313-41dd-8f67-03bde20dcd34","Type":"ContainerDied","Data":"f44b56d0d02f480ae506ad2603aadad8c1c98fb3309f92ebbdf609a967b18ee0"} Oct 31 04:23:49.827134 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:49.827107 4376 scope.go:122] "RemoveContainer" containerID="21210ff755518b3d4a873c5acd780cd29d2224b628113ec9ea4a005dc366c9d5" Oct 31 04:23:49.835109 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:49.835083 4376 generic.go:356] "Generic (PLEG): container finished" podID="a82b667c-713c-4cc1-88fd-07742c8fc3b8" containerID="9a893490a679f1fba3910222551571feaa20673f7e7cbdbe2f1fa9cac19b9ddb" exitCode=0 Oct 31 04:23:49.835203 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:49.835129 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f91cd1-6688fc5c7-x47zn" Oct 31 04:23:49.835203 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:49.835173 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f91cd1-6688fc5c7-x47zn" event={"ID":"a82b667c-713c-4cc1-88fd-07742c8fc3b8","Type":"ContainerDied","Data":"9a893490a679f1fba3910222551571feaa20673f7e7cbdbe2f1fa9cac19b9ddb"} Oct 31 04:23:49.835341 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:49.835240 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f91cd1-6688fc5c7-x47zn" event={"ID":"a82b667c-713c-4cc1-88fd-07742c8fc3b8","Type":"ContainerDied","Data":"d43e4a1430888b87704d79c2c533d006a6a466001d26cc1d99fdb5427406c93e"} Oct 31 04:23:49.904816 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:49.902888 4376 scope.go:122] "RemoveContainer" containerID="21210ff755518b3d4a873c5acd780cd29d2224b628113ec9ea4a005dc366c9d5" Oct 31 04:23:49.904816 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:23:49.904621 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21210ff755518b3d4a873c5acd780cd29d2224b628113ec9ea4a005dc366c9d5\": container with ID starting with 21210ff755518b3d4a873c5acd780cd29d2224b628113ec9ea4a005dc366c9d5 not found: ID does not exist" containerID="21210ff755518b3d4a873c5acd780cd29d2224b628113ec9ea4a005dc366c9d5" Oct 31 04:23:49.904816 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:49.904661 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21210ff755518b3d4a873c5acd780cd29d2224b628113ec9ea4a005dc366c9d5"} err="failed to get container status \"21210ff755518b3d4a873c5acd780cd29d2224b628113ec9ea4a005dc366c9d5\": rpc error: code = NotFound desc = could not find container \"21210ff755518b3d4a873c5acd780cd29d2224b628113ec9ea4a005dc366c9d5\": container with ID starting with 21210ff755518b3d4a873c5acd780cd29d2224b628113ec9ea4a005dc366c9d5 not found: ID does not exist" Oct 31 04:23:49.904816 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:49.904685 4376 scope.go:122] "RemoveContainer" containerID="9a893490a679f1fba3910222551571feaa20673f7e7cbdbe2f1fa9cac19b9ddb" Oct 31 04:23:49.915417 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:49.915344 4376 scope.go:122] "RemoveContainer" containerID="9a893490a679f1fba3910222551571feaa20673f7e7cbdbe2f1fa9cac19b9ddb" Oct 31 04:23:49.915637 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:23:49.915617 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a893490a679f1fba3910222551571feaa20673f7e7cbdbe2f1fa9cac19b9ddb\": container with ID starting with 9a893490a679f1fba3910222551571feaa20673f7e7cbdbe2f1fa9cac19b9ddb not found: ID does not exist" containerID="9a893490a679f1fba3910222551571feaa20673f7e7cbdbe2f1fa9cac19b9ddb" Oct 31 04:23:49.915687 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:49.915644 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a893490a679f1fba3910222551571feaa20673f7e7cbdbe2f1fa9cac19b9ddb"} err="failed to get container status \"9a893490a679f1fba3910222551571feaa20673f7e7cbdbe2f1fa9cac19b9ddb\": rpc error: code = NotFound desc = could not find container \"9a893490a679f1fba3910222551571feaa20673f7e7cbdbe2f1fa9cac19b9ddb\": container with ID starting with 9a893490a679f1fba3910222551571feaa20673f7e7cbdbe2f1fa9cac19b9ddb not found: ID does not exist" Oct 31 04:23:49.920499 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:49.920265 4376 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b6efcad-2313-41dd-8f67-03bde20dcd34-plugins-config-volume\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:23:49.920499 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:49.920291 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kplnr\" (UniqueName: \"kubernetes.io/projected/0b6efcad-2313-41dd-8f67-03bde20dcd34-kube-api-access-kplnr\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:23:49.997815 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:49.997432 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f91cd1-54fbfd87bc-4m89w"] Oct 31 04:23:49.999713 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:49.999687 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f91cd1-54fbfd87bc-4m89w"] Oct 31 04:23:50.002902 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:50.002880 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ceba51-bb5fb7d99-nbdsj"] Oct 31 04:23:50.006748 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:50.006728 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ceba51-5f4d757d95-ss4kp"] Oct 31 04:23:50.009659 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:50.009636 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f91cd1-6688fc5c7-x47zn"] Oct 31 04:23:50.012921 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:50.012900 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f91cd1-6688fc5c7-x47zn"] Oct 31 04:23:50.508987 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:50.508956 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b6efcad-2313-41dd-8f67-03bde20dcd34" path="/var/lib/kubelet/pods/0b6efcad-2313-41dd-8f67-03bde20dcd34/volumes" Oct 31 04:23:50.509443 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:50.509428 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a82b667c-713c-4cc1-88fd-07742c8fc3b8" path="/var/lib/kubelet/pods/a82b667c-713c-4cc1-88fd-07742c8fc3b8/volumes" Oct 31 04:23:51.511149 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:51.511112 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b8bb1e-84466f94c-pjkmm"] Oct 31 04:23:51.511929 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:51.511909 4376 cpu_manager.go:414] "containerMap: removing container" podUID="a82b667c-713c-4cc1-88fd-07742c8fc3b8" containerName="payload-processing" Oct 31 04:23:51.511929 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:51.511928 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a82b667c-713c-4cc1-88fd-07742c8fc3b8" containerName="payload-processing" Oct 31 04:23:51.512062 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:51.511941 4376 cpu_manager.go:414] "containerMap: removing container" podUID="0b6efcad-2313-41dd-8f67-03bde20dcd34" containerName="payload-pre-processing" Oct 31 04:23:51.512062 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:51.511946 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b6efcad-2313-41dd-8f67-03bde20dcd34" containerName="payload-pre-processing" Oct 31 04:23:51.512181 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:51.512095 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="a82b667c-713c-4cc1-88fd-07742c8fc3b8" containerName="payload-processing" Oct 31 04:23:51.512181 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:51.512111 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b6efcad-2313-41dd-8f67-03bde20dcd34" containerName="payload-pre-processing" Oct 31 04:23:51.517097 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:51.517073 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b8bb1e-84466f94c-pjkmm" Oct 31 04:23:51.554352 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:51.538014 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b8bb1e-84466f94c-pjkmm"] Oct 31 04:23:51.554605 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:51.548042 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-b8bb1e-dockercfg-4m5zs\"" Oct 31 04:23:51.554717 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:51.548121 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-b8bb1e\"" Oct 31 04:23:51.577083 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:51.577050 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b8bb1e-8bfbcf887-kchch"] Oct 31 04:23:51.582990 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:51.582964 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b8bb1e-8bfbcf887-kchch" Oct 31 04:23:51.595966 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:51.595939 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b8bb1e-8bfbcf887-kchch"] Oct 31 04:23:51.663673 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:51.663542 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlfwr\" (UniqueName: \"kubernetes.io/projected/4c454206-433b-4b01-92cd-fe229ee6a4de-kube-api-access-jlfwr\") pod \"payload-processing-e2e-ipp-b-b8bb1e-84466f94c-pjkmm\" (UID: \"4c454206-433b-4b01-92cd-fe229ee6a4de\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b8bb1e-84466f94c-pjkmm" Oct 31 04:23:51.663844 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:51.663679 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c454206-433b-4b01-92cd-fe229ee6a4de-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-b8bb1e-84466f94c-pjkmm\" (UID: \"4c454206-433b-4b01-92cd-fe229ee6a4de\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b8bb1e-84466f94c-pjkmm" Oct 31 04:23:51.714372 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:51.710709 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-801218-7bd7574847-2fgt5" Oct 31 04:23:51.719152 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:51.718516 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-801218-5c988b64cf-w2d4l" Oct 31 04:23:51.811357 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:51.778406 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlfwr\" (UniqueName: \"kubernetes.io/projected/4c454206-433b-4b01-92cd-fe229ee6a4de-kube-api-access-jlfwr\") pod \"payload-processing-e2e-ipp-b-b8bb1e-84466f94c-pjkmm\" (UID: \"4c454206-433b-4b01-92cd-fe229ee6a4de\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b8bb1e-84466f94c-pjkmm" Oct 31 04:23:51.811357 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:51.778511 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnwg5\" (UniqueName: \"kubernetes.io/projected/ff8b6e9e-7dce-41d2-95be-4e487d8fd427-kube-api-access-hnwg5\") pod \"payload-pre-processing-e2e-ipp-b-b8bb1e-8bfbcf887-kchch\" (UID: \"ff8b6e9e-7dce-41d2-95be-4e487d8fd427\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b8bb1e-8bfbcf887-kchch" Oct 31 04:23:51.811357 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:51.778620 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c454206-433b-4b01-92cd-fe229ee6a4de-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-b8bb1e-84466f94c-pjkmm\" (UID: \"4c454206-433b-4b01-92cd-fe229ee6a4de\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b8bb1e-84466f94c-pjkmm" Oct 31 04:23:51.811357 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:51.778662 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff8b6e9e-7dce-41d2-95be-4e487d8fd427-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-b8bb1e-8bfbcf887-kchch\" (UID: \"ff8b6e9e-7dce-41d2-95be-4e487d8fd427\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b8bb1e-8bfbcf887-kchch" Oct 31 04:23:51.811357 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:51.780662 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c454206-433b-4b01-92cd-fe229ee6a4de-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-b8bb1e-84466f94c-pjkmm\" (UID: \"4c454206-433b-4b01-92cd-fe229ee6a4de\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b8bb1e-84466f94c-pjkmm" Oct 31 04:23:51.994447 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:51.956387 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnwg5\" (UniqueName: \"kubernetes.io/projected/ff8b6e9e-7dce-41d2-95be-4e487d8fd427-kube-api-access-hnwg5\") pod \"payload-pre-processing-e2e-ipp-b-b8bb1e-8bfbcf887-kchch\" (UID: \"ff8b6e9e-7dce-41d2-95be-4e487d8fd427\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b8bb1e-8bfbcf887-kchch" Oct 31 04:23:51.994447 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:51.956482 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff8b6e9e-7dce-41d2-95be-4e487d8fd427-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-b8bb1e-8bfbcf887-kchch\" (UID: \"ff8b6e9e-7dce-41d2-95be-4e487d8fd427\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b8bb1e-8bfbcf887-kchch" Oct 31 04:23:51.994447 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:51.993610 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff8b6e9e-7dce-41d2-95be-4e487d8fd427-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-b8bb1e-8bfbcf887-kchch\" (UID: \"ff8b6e9e-7dce-41d2-95be-4e487d8fd427\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b8bb1e-8bfbcf887-kchch" Oct 31 04:23:52.003352 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:52.000293 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnwg5\" (UniqueName: \"kubernetes.io/projected/ff8b6e9e-7dce-41d2-95be-4e487d8fd427-kube-api-access-hnwg5\") pod \"payload-pre-processing-e2e-ipp-b-b8bb1e-8bfbcf887-kchch\" (UID: \"ff8b6e9e-7dce-41d2-95be-4e487d8fd427\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b8bb1e-8bfbcf887-kchch" Oct 31 04:23:52.007369 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:52.006938 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlfwr\" (UniqueName: \"kubernetes.io/projected/4c454206-433b-4b01-92cd-fe229ee6a4de-kube-api-access-jlfwr\") pod \"payload-processing-e2e-ipp-b-b8bb1e-84466f94c-pjkmm\" (UID: \"4c454206-433b-4b01-92cd-fe229ee6a4de\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b8bb1e-84466f94c-pjkmm" Oct 31 04:23:52.135348 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:52.135299 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b8bb1e-84466f94c-pjkmm" Oct 31 04:23:52.207561 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:52.207529 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b8bb1e-8bfbcf887-kchch" Oct 31 04:23:52.465206 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:23:52.465173 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c454206_433b_4b01_92cd_fe229ee6a4de.slice/crio-aaa92d3a7afbf9c31eabd9fa60a2d58308717ad8446552b2a978eb8a619df914 WatchSource:0}: Error finding container aaa92d3a7afbf9c31eabd9fa60a2d58308717ad8446552b2a978eb8a619df914: Status 404 returned error can't find the container with id aaa92d3a7afbf9c31eabd9fa60a2d58308717ad8446552b2a978eb8a619df914 Oct 31 04:23:52.469144 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:52.469119 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b8bb1e-84466f94c-pjkmm"] Oct 31 04:23:52.549133 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:52.549102 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b8bb1e-8bfbcf887-kchch"] Oct 31 04:23:52.557881 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:23:52.557854 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff8b6e9e_7dce_41d2_95be_4e487d8fd427.slice/crio-2cd652c2865d5b82b93449a7d308e253777fd30591f00a3e3ae21e3e0b239884 WatchSource:0}: Error finding container 2cd652c2865d5b82b93449a7d308e253777fd30591f00a3e3ae21e3e0b239884: Status 404 returned error can't find the container with id 2cd652c2865d5b82b93449a7d308e253777fd30591f00a3e3ae21e3e0b239884 Oct 31 04:23:53.022469 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:53.014364 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b8bb1e-84466f94c-pjkmm" event={"ID":"4c454206-433b-4b01-92cd-fe229ee6a4de","Type":"ContainerStarted","Data":"694e697cd50dcb0a7656d8013c1e7300b471e9c89bb1761a7bf85b1086b1f7a5"} Oct 31 04:23:53.022469 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:53.014437 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b8bb1e-84466f94c-pjkmm" event={"ID":"4c454206-433b-4b01-92cd-fe229ee6a4de","Type":"ContainerStarted","Data":"aaa92d3a7afbf9c31eabd9fa60a2d58308717ad8446552b2a978eb8a619df914"} Oct 31 04:23:53.022469 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:53.014471 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-b8bb1e-84466f94c-pjkmm" Oct 31 04:23:53.022469 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:53.016220 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b8bb1e-8bfbcf887-kchch" event={"ID":"ff8b6e9e-7dce-41d2-95be-4e487d8fd427","Type":"ContainerStarted","Data":"9aa9d5588dd8ba1ee593665deea69c189aa565c7108561293ff67d8bdb6b95af"} Oct 31 04:23:53.022469 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:53.016282 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b8bb1e-8bfbcf887-kchch" Oct 31 04:23:53.022469 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:53.016307 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b8bb1e-8bfbcf887-kchch" event={"ID":"ff8b6e9e-7dce-41d2-95be-4e487d8fd427","Type":"ContainerStarted","Data":"2cd652c2865d5b82b93449a7d308e253777fd30591f00a3e3ae21e3e0b239884"} Oct 31 04:23:53.058275 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:53.058216 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-b8bb1e-84466f94c-pjkmm" podStartSLOduration=2.058202678 podStartE2EDuration="2.058202678s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:23:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:23:53.056571689 +0000 UTC m=+1180.770906327" watchObservedRunningTime="2026-10-31 04:23:53.058202678 +0000 UTC m=+1180.772537294" Oct 31 04:23:53.099166 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:53.099109 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b8bb1e-8bfbcf887-kchch" podStartSLOduration=2.099096216 podStartE2EDuration="2.099096216s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:23:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:23:53.097465301 +0000 UTC m=+1180.811799980" watchObservedRunningTime="2026-10-31 04:23:53.099096216 +0000 UTC m=+1180.813430833" Oct 31 04:23:55.003552 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:55.003500 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-ceba51-bb5fb7d99-nbdsj" podUID="567e1919-44a9-4c76-ab90-bfdcfb1aef01" containerName="payload-processing" containerID="cri-o://fcba94b185a7b6b69399cbed9c0440ecc0ffda1e0a653ee564d17041175a1395" gracePeriod=25 Oct 31 04:23:55.006985 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:55.006951 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ceba51-5f4d757d95-ss4kp" podUID="407cfc33-fd88-4236-a178-52cfddd54cdc" containerName="payload-pre-processing" containerID="cri-o://c2f9b74936743bf0bd3e5c56851b49d8dbfd13c7e15dfdec16da754661138afd" gracePeriod=25 Oct 31 04:23:55.323076 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:55.323047 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ceba51-5f4d757d95-ss4kp" Oct 31 04:23:55.383460 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:55.372510 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/407cfc33-fd88-4236-a178-52cfddd54cdc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/407cfc33-fd88-4236-a178-52cfddd54cdc-plugins-config-volume\") pod \"407cfc33-fd88-4236-a178-52cfddd54cdc\" (UID: \"407cfc33-fd88-4236-a178-52cfddd54cdc\") " Oct 31 04:23:55.383460 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:55.372780 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/407cfc33-fd88-4236-a178-52cfddd54cdc-kube-api-access-j6nml\" (UniqueName: \"kubernetes.io/projected/407cfc33-fd88-4236-a178-52cfddd54cdc-kube-api-access-j6nml\") pod \"407cfc33-fd88-4236-a178-52cfddd54cdc\" (UID: \"407cfc33-fd88-4236-a178-52cfddd54cdc\") " Oct 31 04:23:55.383460 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:55.383204 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/407cfc33-fd88-4236-a178-52cfddd54cdc-plugins-config-volume" pod "407cfc33-fd88-4236-a178-52cfddd54cdc" (UID: "407cfc33-fd88-4236-a178-52cfddd54cdc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:23:55.394353 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:55.392076 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-ceba51-bb5fb7d99-nbdsj" Oct 31 04:23:55.430028 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:55.429983 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/407cfc33-fd88-4236-a178-52cfddd54cdc-kube-api-access-j6nml" pod "407cfc33-fd88-4236-a178-52cfddd54cdc" (UID: "407cfc33-fd88-4236-a178-52cfddd54cdc"). InnerVolumeSpecName "kube-api-access-j6nml". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:23:55.480584 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:55.478860 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/567e1919-44a9-4c76-ab90-bfdcfb1aef01-kube-api-access-768mf\" (UniqueName: \"kubernetes.io/projected/567e1919-44a9-4c76-ab90-bfdcfb1aef01-kube-api-access-768mf\") pod \"567e1919-44a9-4c76-ab90-bfdcfb1aef01\" (UID: \"567e1919-44a9-4c76-ab90-bfdcfb1aef01\") " Oct 31 04:23:55.480584 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:55.479024 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/567e1919-44a9-4c76-ab90-bfdcfb1aef01-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/567e1919-44a9-4c76-ab90-bfdcfb1aef01-plugins-config-volume\") pod \"567e1919-44a9-4c76-ab90-bfdcfb1aef01\" (UID: \"567e1919-44a9-4c76-ab90-bfdcfb1aef01\") " Oct 31 04:23:55.480584 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:55.479645 4376 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/407cfc33-fd88-4236-a178-52cfddd54cdc-plugins-config-volume\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:23:55.480584 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:55.479670 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j6nml\" (UniqueName: \"kubernetes.io/projected/407cfc33-fd88-4236-a178-52cfddd54cdc-kube-api-access-j6nml\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:23:55.480584 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:55.480534 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/567e1919-44a9-4c76-ab90-bfdcfb1aef01-plugins-config-volume" pod "567e1919-44a9-4c76-ab90-bfdcfb1aef01" (UID: "567e1919-44a9-4c76-ab90-bfdcfb1aef01"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:23:55.487412 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:55.487065 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/567e1919-44a9-4c76-ab90-bfdcfb1aef01-kube-api-access-768mf" pod "567e1919-44a9-4c76-ab90-bfdcfb1aef01" (UID: "567e1919-44a9-4c76-ab90-bfdcfb1aef01"). InnerVolumeSpecName "kube-api-access-768mf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:23:55.581341 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:55.581248 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-768mf\" (UniqueName: \"kubernetes.io/projected/567e1919-44a9-4c76-ab90-bfdcfb1aef01-kube-api-access-768mf\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:23:55.581341 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:55.581283 4376 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/567e1919-44a9-4c76-ab90-bfdcfb1aef01-plugins-config-volume\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:23:56.040706 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:56.040671 4376 generic.go:356] "Generic (PLEG): container finished" podID="567e1919-44a9-4c76-ab90-bfdcfb1aef01" containerID="fcba94b185a7b6b69399cbed9c0440ecc0ffda1e0a653ee564d17041175a1395" exitCode=0 Oct 31 04:23:56.041160 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:56.040720 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-ceba51-bb5fb7d99-nbdsj" Oct 31 04:23:56.041160 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:56.040756 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-ceba51-bb5fb7d99-nbdsj" event={"ID":"567e1919-44a9-4c76-ab90-bfdcfb1aef01","Type":"ContainerDied","Data":"fcba94b185a7b6b69399cbed9c0440ecc0ffda1e0a653ee564d17041175a1395"} Oct 31 04:23:56.041160 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:56.040808 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-ceba51-bb5fb7d99-nbdsj" event={"ID":"567e1919-44a9-4c76-ab90-bfdcfb1aef01","Type":"ContainerDied","Data":"6796d3971d9e36c1dc8c9ca2ae914de8aba64f537b904c0943e06ed35d48b641"} Oct 31 04:23:56.041160 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:56.040836 4376 scope.go:122] "RemoveContainer" containerID="fcba94b185a7b6b69399cbed9c0440ecc0ffda1e0a653ee564d17041175a1395" Oct 31 04:23:56.042201 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:56.042180 4376 generic.go:356] "Generic (PLEG): container finished" podID="407cfc33-fd88-4236-a178-52cfddd54cdc" containerID="c2f9b74936743bf0bd3e5c56851b49d8dbfd13c7e15dfdec16da754661138afd" exitCode=0 Oct 31 04:23:56.042277 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:56.042219 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ceba51-5f4d757d95-ss4kp" Oct 31 04:23:56.042374 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:56.042265 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ceba51-5f4d757d95-ss4kp" event={"ID":"407cfc33-fd88-4236-a178-52cfddd54cdc","Type":"ContainerDied","Data":"c2f9b74936743bf0bd3e5c56851b49d8dbfd13c7e15dfdec16da754661138afd"} Oct 31 04:23:56.042374 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:56.042315 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ceba51-5f4d757d95-ss4kp" event={"ID":"407cfc33-fd88-4236-a178-52cfddd54cdc","Type":"ContainerDied","Data":"7c4ff67290e37a425dba21b28fc78019a7281d5abe2d51a9a7083964adae915e"} Oct 31 04:23:56.045364 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:56.045316 4376 generic.go:356] "Generic (PLEG): container finished" podID="3ade7461-87d8-42ea-a721-512bae1f5f65" containerID="137eadc48924c8f862ed0e8bb4ad99217a5739ed1579e0b78d0f8cdfab1423d3" exitCode=0 Oct 31 04:23:56.045459 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:56.045369 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3430f6-openshift-default-595d4f6b8c-ltm6g" event={"ID":"3ade7461-87d8-42ea-a721-512bae1f5f65","Type":"ContainerDied","Data":"137eadc48924c8f862ed0e8bb4ad99217a5739ed1579e0b78d0f8cdfab1423d3"} Oct 31 04:23:56.045855 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:56.045836 4376 scope.go:122] "RemoveContainer" containerID="137eadc48924c8f862ed0e8bb4ad99217a5739ed1579e0b78d0f8cdfab1423d3" Oct 31 04:23:56.051258 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:56.051184 4376 scope.go:122] "RemoveContainer" containerID="fcba94b185a7b6b69399cbed9c0440ecc0ffda1e0a653ee564d17041175a1395" Oct 31 04:23:56.051508 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:56.051478 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:23:56.051615 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:56.051547 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:23:56.051615 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:23:56.051487 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fcba94b185a7b6b69399cbed9c0440ecc0ffda1e0a653ee564d17041175a1395\": container with ID starting with fcba94b185a7b6b69399cbed9c0440ecc0ffda1e0a653ee564d17041175a1395 not found: ID does not exist" containerID="fcba94b185a7b6b69399cbed9c0440ecc0ffda1e0a653ee564d17041175a1395" Oct 31 04:23:56.051615 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:56.051582 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:23:56.051615 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:56.051586 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fcba94b185a7b6b69399cbed9c0440ecc0ffda1e0a653ee564d17041175a1395"} err="failed to get container status \"fcba94b185a7b6b69399cbed9c0440ecc0ffda1e0a653ee564d17041175a1395\": rpc error: code = NotFound desc = could not find container \"fcba94b185a7b6b69399cbed9c0440ecc0ffda1e0a653ee564d17041175a1395\": container with ID starting with fcba94b185a7b6b69399cbed9c0440ecc0ffda1e0a653ee564d17041175a1395 not found: ID does not exist" Oct 31 04:23:56.051615 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:56.051607 4376 scope.go:122] "RemoveContainer" containerID="c2f9b74936743bf0bd3e5c56851b49d8dbfd13c7e15dfdec16da754661138afd" Oct 31 04:23:56.064343 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:56.064310 4376 scope.go:122] "RemoveContainer" containerID="c2f9b74936743bf0bd3e5c56851b49d8dbfd13c7e15dfdec16da754661138afd" Oct 31 04:23:56.064599 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:23:56.064582 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2f9b74936743bf0bd3e5c56851b49d8dbfd13c7e15dfdec16da754661138afd\": container with ID starting with c2f9b74936743bf0bd3e5c56851b49d8dbfd13c7e15dfdec16da754661138afd not found: ID does not exist" containerID="c2f9b74936743bf0bd3e5c56851b49d8dbfd13c7e15dfdec16da754661138afd" Oct 31 04:23:56.064660 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:56.064605 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2f9b74936743bf0bd3e5c56851b49d8dbfd13c7e15dfdec16da754661138afd"} err="failed to get container status \"c2f9b74936743bf0bd3e5c56851b49d8dbfd13c7e15dfdec16da754661138afd\": rpc error: code = NotFound desc = could not find container \"c2f9b74936743bf0bd3e5c56851b49d8dbfd13c7e15dfdec16da754661138afd\": container with ID starting with c2f9b74936743bf0bd3e5c56851b49d8dbfd13c7e15dfdec16da754661138afd not found: ID does not exist" Oct 31 04:23:56.150607 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:56.150561 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ceba51-bb5fb7d99-nbdsj"] Oct 31 04:23:56.156626 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:56.156363 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ceba51-bb5fb7d99-nbdsj"] Oct 31 04:23:56.183924 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:56.183892 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ceba51-5f4d757d95-ss4kp"] Oct 31 04:23:56.188970 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:56.188948 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ceba51-5f4d757d95-ss4kp"] Oct 31 04:23:56.445743 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:56.445710 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="407cfc33-fd88-4236-a178-52cfddd54cdc" path="/var/lib/kubelet/pods/407cfc33-fd88-4236-a178-52cfddd54cdc/volumes" Oct 31 04:23:56.446152 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:56.446136 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="567e1919-44a9-4c76-ab90-bfdcfb1aef01" path="/var/lib/kubelet/pods/567e1919-44a9-4c76-ab90-bfdcfb1aef01/volumes" Oct 31 04:23:57.071583 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:57.071552 4376 generic.go:356] "Generic (PLEG): container finished" podID="fd4632c9-069d-466e-b00a-c75aa4976d0d" containerID="3dd4da46f8a97e21d8f1fa204272a8ef9df80bb2368eb2ad619cd0b2063cfbb8" exitCode=0 Oct 31 04:23:57.072021 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:57.071626 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f91cd1-openshift-default-556cb85755-v94km" event={"ID":"fd4632c9-069d-466e-b00a-c75aa4976d0d","Type":"ContainerDied","Data":"3dd4da46f8a97e21d8f1fa204272a8ef9df80bb2368eb2ad619cd0b2063cfbb8"} Oct 31 04:23:57.072101 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:57.072076 4376 scope.go:122] "RemoveContainer" containerID="3dd4da46f8a97e21d8f1fa204272a8ef9df80bb2368eb2ad619cd0b2063cfbb8" Oct 31 04:23:57.073746 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:57.073727 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3430f6-openshift-default-595d4f6b8c-ltm6g" event={"ID":"3ade7461-87d8-42ea-a721-512bae1f5f65","Type":"ContainerStarted","Data":"b8ee493ca4d59dc205b420f6ff1185cc92bd561067da0ac75193342d13b2504a"} Oct 31 04:23:57.073851 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:57.073815 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:23:57.073927 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:57.073910 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:23:57.073972 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:57.073957 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:23:57.580389 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:57.580271 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-3430f6-openshift-default-595d4f6b8c-ltm6g" Oct 31 04:23:57.580788 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:57.580767 4376 patch_prober.go:29] interesting pod/e2e-ipp-a-3430f6-openshift-default-595d4f6b8c-ltm6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Oct 31 04:23:57.581457 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:57.581426 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3430f6-openshift-default-595d4f6b8c-ltm6g" podUID="3ade7461-87d8-42ea-a721-512bae1f5f65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Oct 31 04:23:58.096653 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:58.096619 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f91cd1-openshift-default-556cb85755-v94km" event={"ID":"fd4632c9-069d-466e-b00a-c75aa4976d0d","Type":"ContainerStarted","Data":"ba339f16daedb9c2675cf1aac882661a7c9e2bdb4351accdfd30732a6e6f240d"} Oct 31 04:23:58.591351 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:58.589758 4376 patch_prober.go:29] interesting pod/e2e-ipp-a-3430f6-openshift-default-595d4f6b8c-ltm6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Oct 31 04:23:58.591351 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:58.589997 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3430f6-openshift-default-595d4f6b8c-ltm6g" podUID="3ade7461-87d8-42ea-a721-512bae1f5f65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Oct 31 04:23:59.063352 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:59.060249 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-f91cd1-openshift-default-556cb85755-v94km" Oct 31 04:23:59.063352 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:59.062434 4376 patch_prober.go:29] interesting pod/e2e-ipp-a-f91cd1-openshift-default-556cb85755-v94km container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Oct 31 04:23:59.063352 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:59.062485 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f91cd1-openshift-default-556cb85755-v94km" podUID="fd4632c9-069d-466e-b00a-c75aa4976d0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Oct 31 04:23:59.579353 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:59.578672 4376 patch_prober.go:29] interesting pod/e2e-ipp-a-3430f6-openshift-default-595d4f6b8c-ltm6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Oct 31 04:23:59.579353 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:59.578730 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3430f6-openshift-default-595d4f6b8c-ltm6g" podUID="3ade7461-87d8-42ea-a721-512bae1f5f65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Oct 31 04:23:59.793362 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:23:59.792525 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-3430f6-openshift-default-595d4f6b8c-ltm6g" Oct 31 04:24:00.086381 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:00.081932 4376 patch_prober.go:29] interesting pod/e2e-ipp-a-f91cd1-openshift-default-556cb85755-v94km container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Oct 31 04:24:00.086381 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:00.081994 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f91cd1-openshift-default-556cb85755-v94km" podUID="fd4632c9-069d-466e-b00a-c75aa4976d0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Oct 31 04:24:00.588317 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:00.587848 4376 patch_prober.go:29] interesting pod/e2e-ipp-a-3430f6-openshift-default-595d4f6b8c-ltm6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Oct 31 04:24:00.588317 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:00.587906 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3430f6-openshift-default-595d4f6b8c-ltm6g" podUID="3ade7461-87d8-42ea-a721-512bae1f5f65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Oct 31 04:24:00.800721 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:00.800680 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-f91cd1-openshift-default-556cb85755-v94km" Oct 31 04:24:01.057810 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:01.053681 4376 patch_prober.go:29] interesting pod/e2e-ipp-a-f91cd1-openshift-default-556cb85755-v94km container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Oct 31 04:24:01.057810 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:01.053742 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f91cd1-openshift-default-556cb85755-v94km" podUID="fd4632c9-069d-466e-b00a-c75aa4976d0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Oct 31 04:24:01.580828 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:01.580385 4376 patch_prober.go:29] interesting pod/e2e-ipp-a-3430f6-openshift-default-595d4f6b8c-ltm6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Oct 31 04:24:01.580828 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:01.580449 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3430f6-openshift-default-595d4f6b8c-ltm6g" podUID="3ade7461-87d8-42ea-a721-512bae1f5f65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Oct 31 04:24:01.743480 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:01.743407 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3430f6-84fd4c6898-7rzjt"] Oct 31 04:24:01.745128 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:01.745099 4376 cpu_manager.go:414] "containerMap: removing container" podUID="407cfc33-fd88-4236-a178-52cfddd54cdc" containerName="payload-pre-processing" Oct 31 04:24:01.745292 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:01.745278 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="407cfc33-fd88-4236-a178-52cfddd54cdc" containerName="payload-pre-processing" Oct 31 04:24:01.745425 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:01.745413 4376 cpu_manager.go:414] "containerMap: removing container" podUID="567e1919-44a9-4c76-ab90-bfdcfb1aef01" containerName="payload-processing" Oct 31 04:24:01.745512 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:01.745502 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="567e1919-44a9-4c76-ab90-bfdcfb1aef01" containerName="payload-processing" Oct 31 04:24:01.745769 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:01.745756 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="407cfc33-fd88-4236-a178-52cfddd54cdc" containerName="payload-pre-processing" Oct 31 04:24:01.745864 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:01.745855 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="567e1919-44a9-4c76-ab90-bfdcfb1aef01" containerName="payload-processing" Oct 31 04:24:01.805918 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:01.805883 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3430f6-84fd4c6898-7rzjt"] Oct 31 04:24:01.806264 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:01.806246 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3430f6-84fd4c6898-7rzjt" Oct 31 04:24:01.832751 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:01.832640 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-3430f6\"" Oct 31 04:24:02.034350 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:02.031406 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-597d9\" (UniqueName: \"kubernetes.io/projected/38a822e5-1a32-4126-b19a-1264d09c66cc-kube-api-access-597d9\") pod \"payload-processing-e2e-ipp-a-3430f6-84fd4c6898-7rzjt\" (UID: \"38a822e5-1a32-4126-b19a-1264d09c66cc\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3430f6-84fd4c6898-7rzjt" Oct 31 04:24:02.034350 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:02.031485 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38a822e5-1a32-4126-b19a-1264d09c66cc-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-3430f6-84fd4c6898-7rzjt\" (UID: \"38a822e5-1a32-4126-b19a-1264d09c66cc\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3430f6-84fd4c6898-7rzjt" Oct 31 04:24:02.113416 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:02.106861 4376 patch_prober.go:29] interesting pod/e2e-ipp-a-f91cd1-openshift-default-556cb85755-v94km container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Oct 31 04:24:02.113416 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:02.106922 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f91cd1-openshift-default-556cb85755-v94km" podUID="fd4632c9-069d-466e-b00a-c75aa4976d0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Oct 31 04:24:02.205722 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:02.205673 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-597d9\" (UniqueName: \"kubernetes.io/projected/38a822e5-1a32-4126-b19a-1264d09c66cc-kube-api-access-597d9\") pod \"payload-processing-e2e-ipp-a-3430f6-84fd4c6898-7rzjt\" (UID: \"38a822e5-1a32-4126-b19a-1264d09c66cc\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3430f6-84fd4c6898-7rzjt" Oct 31 04:24:02.206490 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:02.206459 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38a822e5-1a32-4126-b19a-1264d09c66cc-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-3430f6-84fd4c6898-7rzjt\" (UID: \"38a822e5-1a32-4126-b19a-1264d09c66cc\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3430f6-84fd4c6898-7rzjt" Oct 31 04:24:02.207712 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:02.207450 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38a822e5-1a32-4126-b19a-1264d09c66cc-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-3430f6-84fd4c6898-7rzjt\" (UID: \"38a822e5-1a32-4126-b19a-1264d09c66cc\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3430f6-84fd4c6898-7rzjt" Oct 31 04:24:02.260390 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:02.256980 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-597d9\" (UniqueName: \"kubernetes.io/projected/38a822e5-1a32-4126-b19a-1264d09c66cc-kube-api-access-597d9\") pod \"payload-processing-e2e-ipp-a-3430f6-84fd4c6898-7rzjt\" (UID: \"38a822e5-1a32-4126-b19a-1264d09c66cc\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3430f6-84fd4c6898-7rzjt" Oct 31 04:24:02.498380 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:02.498352 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3430f6-84fd4c6898-7rzjt" Oct 31 04:24:02.583652 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:02.583272 4376 patch_prober.go:29] interesting pod/e2e-ipp-a-3430f6-openshift-default-595d4f6b8c-ltm6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Oct 31 04:24:02.583652 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:02.583352 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3430f6-openshift-default-595d4f6b8c-ltm6g" podUID="3ade7461-87d8-42ea-a721-512bae1f5f65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Oct 31 04:24:02.864383 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:02.864265 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3430f6-84fd4c6898-7rzjt"] Oct 31 04:24:02.941966 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:24:02.931802 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38a822e5_1a32_4126_b19a_1264d09c66cc.slice/crio-b754c242bb4e6883e7992105c3037d8f3f1506af9c8fac0749fadabbc32dd7b0 WatchSource:0}: Error finding container b754c242bb4e6883e7992105c3037d8f3f1506af9c8fac0749fadabbc32dd7b0: Status 404 returned error can't find the container with id b754c242bb4e6883e7992105c3037d8f3f1506af9c8fac0749fadabbc32dd7b0 Oct 31 04:24:03.064418 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:03.052710 4376 patch_prober.go:29] interesting pod/e2e-ipp-a-f91cd1-openshift-default-556cb85755-v94km container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Oct 31 04:24:03.064418 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:03.052769 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f91cd1-openshift-default-556cb85755-v94km" podUID="fd4632c9-069d-466e-b00a-c75aa4976d0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Oct 31 04:24:03.206044 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:03.206006 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3430f6-84fd4c6898-7rzjt" event={"ID":"38a822e5-1a32-4126-b19a-1264d09c66cc","Type":"ContainerStarted","Data":"b754c242bb4e6883e7992105c3037d8f3f1506af9c8fac0749fadabbc32dd7b0"} Oct 31 04:24:03.582892 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:03.582811 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-3430f6-openshift-default-595d4f6b8c-ltm6g" Oct 31 04:24:03.584771 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:03.584750 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-3430f6-openshift-default-595d4f6b8c-ltm6g" Oct 31 04:24:04.028173 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:04.028139 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b8bb1e-8bfbcf887-kchch" Oct 31 04:24:04.029682 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:04.029658 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-b8bb1e-84466f94c-pjkmm" Oct 31 04:24:04.100793 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:04.100761 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-f91cd1-openshift-default-556cb85755-v94km" Oct 31 04:24:04.130626 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:04.130382 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-f91cd1-openshift-default-556cb85755-v94km" Oct 31 04:24:05.235241 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:05.235206 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3430f6-84fd4c6898-7rzjt" event={"ID":"38a822e5-1a32-4126-b19a-1264d09c66cc","Type":"ContainerStarted","Data":"8bdd819bc76dcc8338813f4c7f4153ecefb2c92aa76ad06296ade5a02922efd1"} Oct 31 04:24:05.235750 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:05.235343 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-3430f6-84fd4c6898-7rzjt" Oct 31 04:24:05.280914 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:05.280807 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-3430f6-84fd4c6898-7rzjt" podStartSLOduration=2.231776565 podStartE2EDuration="4.280789596s" totalImagesPullingTime="2.049013031s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:24:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:24:05.277105132 +0000 UTC m=+1192.991439767" watchObservedRunningTime="2026-10-31 04:24:05.280789596 +0000 UTC m=+1192.995124214" Oct 31 04:24:11.037956 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:24:11.037923 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 04:24:13.843963 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:13.830642 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ceba51-6c8fc9774c-vk52b"] Oct 31 04:24:14.023021 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:14.022981 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ceba51-6c8fc9774c-vk52b" Oct 31 04:24:14.026654 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:14.026593 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ceba51-6c8fc9774c-vk52b"] Oct 31 04:24:14.040236 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:14.040183 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-ceba51\"" Oct 31 04:24:14.040555 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:14.040201 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-ceba51-dockercfg-f9w4l\"" Oct 31 04:24:14.172389 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:14.172309 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txzfs\" (UniqueName: \"kubernetes.io/projected/c503874b-c276-4ecf-b767-952b44173d94-kube-api-access-txzfs\") pod \"payload-pre-processing-e2e-ipp-a-ceba51-6c8fc9774c-vk52b\" (UID: \"c503874b-c276-4ecf-b767-952b44173d94\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ceba51-6c8fc9774c-vk52b" Oct 31 04:24:14.172611 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:14.172558 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c503874b-c276-4ecf-b767-952b44173d94-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-ceba51-6c8fc9774c-vk52b\" (UID: \"c503874b-c276-4ecf-b767-952b44173d94\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ceba51-6c8fc9774c-vk52b" Oct 31 04:24:14.273807 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:14.273769 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txzfs\" (UniqueName: \"kubernetes.io/projected/c503874b-c276-4ecf-b767-952b44173d94-kube-api-access-txzfs\") pod \"payload-pre-processing-e2e-ipp-a-ceba51-6c8fc9774c-vk52b\" (UID: \"c503874b-c276-4ecf-b767-952b44173d94\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ceba51-6c8fc9774c-vk52b" Oct 31 04:24:14.273984 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:14.273925 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c503874b-c276-4ecf-b767-952b44173d94-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-ceba51-6c8fc9774c-vk52b\" (UID: \"c503874b-c276-4ecf-b767-952b44173d94\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ceba51-6c8fc9774c-vk52b" Oct 31 04:24:14.274699 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:14.274678 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c503874b-c276-4ecf-b767-952b44173d94-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-ceba51-6c8fc9774c-vk52b\" (UID: \"c503874b-c276-4ecf-b767-952b44173d94\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ceba51-6c8fc9774c-vk52b" Oct 31 04:24:14.297377 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:14.297346 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-txzfs\" (UniqueName: \"kubernetes.io/projected/c503874b-c276-4ecf-b767-952b44173d94-kube-api-access-txzfs\") pod \"payload-pre-processing-e2e-ipp-a-ceba51-6c8fc9774c-vk52b\" (UID: \"c503874b-c276-4ecf-b767-952b44173d94\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ceba51-6c8fc9774c-vk52b" Oct 31 04:24:14.392113 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:14.392079 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ceba51-6c8fc9774c-vk52b" Oct 31 04:24:14.496044 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:14.496019 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b8bb1e-84bc5d79f5-pw6x4"] Oct 31 04:24:14.500805 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:14.500753 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b8bb1e-84bc5d79f5-pw6x4" Oct 31 04:24:14.522342 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:14.521700 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-b8bb1e-serving-cert\"" Oct 31 04:24:14.525801 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:14.525768 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b8bb1e-84bc5d79f5-pw6x4"] Oct 31 04:24:14.578519 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:14.578480 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/31582135-4095-446e-b948-86fe2c21278a-maas-api-tls\") pod \"maas-api-e2e-ipp-b-b8bb1e-84bc5d79f5-pw6x4\" (UID: \"31582135-4095-446e-b948-86fe2c21278a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b8bb1e-84bc5d79f5-pw6x4" Oct 31 04:24:14.578709 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:14.578606 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/31582135-4095-446e-b948-86fe2c21278a-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-b8bb1e-84bc5d79f5-pw6x4\" (UID: \"31582135-4095-446e-b948-86fe2c21278a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b8bb1e-84bc5d79f5-pw6x4" Oct 31 04:24:14.578709 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:14.578648 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/31582135-4095-446e-b948-86fe2c21278a-metrics-certs\") pod \"maas-api-e2e-ipp-b-b8bb1e-84bc5d79f5-pw6x4\" (UID: \"31582135-4095-446e-b948-86fe2c21278a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b8bb1e-84bc5d79f5-pw6x4" Oct 31 04:24:14.578949 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:14.578920 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxn5h\" (UniqueName: \"kubernetes.io/projected/31582135-4095-446e-b948-86fe2c21278a-kube-api-access-xxn5h\") pod \"maas-api-e2e-ipp-b-b8bb1e-84bc5d79f5-pw6x4\" (UID: \"31582135-4095-446e-b948-86fe2c21278a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b8bb1e-84bc5d79f5-pw6x4" Oct 31 04:24:14.680308 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:14.680276 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxn5h\" (UniqueName: \"kubernetes.io/projected/31582135-4095-446e-b948-86fe2c21278a-kube-api-access-xxn5h\") pod \"maas-api-e2e-ipp-b-b8bb1e-84bc5d79f5-pw6x4\" (UID: \"31582135-4095-446e-b948-86fe2c21278a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b8bb1e-84bc5d79f5-pw6x4" Oct 31 04:24:14.680512 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:14.680450 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/31582135-4095-446e-b948-86fe2c21278a-maas-api-tls\") pod \"maas-api-e2e-ipp-b-b8bb1e-84bc5d79f5-pw6x4\" (UID: \"31582135-4095-446e-b948-86fe2c21278a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b8bb1e-84bc5d79f5-pw6x4" Oct 31 04:24:14.680512 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:14.680495 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/31582135-4095-446e-b948-86fe2c21278a-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-b8bb1e-84bc5d79f5-pw6x4\" (UID: \"31582135-4095-446e-b948-86fe2c21278a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b8bb1e-84bc5d79f5-pw6x4" Oct 31 04:24:14.680651 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:14.680525 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/31582135-4095-446e-b948-86fe2c21278a-metrics-certs\") pod \"maas-api-e2e-ipp-b-b8bb1e-84bc5d79f5-pw6x4\" (UID: \"31582135-4095-446e-b948-86fe2c21278a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b8bb1e-84bc5d79f5-pw6x4" Oct 31 04:24:14.681261 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:14.681235 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/31582135-4095-446e-b948-86fe2c21278a-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-b8bb1e-84bc5d79f5-pw6x4\" (UID: \"31582135-4095-446e-b948-86fe2c21278a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b8bb1e-84bc5d79f5-pw6x4" Oct 31 04:24:14.683018 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:14.682995 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/31582135-4095-446e-b948-86fe2c21278a-maas-api-tls\") pod \"maas-api-e2e-ipp-b-b8bb1e-84bc5d79f5-pw6x4\" (UID: \"31582135-4095-446e-b948-86fe2c21278a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b8bb1e-84bc5d79f5-pw6x4" Oct 31 04:24:14.683127 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:14.683110 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/31582135-4095-446e-b948-86fe2c21278a-metrics-certs\") pod \"maas-api-e2e-ipp-b-b8bb1e-84bc5d79f5-pw6x4\" (UID: \"31582135-4095-446e-b948-86fe2c21278a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b8bb1e-84bc5d79f5-pw6x4" Oct 31 04:24:14.703786 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:14.703751 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxn5h\" (UniqueName: \"kubernetes.io/projected/31582135-4095-446e-b948-86fe2c21278a-kube-api-access-xxn5h\") pod \"maas-api-e2e-ipp-b-b8bb1e-84bc5d79f5-pw6x4\" (UID: \"31582135-4095-446e-b948-86fe2c21278a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b8bb1e-84bc5d79f5-pw6x4" Oct 31 04:24:14.714491 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:24:14.714464 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc503874b_c276_4ecf_b767_952b44173d94.slice/crio-6aeb1a9a7190a5b2a66fbedac89e61ecc8f3035750a140552ac301456df4087e WatchSource:0}: Error finding container 6aeb1a9a7190a5b2a66fbedac89e61ecc8f3035750a140552ac301456df4087e: Status 404 returned error can't find the container with id 6aeb1a9a7190a5b2a66fbedac89e61ecc8f3035750a140552ac301456df4087e Oct 31 04:24:14.716472 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:14.716453 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ceba51-6c8fc9774c-vk52b"] Oct 31 04:24:14.829395 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:14.816409 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b8bb1e-84bc5d79f5-pw6x4" Oct 31 04:24:15.424178 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:15.422157 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b8bb1e-84bc5d79f5-pw6x4"] Oct 31 04:24:15.424659 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:24:15.424400 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31582135_4095_446e_b948_86fe2c21278a.slice/crio-38b761c3e17db80ec7ee5cad261a9ea973c11800354f8816b5fad723db032cf0 WatchSource:0}: Error finding container 38b761c3e17db80ec7ee5cad261a9ea973c11800354f8816b5fad723db032cf0: Status 404 returned error can't find the container with id 38b761c3e17db80ec7ee5cad261a9ea973c11800354f8816b5fad723db032cf0 Oct 31 04:24:15.436404 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:15.436376 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ceba51-6c8fc9774c-vk52b" event={"ID":"c503874b-c276-4ecf-b767-952b44173d94","Type":"ContainerStarted","Data":"f53d83918d3952eae679a0f9744e5d9ef0f06c2cb77f0b740c631e12b7464894"} Oct 31 04:24:15.436512 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:15.436419 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ceba51-6c8fc9774c-vk52b" event={"ID":"c503874b-c276-4ecf-b767-952b44173d94","Type":"ContainerStarted","Data":"6aeb1a9a7190a5b2a66fbedac89e61ecc8f3035750a140552ac301456df4087e"} Oct 31 04:24:15.437119 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:15.437103 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ceba51-6c8fc9774c-vk52b" Oct 31 04:24:15.479940 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:15.479887 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ceba51-6c8fc9774c-vk52b" podStartSLOduration=2.479872544 podStartE2EDuration="2.479872544s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:24:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:24:15.478631597 +0000 UTC m=+1203.192966235" watchObservedRunningTime="2026-10-31 04:24:15.479872544 +0000 UTC m=+1203.194207161" Oct 31 04:24:16.244659 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:16.244494 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-3430f6-84fd4c6898-7rzjt" Oct 31 04:24:16.578423 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:16.577763 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b8bb1e-84bc5d79f5-pw6x4" Oct 31 04:24:16.578423 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:16.577827 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b8bb1e-84bc5d79f5-pw6x4" event={"ID":"31582135-4095-446e-b948-86fe2c21278a","Type":"ContainerStarted","Data":"adbf07c1f6d0a310859fb6f4ef4521330ad3c879e1f8236f138a57755a88d929"} Oct 31 04:24:16.578423 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:16.577858 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b8bb1e-84bc5d79f5-pw6x4" event={"ID":"31582135-4095-446e-b948-86fe2c21278a","Type":"ContainerStarted","Data":"38b761c3e17db80ec7ee5cad261a9ea973c11800354f8816b5fad723db032cf0"} Oct 31 04:24:16.582351 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:16.581199 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b8bb1e-84bc5d79f5-pw6x4" podStartSLOduration=2.088274241 podStartE2EDuration="2.58117778s" totalImagesPullingTime="492.903539ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:24:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:24:16.547701109 +0000 UTC m=+1204.262035727" watchObservedRunningTime="2026-10-31 04:24:16.58117778 +0000 UTC m=+1204.295512399" Oct 31 04:24:21.633505 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:21.633464 4376 patch_prober.go:29] interesting pod/e2e-api-a-801218-openshift-default-58b4b8685c-8gpjq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 04:24:21.635852 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:21.633543 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-801218-openshift-default-58b4b8685c-8gpjq" podUID="e3f3e12d-e06e-4b4e-a23a-784e49d5cbaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 04:24:22.464624 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:22.464589 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b8bb1e-84bc5d79f5-pw6x4" Oct 31 04:24:22.526512 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:22.526477 4376 generic.go:356] "Generic (PLEG): container finished" podID="e3f3e12d-e06e-4b4e-a23a-784e49d5cbaf" containerID="f9cbfef45650c56699306fde4739e979d5b6d74a078474c21b5a6d2d92557af2" exitCode=0 Oct 31 04:24:22.526751 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:22.526733 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-801218-openshift-default-58b4b8685c-8gpjq" event={"ID":"e3f3e12d-e06e-4b4e-a23a-784e49d5cbaf","Type":"ContainerDied","Data":"f9cbfef45650c56699306fde4739e979d5b6d74a078474c21b5a6d2d92557af2"} Oct 31 04:24:22.527255 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:22.527238 4376 scope.go:122] "RemoveContainer" containerID="f9cbfef45650c56699306fde4739e979d5b6d74a078474c21b5a6d2d92557af2" Oct 31 04:24:22.529043 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:22.529012 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:24:22.529135 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:22.529092 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:24:22.529190 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:22.529137 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:24:23.536454 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:23.536414 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-801218-openshift-default-58b4b8685c-8gpjq" event={"ID":"e3f3e12d-e06e-4b4e-a23a-784e49d5cbaf","Type":"ContainerStarted","Data":"05fb4b4a647bbba5071604f9fa08cb58fe5b76ede194d21403f43d1d9745942f"} Oct 31 04:24:23.815621 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:23.815542 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-801218-openshift-default-58b4b8685c-8gpjq" Oct 31 04:24:23.815795 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:23.815699 4376 patch_prober.go:29] interesting pod/e2e-api-a-801218-openshift-default-58b4b8685c-8gpjq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Oct 31 04:24:23.815795 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:23.815760 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-801218-openshift-default-58b4b8685c-8gpjq" podUID="e3f3e12d-e06e-4b4e-a23a-784e49d5cbaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Oct 31 04:24:24.816702 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:24.816614 4376 patch_prober.go:29] interesting pod/e2e-api-a-801218-openshift-default-58b4b8685c-8gpjq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Oct 31 04:24:24.816702 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:24.816682 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-801218-openshift-default-58b4b8685c-8gpjq" podUID="e3f3e12d-e06e-4b4e-a23a-784e49d5cbaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Oct 31 04:24:25.517849 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:25.517813 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7b4185-898d6c9df-fjswd"] Oct 31 04:24:25.523858 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:25.523831 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-7b4185-898d6c9df-fjswd" Oct 31 04:24:25.552581 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:25.552552 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-7b4185-dockercfg-jb92g\"" Oct 31 04:24:25.552896 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:25.552875 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-7b4185\"" Oct 31 04:24:25.570352 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:25.554255 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7b4185-898d6c9df-fjswd"] Oct 31 04:24:25.585440 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:25.585405 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7b4185-7848cfbf55-jjpxl"] Oct 31 04:24:25.586232 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:25.586198 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83a9eaed-ab99-4fd4-9ffe-a1d863a85cf8-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-7b4185-898d6c9df-fjswd\" (UID: \"83a9eaed-ab99-4fd4-9ffe-a1d863a85cf8\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7b4185-898d6c9df-fjswd" Oct 31 04:24:25.586546 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:25.586530 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxbzf\" (UniqueName: \"kubernetes.io/projected/83a9eaed-ab99-4fd4-9ffe-a1d863a85cf8-kube-api-access-bxbzf\") pod \"payload-processing-e2e-ipp-b-7b4185-898d6c9df-fjswd\" (UID: \"83a9eaed-ab99-4fd4-9ffe-a1d863a85cf8\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7b4185-898d6c9df-fjswd" Oct 31 04:24:25.590414 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:25.590399 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7b4185-7848cfbf55-jjpxl" Oct 31 04:24:25.609700 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:25.609676 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7b4185-7848cfbf55-jjpxl"] Oct 31 04:24:25.694953 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:25.694906 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxbzf\" (UniqueName: \"kubernetes.io/projected/83a9eaed-ab99-4fd4-9ffe-a1d863a85cf8-kube-api-access-bxbzf\") pod \"payload-processing-e2e-ipp-b-7b4185-898d6c9df-fjswd\" (UID: \"83a9eaed-ab99-4fd4-9ffe-a1d863a85cf8\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7b4185-898d6c9df-fjswd" Oct 31 04:24:25.695528 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:25.695501 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vb4gq\" (UniqueName: \"kubernetes.io/projected/419996ea-d44d-44f4-b414-583068843613-kube-api-access-vb4gq\") pod \"payload-pre-processing-e2e-ipp-b-7b4185-7848cfbf55-jjpxl\" (UID: \"419996ea-d44d-44f4-b414-583068843613\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7b4185-7848cfbf55-jjpxl" Oct 31 04:24:25.695777 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:25.695761 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83a9eaed-ab99-4fd4-9ffe-a1d863a85cf8-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-7b4185-898d6c9df-fjswd\" (UID: \"83a9eaed-ab99-4fd4-9ffe-a1d863a85cf8\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7b4185-898d6c9df-fjswd" Oct 31 04:24:25.709933 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:25.696010 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/419996ea-d44d-44f4-b414-583068843613-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-7b4185-7848cfbf55-jjpxl\" (UID: \"419996ea-d44d-44f4-b414-583068843613\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7b4185-7848cfbf55-jjpxl" Oct 31 04:24:25.711429 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:25.711404 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83a9eaed-ab99-4fd4-9ffe-a1d863a85cf8-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-7b4185-898d6c9df-fjswd\" (UID: \"83a9eaed-ab99-4fd4-9ffe-a1d863a85cf8\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7b4185-898d6c9df-fjswd" Oct 31 04:24:25.876430 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:25.841026 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/419996ea-d44d-44f4-b414-583068843613-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-7b4185-7848cfbf55-jjpxl\" (UID: \"419996ea-d44d-44f4-b414-583068843613\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7b4185-7848cfbf55-jjpxl" Oct 31 04:24:25.876430 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:25.841346 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vb4gq\" (UniqueName: \"kubernetes.io/projected/419996ea-d44d-44f4-b414-583068843613-kube-api-access-vb4gq\") pod \"payload-pre-processing-e2e-ipp-b-7b4185-7848cfbf55-jjpxl\" (UID: \"419996ea-d44d-44f4-b414-583068843613\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7b4185-7848cfbf55-jjpxl" Oct 31 04:24:25.876430 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:25.842632 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/419996ea-d44d-44f4-b414-583068843613-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-7b4185-7848cfbf55-jjpxl\" (UID: \"419996ea-d44d-44f4-b414-583068843613\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7b4185-7848cfbf55-jjpxl" Oct 31 04:24:25.876430 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:25.842956 4376 patch_prober.go:29] interesting pod/e2e-api-a-801218-openshift-default-58b4b8685c-8gpjq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Oct 31 04:24:25.943489 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:25.940960 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-801218-openshift-default-58b4b8685c-8gpjq" podUID="e3f3e12d-e06e-4b4e-a23a-784e49d5cbaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Oct 31 04:24:26.049388 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:26.024693 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxbzf\" (UniqueName: \"kubernetes.io/projected/83a9eaed-ab99-4fd4-9ffe-a1d863a85cf8-kube-api-access-bxbzf\") pod \"payload-processing-e2e-ipp-b-7b4185-898d6c9df-fjswd\" (UID: \"83a9eaed-ab99-4fd4-9ffe-a1d863a85cf8\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7b4185-898d6c9df-fjswd" Oct 31 04:24:26.049388 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:26.039171 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-7b4185-898d6c9df-fjswd" Oct 31 04:24:26.049388 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:26.043822 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vb4gq\" (UniqueName: \"kubernetes.io/projected/419996ea-d44d-44f4-b414-583068843613-kube-api-access-vb4gq\") pod \"payload-pre-processing-e2e-ipp-b-7b4185-7848cfbf55-jjpxl\" (UID: \"419996ea-d44d-44f4-b414-583068843613\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7b4185-7848cfbf55-jjpxl" Oct 31 04:24:26.210129 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:26.210095 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7b4185-7848cfbf55-jjpxl" Oct 31 04:24:26.469167 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:24:26.469134 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83a9eaed_ab99_4fd4_9ffe_a1d863a85cf8.slice/crio-3fe574e047d003a78ffa28ca072ee9f35c6ac3b26ba7f21707b64bb7a515f2c5 WatchSource:0}: Error finding container 3fe574e047d003a78ffa28ca072ee9f35c6ac3b26ba7f21707b64bb7a515f2c5: Status 404 returned error can't find the container with id 3fe574e047d003a78ffa28ca072ee9f35c6ac3b26ba7f21707b64bb7a515f2c5 Oct 31 04:24:26.473750 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:26.473723 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7b4185-898d6c9df-fjswd"] Oct 31 04:24:26.571554 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:26.571525 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7b4185-7848cfbf55-jjpxl"] Oct 31 04:24:26.573130 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:24:26.573104 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod419996ea_d44d_44f4_b414_583068843613.slice/crio-88cb932bca2789dfa336a3e95dc2c9d27860c1de9e4148f75a8dc4e85f930bf2 WatchSource:0}: Error finding container 88cb932bca2789dfa336a3e95dc2c9d27860c1de9e4148f75a8dc4e85f930bf2: Status 404 returned error can't find the container with id 88cb932bca2789dfa336a3e95dc2c9d27860c1de9e4148f75a8dc4e85f930bf2 Oct 31 04:24:26.588933 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:26.588907 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7b4185-7848cfbf55-jjpxl" event={"ID":"419996ea-d44d-44f4-b414-583068843613","Type":"ContainerStarted","Data":"88cb932bca2789dfa336a3e95dc2c9d27860c1de9e4148f75a8dc4e85f930bf2"} Oct 31 04:24:26.595569 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:26.595545 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-7b4185-898d6c9df-fjswd" event={"ID":"83a9eaed-ab99-4fd4-9ffe-a1d863a85cf8","Type":"ContainerStarted","Data":"177ffa7257564a79bf161becd0d8491a0eb1dc62d94a2feb9a3d35c26c731974"} Oct 31 04:24:26.595662 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:26.595589 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-7b4185-898d6c9df-fjswd" event={"ID":"83a9eaed-ab99-4fd4-9ffe-a1d863a85cf8","Type":"ContainerStarted","Data":"3fe574e047d003a78ffa28ca072ee9f35c6ac3b26ba7f21707b64bb7a515f2c5"} Oct 31 04:24:26.596408 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:26.596391 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-7b4185-898d6c9df-fjswd" Oct 31 04:24:26.670496 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:26.670439 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-7b4185-898d6c9df-fjswd" podStartSLOduration=1.670423956 podStartE2EDuration="1.670423956s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:24:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:24:26.664253241 +0000 UTC m=+1214.378587863" watchObservedRunningTime="2026-10-31 04:24:26.670423956 +0000 UTC m=+1214.384758592" Oct 31 04:24:26.820420 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:26.818552 4376 patch_prober.go:29] interesting pod/e2e-api-a-801218-openshift-default-58b4b8685c-8gpjq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Oct 31 04:24:26.820420 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:26.818613 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-801218-openshift-default-58b4b8685c-8gpjq" podUID="e3f3e12d-e06e-4b4e-a23a-784e49d5cbaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Oct 31 04:24:27.472354 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:27.471629 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ceba51-6c8fc9774c-vk52b" Oct 31 04:24:27.618981 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:27.618941 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7b4185-7848cfbf55-jjpxl" event={"ID":"419996ea-d44d-44f4-b414-583068843613","Type":"ContainerStarted","Data":"555127ce7d1b668d1db820998d034a68709197d76886907a2a6a8c9ffbbfa3fa"} Oct 31 04:24:27.619157 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:27.619019 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7b4185-7848cfbf55-jjpxl" Oct 31 04:24:27.683388 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:27.683315 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7b4185-7848cfbf55-jjpxl" podStartSLOduration=2.683296603 podStartE2EDuration="2.683296603s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:24:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:24:27.672908545 +0000 UTC m=+1215.387243155" watchObservedRunningTime="2026-10-31 04:24:27.683296603 +0000 UTC m=+1215.397631227" Oct 31 04:24:27.818510 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:27.817473 4376 patch_prober.go:29] interesting pod/e2e-api-a-801218-openshift-default-58b4b8685c-8gpjq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Oct 31 04:24:27.818510 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:27.817549 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-801218-openshift-default-58b4b8685c-8gpjq" podUID="e3f3e12d-e06e-4b4e-a23a-784e49d5cbaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Oct 31 04:24:28.816057 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:28.816024 4376 patch_prober.go:29] interesting pod/e2e-api-a-801218-openshift-default-58b4b8685c-8gpjq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Oct 31 04:24:28.816489 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:28.816078 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-801218-openshift-default-58b4b8685c-8gpjq" podUID="e3f3e12d-e06e-4b4e-a23a-784e49d5cbaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Oct 31 04:24:29.820497 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:29.820465 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-801218-openshift-default-58b4b8685c-8gpjq" Oct 31 04:24:29.820878 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:29.820751 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-801218-openshift-default-58b4b8685c-8gpjq" Oct 31 04:24:29.821403 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:29.821384 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-801218-openshift-default-58b4b8685c-8gpjq" Oct 31 04:24:33.266983 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:33.266949 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-9154a6-openshift-default-7f74d4f9bc-v6fc7"] Oct 31 04:24:33.271540 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:33.271516 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-9154a6-openshift-default-7f74d4f9bc-v6fc7" Oct 31 04:24:33.277885 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:33.277849 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a7c4b33-999a-449b-bea8-9c55c838df0a-istio-token\") pod \"e2e-api-b-9154a6-openshift-default-7f74d4f9bc-v6fc7\" (UID: \"1a7c4b33-999a-449b-bea8-9c55c838df0a\") " pod="openshift-ingress/e2e-api-b-9154a6-openshift-default-7f74d4f9bc-v6fc7" Oct 31 04:24:33.279004 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:33.277910 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a7c4b33-999a-449b-bea8-9c55c838df0a-istio-data\") pod \"e2e-api-b-9154a6-openshift-default-7f74d4f9bc-v6fc7\" (UID: \"1a7c4b33-999a-449b-bea8-9c55c838df0a\") " pod="openshift-ingress/e2e-api-b-9154a6-openshift-default-7f74d4f9bc-v6fc7" Oct 31 04:24:33.279004 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:33.277942 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a7c4b33-999a-449b-bea8-9c55c838df0a-istio-envoy\") pod \"e2e-api-b-9154a6-openshift-default-7f74d4f9bc-v6fc7\" (UID: \"1a7c4b33-999a-449b-bea8-9c55c838df0a\") " pod="openshift-ingress/e2e-api-b-9154a6-openshift-default-7f74d4f9bc-v6fc7" Oct 31 04:24:33.279004 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:33.278016 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a7c4b33-999a-449b-bea8-9c55c838df0a-istiod-ca-cert\") pod \"e2e-api-b-9154a6-openshift-default-7f74d4f9bc-v6fc7\" (UID: \"1a7c4b33-999a-449b-bea8-9c55c838df0a\") " pod="openshift-ingress/e2e-api-b-9154a6-openshift-default-7f74d4f9bc-v6fc7" Oct 31 04:24:33.279004 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:33.278069 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a7c4b33-999a-449b-bea8-9c55c838df0a-credential-socket\") pod \"e2e-api-b-9154a6-openshift-default-7f74d4f9bc-v6fc7\" (UID: \"1a7c4b33-999a-449b-bea8-9c55c838df0a\") " pod="openshift-ingress/e2e-api-b-9154a6-openshift-default-7f74d4f9bc-v6fc7" Oct 31 04:24:33.279004 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:33.278128 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a7c4b33-999a-449b-bea8-9c55c838df0a-workload-certs\") pod \"e2e-api-b-9154a6-openshift-default-7f74d4f9bc-v6fc7\" (UID: \"1a7c4b33-999a-449b-bea8-9c55c838df0a\") " pod="openshift-ingress/e2e-api-b-9154a6-openshift-default-7f74d4f9bc-v6fc7" Oct 31 04:24:33.279004 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:33.278195 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpkrn\" (UniqueName: \"kubernetes.io/projected/1a7c4b33-999a-449b-bea8-9c55c838df0a-kube-api-access-jpkrn\") pod \"e2e-api-b-9154a6-openshift-default-7f74d4f9bc-v6fc7\" (UID: \"1a7c4b33-999a-449b-bea8-9c55c838df0a\") " pod="openshift-ingress/e2e-api-b-9154a6-openshift-default-7f74d4f9bc-v6fc7" Oct 31 04:24:33.279004 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:33.278477 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a7c4b33-999a-449b-bea8-9c55c838df0a-istio-podinfo\") pod \"e2e-api-b-9154a6-openshift-default-7f74d4f9bc-v6fc7\" (UID: \"1a7c4b33-999a-449b-bea8-9c55c838df0a\") " pod="openshift-ingress/e2e-api-b-9154a6-openshift-default-7f74d4f9bc-v6fc7" Oct 31 04:24:33.279004 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:33.278661 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a7c4b33-999a-449b-bea8-9c55c838df0a-workload-socket\") pod \"e2e-api-b-9154a6-openshift-default-7f74d4f9bc-v6fc7\" (UID: \"1a7c4b33-999a-449b-bea8-9c55c838df0a\") " pod="openshift-ingress/e2e-api-b-9154a6-openshift-default-7f74d4f9bc-v6fc7" Oct 31 04:24:33.296725 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:33.296680 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-9154a6-openshift-default-7f74d4f9bc-v6fc7"] Oct 31 04:24:33.302933 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:33.302898 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-9154a6-openshift-default-dockercfg-fknmd\"" Oct 31 04:24:33.379995 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:33.379960 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a7c4b33-999a-449b-bea8-9c55c838df0a-credential-socket\") pod \"e2e-api-b-9154a6-openshift-default-7f74d4f9bc-v6fc7\" (UID: \"1a7c4b33-999a-449b-bea8-9c55c838df0a\") " pod="openshift-ingress/e2e-api-b-9154a6-openshift-default-7f74d4f9bc-v6fc7" Oct 31 04:24:33.380174 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:33.380010 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a7c4b33-999a-449b-bea8-9c55c838df0a-workload-certs\") pod \"e2e-api-b-9154a6-openshift-default-7f74d4f9bc-v6fc7\" (UID: \"1a7c4b33-999a-449b-bea8-9c55c838df0a\") " pod="openshift-ingress/e2e-api-b-9154a6-openshift-default-7f74d4f9bc-v6fc7" Oct 31 04:24:33.380174 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:33.380047 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpkrn\" (UniqueName: \"kubernetes.io/projected/1a7c4b33-999a-449b-bea8-9c55c838df0a-kube-api-access-jpkrn\") pod \"e2e-api-b-9154a6-openshift-default-7f74d4f9bc-v6fc7\" (UID: \"1a7c4b33-999a-449b-bea8-9c55c838df0a\") " pod="openshift-ingress/e2e-api-b-9154a6-openshift-default-7f74d4f9bc-v6fc7" Oct 31 04:24:33.380174 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:33.380155 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a7c4b33-999a-449b-bea8-9c55c838df0a-istio-podinfo\") pod \"e2e-api-b-9154a6-openshift-default-7f74d4f9bc-v6fc7\" (UID: \"1a7c4b33-999a-449b-bea8-9c55c838df0a\") " pod="openshift-ingress/e2e-api-b-9154a6-openshift-default-7f74d4f9bc-v6fc7" Oct 31 04:24:33.380276 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:33.380234 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a7c4b33-999a-449b-bea8-9c55c838df0a-workload-socket\") pod \"e2e-api-b-9154a6-openshift-default-7f74d4f9bc-v6fc7\" (UID: \"1a7c4b33-999a-449b-bea8-9c55c838df0a\") " pod="openshift-ingress/e2e-api-b-9154a6-openshift-default-7f74d4f9bc-v6fc7" Oct 31 04:24:33.380323 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:33.380286 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a7c4b33-999a-449b-bea8-9c55c838df0a-istio-token\") pod \"e2e-api-b-9154a6-openshift-default-7f74d4f9bc-v6fc7\" (UID: \"1a7c4b33-999a-449b-bea8-9c55c838df0a\") " pod="openshift-ingress/e2e-api-b-9154a6-openshift-default-7f74d4f9bc-v6fc7" Oct 31 04:24:33.380393 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:33.380322 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a7c4b33-999a-449b-bea8-9c55c838df0a-credential-socket\") pod \"e2e-api-b-9154a6-openshift-default-7f74d4f9bc-v6fc7\" (UID: \"1a7c4b33-999a-449b-bea8-9c55c838df0a\") " pod="openshift-ingress/e2e-api-b-9154a6-openshift-default-7f74d4f9bc-v6fc7" Oct 31 04:24:33.380393 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:33.380365 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a7c4b33-999a-449b-bea8-9c55c838df0a-istio-data\") pod \"e2e-api-b-9154a6-openshift-default-7f74d4f9bc-v6fc7\" (UID: \"1a7c4b33-999a-449b-bea8-9c55c838df0a\") " pod="openshift-ingress/e2e-api-b-9154a6-openshift-default-7f74d4f9bc-v6fc7" Oct 31 04:24:33.380492 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:33.380396 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a7c4b33-999a-449b-bea8-9c55c838df0a-istio-envoy\") pod \"e2e-api-b-9154a6-openshift-default-7f74d4f9bc-v6fc7\" (UID: \"1a7c4b33-999a-449b-bea8-9c55c838df0a\") " pod="openshift-ingress/e2e-api-b-9154a6-openshift-default-7f74d4f9bc-v6fc7" Oct 31 04:24:33.380492 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:33.380437 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a7c4b33-999a-449b-bea8-9c55c838df0a-workload-certs\") pod \"e2e-api-b-9154a6-openshift-default-7f74d4f9bc-v6fc7\" (UID: \"1a7c4b33-999a-449b-bea8-9c55c838df0a\") " pod="openshift-ingress/e2e-api-b-9154a6-openshift-default-7f74d4f9bc-v6fc7" Oct 31 04:24:33.380492 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:33.380461 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a7c4b33-999a-449b-bea8-9c55c838df0a-istiod-ca-cert\") pod \"e2e-api-b-9154a6-openshift-default-7f74d4f9bc-v6fc7\" (UID: \"1a7c4b33-999a-449b-bea8-9c55c838df0a\") " pod="openshift-ingress/e2e-api-b-9154a6-openshift-default-7f74d4f9bc-v6fc7" Oct 31 04:24:33.380650 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:33.380573 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a7c4b33-999a-449b-bea8-9c55c838df0a-workload-socket\") pod \"e2e-api-b-9154a6-openshift-default-7f74d4f9bc-v6fc7\" (UID: \"1a7c4b33-999a-449b-bea8-9c55c838df0a\") " pod="openshift-ingress/e2e-api-b-9154a6-openshift-default-7f74d4f9bc-v6fc7" Oct 31 04:24:33.380741 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:33.380712 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a7c4b33-999a-449b-bea8-9c55c838df0a-istio-data\") pod \"e2e-api-b-9154a6-openshift-default-7f74d4f9bc-v6fc7\" (UID: \"1a7c4b33-999a-449b-bea8-9c55c838df0a\") " pod="openshift-ingress/e2e-api-b-9154a6-openshift-default-7f74d4f9bc-v6fc7" Oct 31 04:24:33.381080 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:33.381062 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a7c4b33-999a-449b-bea8-9c55c838df0a-istiod-ca-cert\") pod \"e2e-api-b-9154a6-openshift-default-7f74d4f9bc-v6fc7\" (UID: \"1a7c4b33-999a-449b-bea8-9c55c838df0a\") " pod="openshift-ingress/e2e-api-b-9154a6-openshift-default-7f74d4f9bc-v6fc7" Oct 31 04:24:33.382528 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:33.382503 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a7c4b33-999a-449b-bea8-9c55c838df0a-istio-envoy\") pod \"e2e-api-b-9154a6-openshift-default-7f74d4f9bc-v6fc7\" (UID: \"1a7c4b33-999a-449b-bea8-9c55c838df0a\") " pod="openshift-ingress/e2e-api-b-9154a6-openshift-default-7f74d4f9bc-v6fc7" Oct 31 04:24:33.382812 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:33.382792 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a7c4b33-999a-449b-bea8-9c55c838df0a-istio-podinfo\") pod \"e2e-api-b-9154a6-openshift-default-7f74d4f9bc-v6fc7\" (UID: \"1a7c4b33-999a-449b-bea8-9c55c838df0a\") " pod="openshift-ingress/e2e-api-b-9154a6-openshift-default-7f74d4f9bc-v6fc7" Oct 31 04:24:33.405169 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:33.405105 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a7c4b33-999a-449b-bea8-9c55c838df0a-istio-token\") pod \"e2e-api-b-9154a6-openshift-default-7f74d4f9bc-v6fc7\" (UID: \"1a7c4b33-999a-449b-bea8-9c55c838df0a\") " pod="openshift-ingress/e2e-api-b-9154a6-openshift-default-7f74d4f9bc-v6fc7" Oct 31 04:24:33.407877 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:33.405512 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpkrn\" (UniqueName: \"kubernetes.io/projected/1a7c4b33-999a-449b-bea8-9c55c838df0a-kube-api-access-jpkrn\") pod \"e2e-api-b-9154a6-openshift-default-7f74d4f9bc-v6fc7\" (UID: \"1a7c4b33-999a-449b-bea8-9c55c838df0a\") " pod="openshift-ingress/e2e-api-b-9154a6-openshift-default-7f74d4f9bc-v6fc7" Oct 31 04:24:33.600762 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:33.600423 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-9154a6-openshift-default-7f74d4f9bc-v6fc7" Oct 31 04:24:34.066629 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:24:34.066590 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a7c4b33_999a_449b_bea8_9c55c838df0a.slice/crio-d36093d5520f0f5d2c9fce56ead55f790432267f2a465f9f74a555bfa424b58b WatchSource:0}: Error finding container d36093d5520f0f5d2c9fce56ead55f790432267f2a465f9f74a555bfa424b58b: Status 404 returned error can't find the container with id d36093d5520f0f5d2c9fce56ead55f790432267f2a465f9f74a555bfa424b58b Oct 31 04:24:34.068651 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:34.068620 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-9154a6-openshift-default-7f74d4f9bc-v6fc7"] Oct 31 04:24:34.069389 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:34.069355 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:24:34.069451 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:34.069422 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:24:34.069487 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:34.069455 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:24:34.701353 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:34.690009 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-9154a6-openshift-default-7f74d4f9bc-v6fc7" event={"ID":"1a7c4b33-999a-449b-bea8-9c55c838df0a","Type":"ContainerStarted","Data":"d97124d31fee309b2b61051e27f2842e7c617becb5efaaa5d5b2133566921f90"} Oct 31 04:24:34.701353 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:34.690093 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-9154a6-openshift-default-7f74d4f9bc-v6fc7" event={"ID":"1a7c4b33-999a-449b-bea8-9c55c838df0a","Type":"ContainerStarted","Data":"d36093d5520f0f5d2c9fce56ead55f790432267f2a465f9f74a555bfa424b58b"} Oct 31 04:24:34.762908 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:34.762848 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-9154a6-openshift-default-7f74d4f9bc-v6fc7" podStartSLOduration=1.762831281 podStartE2EDuration="1.762831281s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:24:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:24:34.749026098 +0000 UTC m=+1222.463360741" watchObservedRunningTime="2026-10-31 04:24:34.762831281 +0000 UTC m=+1222.477165899" Oct 31 04:24:35.027448 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:34.999685 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 04:24:35.611320 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:35.611279 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-9154a6-openshift-default-7f74d4f9bc-v6fc7" Oct 31 04:24:35.622411 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:35.622379 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-9154a6-openshift-default-7f74d4f9bc-v6fc7" Oct 31 04:24:35.697314 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:35.697286 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-9154a6-openshift-default-7f74d4f9bc-v6fc7" Oct 31 04:24:35.698434 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:35.698413 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-9154a6-openshift-default-7f74d4f9bc-v6fc7" Oct 31 04:24:38.632901 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:38.632860 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7b4185-7848cfbf55-jjpxl" Oct 31 04:24:38.633395 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:38.632968 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-7b4185-898d6c9df-fjswd" Oct 31 04:24:40.592184 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:40.592140 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-9154a6-85c9f9448c-d4pt4"] Oct 31 04:24:40.598095 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:40.598069 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9154a6-85c9f9448c-d4pt4" Oct 31 04:24:40.618194 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:40.618164 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-9154a6-85c9f9448c-d4pt4"] Oct 31 04:24:40.622070 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:40.622036 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-9154a6-dockercfg-jn867\"" Oct 31 04:24:40.674354 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:40.674106 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-9154a6\"" Oct 31 04:24:40.797353 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:40.796193 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-947fp\" (UniqueName: \"kubernetes.io/projected/097d0283-9c56-4738-b296-3255c3367cd8-kube-api-access-947fp\") pod \"payload-pre-processing-e2e-api-b-9154a6-85c9f9448c-d4pt4\" (UID: \"097d0283-9c56-4738-b296-3255c3367cd8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-9154a6-85c9f9448c-d4pt4" Oct 31 04:24:40.797353 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:40.796407 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/097d0283-9c56-4738-b296-3255c3367cd8-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-9154a6-85c9f9448c-d4pt4\" (UID: \"097d0283-9c56-4738-b296-3255c3367cd8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-9154a6-85c9f9448c-d4pt4" Oct 31 04:24:40.905760 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:40.905707 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-947fp\" (UniqueName: \"kubernetes.io/projected/097d0283-9c56-4738-b296-3255c3367cd8-kube-api-access-947fp\") pod \"payload-pre-processing-e2e-api-b-9154a6-85c9f9448c-d4pt4\" (UID: \"097d0283-9c56-4738-b296-3255c3367cd8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-9154a6-85c9f9448c-d4pt4" Oct 31 04:24:40.906261 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:40.906239 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/097d0283-9c56-4738-b296-3255c3367cd8-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-9154a6-85c9f9448c-d4pt4\" (UID: \"097d0283-9c56-4738-b296-3255c3367cd8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-9154a6-85c9f9448c-d4pt4" Oct 31 04:24:40.907791 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:40.907771 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/097d0283-9c56-4738-b296-3255c3367cd8-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-9154a6-85c9f9448c-d4pt4\" (UID: \"097d0283-9c56-4738-b296-3255c3367cd8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-9154a6-85c9f9448c-d4pt4" Oct 31 04:24:40.967215 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:40.962156 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-947fp\" (UniqueName: \"kubernetes.io/projected/097d0283-9c56-4738-b296-3255c3367cd8-kube-api-access-947fp\") pod \"payload-pre-processing-e2e-api-b-9154a6-85c9f9448c-d4pt4\" (UID: \"097d0283-9c56-4738-b296-3255c3367cd8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-9154a6-85c9f9448c-d4pt4" Oct 31 04:24:41.125378 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:24:41.125322 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 04:24:41.217688 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:41.217599 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9154a6-85c9f9448c-d4pt4" Oct 31 04:24:41.535955 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:24:41.535925 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod097d0283_9c56_4738_b296_3255c3367cd8.slice/crio-273d1f93632615947158cce1b8e072a3628894794326e256c5fc71292d6bd27f WatchSource:0}: Error finding container 273d1f93632615947158cce1b8e072a3628894794326e256c5fc71292d6bd27f: Status 404 returned error can't find the container with id 273d1f93632615947158cce1b8e072a3628894794326e256c5fc71292d6bd27f Oct 31 04:24:41.546358 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:41.539121 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-9154a6-85c9f9448c-d4pt4"] Oct 31 04:24:41.976114 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:41.972907 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9154a6-85c9f9448c-d4pt4" event={"ID":"097d0283-9c56-4738-b296-3255c3367cd8","Type":"ContainerStarted","Data":"c1a4d54b05678e409b431b08e933d0abc0df1ec303686498a3937c6a0f74a505"} Oct 31 04:24:41.976114 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:41.973019 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9154a6-85c9f9448c-d4pt4" event={"ID":"097d0283-9c56-4738-b296-3255c3367cd8","Type":"ContainerStarted","Data":"273d1f93632615947158cce1b8e072a3628894794326e256c5fc71292d6bd27f"} Oct 31 04:24:41.976114 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:41.973311 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9154a6-85c9f9448c-d4pt4" Oct 31 04:24:42.018839 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:42.018783 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9154a6-85c9f9448c-d4pt4" podStartSLOduration=2.018769616 podStartE2EDuration="2.018769616s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:24:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:24:42.01491879 +0000 UTC m=+1229.729253424" watchObservedRunningTime="2026-10-31 04:24:42.018769616 +0000 UTC m=+1229.733104232" Oct 31 04:24:47.155036 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:47.155002 4376 generic.go:356] "Generic (PLEG): container finished" podID="61a59ba0-c79d-4942-8b8a-1493cb9e9182" containerID="4fd82931ebe743135b6a825d8f86ecdb8db0e895ee103a632b0318de61547947" exitCode=0 Oct 31 04:24:47.155557 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:47.155056 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b8bb1e-openshift-default-557bbf897c-phbbm" event={"ID":"61a59ba0-c79d-4942-8b8a-1493cb9e9182","Type":"ContainerDied","Data":"4fd82931ebe743135b6a825d8f86ecdb8db0e895ee103a632b0318de61547947"} Oct 31 04:24:47.155557 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:47.155530 4376 scope.go:122] "RemoveContainer" containerID="4fd82931ebe743135b6a825d8f86ecdb8db0e895ee103a632b0318de61547947" Oct 31 04:24:47.157302 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:47.157267 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:24:47.157393 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:47.157377 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:24:47.157431 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:47.157415 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:24:47.812300 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:47.812272 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-b8bb1e-openshift-default-557bbf897c-phbbm" Oct 31 04:24:48.171244 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:48.171208 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b8bb1e-openshift-default-557bbf897c-phbbm" event={"ID":"61a59ba0-c79d-4942-8b8a-1493cb9e9182","Type":"ContainerStarted","Data":"61198db9bad5b173ddd7905ca76f72f06a462385c41b993e199df5c09cd97cb0"} Oct 31 04:24:48.982797 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:48.982766 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-b8bb1e-openshift-default-557bbf897c-phbbm" Oct 31 04:24:48.984191 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:48.984162 4376 patch_prober.go:29] interesting pod/e2e-ipp-b-b8bb1e-openshift-default-557bbf897c-phbbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body= Oct 31 04:24:48.984346 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:48.984220 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-b8bb1e-openshift-default-557bbf897c-phbbm" podUID="61a59ba0-c79d-4942-8b8a-1493cb9e9182" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" Oct 31 04:24:49.983341 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:49.983287 4376 patch_prober.go:29] interesting pod/e2e-ipp-b-b8bb1e-openshift-default-557bbf897c-phbbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body= Oct 31 04:24:49.992481 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:49.992428 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-b8bb1e-openshift-default-557bbf897c-phbbm" podUID="61a59ba0-c79d-4942-8b8a-1493cb9e9182" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" Oct 31 04:24:50.084417 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:50.082531 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b8bb1e-84466f94c-pjkmm"] Oct 31 04:24:50.119226 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:50.119079 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b8bb1e-8bfbcf887-kchch"] Oct 31 04:24:50.990672 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:50.990636 4376 patch_prober.go:29] interesting pod/e2e-ipp-b-b8bb1e-openshift-default-557bbf897c-phbbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body= Oct 31 04:24:50.991235 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:50.991201 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-b8bb1e-openshift-default-557bbf897c-phbbm" podUID="61a59ba0-c79d-4942-8b8a-1493cb9e9182" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" Oct 31 04:24:51.983443 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:51.983410 4376 patch_prober.go:29] interesting pod/e2e-ipp-b-b8bb1e-openshift-default-557bbf897c-phbbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body= Oct 31 04:24:51.983635 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:51.983462 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-b8bb1e-openshift-default-557bbf897c-phbbm" podUID="61a59ba0-c79d-4942-8b8a-1493cb9e9182" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" Oct 31 04:24:52.985760 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:52.985682 4376 patch_prober.go:29] interesting pod/e2e-ipp-b-b8bb1e-openshift-default-557bbf897c-phbbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body= Oct 31 04:24:52.985760 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:52.985740 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-b8bb1e-openshift-default-557bbf897c-phbbm" podUID="61a59ba0-c79d-4942-8b8a-1493cb9e9182" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" Oct 31 04:24:53.025951 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:53.025917 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9154a6-85c9f9448c-d4pt4" Oct 31 04:24:53.992505 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:53.992457 4376 patch_prober.go:29] interesting pod/e2e-ipp-b-b8bb1e-openshift-default-557bbf897c-phbbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body= Oct 31 04:24:53.993038 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:53.992541 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-b8bb1e-openshift-default-557bbf897c-phbbm" podUID="61a59ba0-c79d-4942-8b8a-1493cb9e9182" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" Oct 31 04:24:54.988877 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:54.988822 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-b8bb1e-openshift-default-557bbf897c-phbbm" Oct 31 04:24:54.989509 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:54.989484 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-b8bb1e-openshift-default-557bbf897c-phbbm" Oct 31 04:24:54.990127 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:54.990108 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-b8bb1e-openshift-default-557bbf897c-phbbm" Oct 31 04:24:55.083461 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:55.083404 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-b8bb1e-84466f94c-pjkmm" podUID="4c454206-433b-4b01-92cd-fe229ee6a4de" containerName="payload-processing" containerID="cri-o://694e697cd50dcb0a7656d8013c1e7300b471e9c89bb1761a7bf85b1086b1f7a5" gracePeriod=25 Oct 31 04:24:55.122198 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:55.122145 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b8bb1e-8bfbcf887-kchch" podUID="ff8b6e9e-7dce-41d2-95be-4e487d8fd427" containerName="payload-pre-processing" containerID="cri-o://9aa9d5588dd8ba1ee593665deea69c189aa565c7108561293ff67d8bdb6b95af" gracePeriod=25 Oct 31 04:24:55.415043 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:55.413936 4376 generic.go:356] "Generic (PLEG): container finished" podID="4c454206-433b-4b01-92cd-fe229ee6a4de" containerID="694e697cd50dcb0a7656d8013c1e7300b471e9c89bb1761a7bf85b1086b1f7a5" exitCode=0 Oct 31 04:24:55.415043 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:55.414067 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b8bb1e-84466f94c-pjkmm" event={"ID":"4c454206-433b-4b01-92cd-fe229ee6a4de","Type":"ContainerDied","Data":"694e697cd50dcb0a7656d8013c1e7300b471e9c89bb1761a7bf85b1086b1f7a5"} Oct 31 04:24:55.415043 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:55.414123 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b8bb1e-84466f94c-pjkmm" event={"ID":"4c454206-433b-4b01-92cd-fe229ee6a4de","Type":"ContainerDied","Data":"aaa92d3a7afbf9c31eabd9fa60a2d58308717ad8446552b2a978eb8a619df914"} Oct 31 04:24:55.415043 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:55.414150 4376 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aaa92d3a7afbf9c31eabd9fa60a2d58308717ad8446552b2a978eb8a619df914" Oct 31 04:24:55.417263 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:55.417217 4376 generic.go:356] "Generic (PLEG): container finished" podID="ff8b6e9e-7dce-41d2-95be-4e487d8fd427" containerID="9aa9d5588dd8ba1ee593665deea69c189aa565c7108561293ff67d8bdb6b95af" exitCode=0 Oct 31 04:24:55.418408 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:55.418381 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b8bb1e-8bfbcf887-kchch" event={"ID":"ff8b6e9e-7dce-41d2-95be-4e487d8fd427","Type":"ContainerDied","Data":"9aa9d5588dd8ba1ee593665deea69c189aa565c7108561293ff67d8bdb6b95af"} Oct 31 04:24:55.421259 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:55.421239 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b8bb1e-84466f94c-pjkmm" Oct 31 04:24:55.452178 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:55.452152 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b8bb1e-8bfbcf887-kchch" Oct 31 04:24:55.499278 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:55.499192 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff8b6e9e-7dce-41d2-95be-4e487d8fd427-kube-api-access-hnwg5\" (UniqueName: \"kubernetes.io/projected/ff8b6e9e-7dce-41d2-95be-4e487d8fd427-kube-api-access-hnwg5\") pod \"ff8b6e9e-7dce-41d2-95be-4e487d8fd427\" (UID: \"ff8b6e9e-7dce-41d2-95be-4e487d8fd427\") " Oct 31 04:24:55.499529 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:55.499508 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c454206-433b-4b01-92cd-fe229ee6a4de-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c454206-433b-4b01-92cd-fe229ee6a4de-plugins-config-volume\") pod \"4c454206-433b-4b01-92cd-fe229ee6a4de\" (UID: \"4c454206-433b-4b01-92cd-fe229ee6a4de\") " Oct 31 04:24:55.499618 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:55.499554 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c454206-433b-4b01-92cd-fe229ee6a4de-kube-api-access-jlfwr\" (UniqueName: \"kubernetes.io/projected/4c454206-433b-4b01-92cd-fe229ee6a4de-kube-api-access-jlfwr\") pod \"4c454206-433b-4b01-92cd-fe229ee6a4de\" (UID: \"4c454206-433b-4b01-92cd-fe229ee6a4de\") " Oct 31 04:24:55.499676 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:55.499627 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff8b6e9e-7dce-41d2-95be-4e487d8fd427-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff8b6e9e-7dce-41d2-95be-4e487d8fd427-plugins-config-volume\") pod \"ff8b6e9e-7dce-41d2-95be-4e487d8fd427\" (UID: \"ff8b6e9e-7dce-41d2-95be-4e487d8fd427\") " Oct 31 04:24:55.500815 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:55.499934 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c454206-433b-4b01-92cd-fe229ee6a4de-plugins-config-volume" pod "4c454206-433b-4b01-92cd-fe229ee6a4de" (UID: "4c454206-433b-4b01-92cd-fe229ee6a4de"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:24:55.500815 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:55.500149 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff8b6e9e-7dce-41d2-95be-4e487d8fd427-plugins-config-volume" pod "ff8b6e9e-7dce-41d2-95be-4e487d8fd427" (UID: "ff8b6e9e-7dce-41d2-95be-4e487d8fd427"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:24:55.500815 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:55.500691 4376 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c454206-433b-4b01-92cd-fe229ee6a4de-plugins-config-volume\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:24:55.501065 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:55.501030 4376 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff8b6e9e-7dce-41d2-95be-4e487d8fd427-plugins-config-volume\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:24:55.510102 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:55.503939 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c454206-433b-4b01-92cd-fe229ee6a4de-kube-api-access-jlfwr" pod "4c454206-433b-4b01-92cd-fe229ee6a4de" (UID: "4c454206-433b-4b01-92cd-fe229ee6a4de"). InnerVolumeSpecName "kube-api-access-jlfwr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:24:55.510102 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:55.504123 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff8b6e9e-7dce-41d2-95be-4e487d8fd427-kube-api-access-hnwg5" pod "ff8b6e9e-7dce-41d2-95be-4e487d8fd427" (UID: "ff8b6e9e-7dce-41d2-95be-4e487d8fd427"). InnerVolumeSpecName "kube-api-access-hnwg5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:24:55.602498 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:55.602459 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jlfwr\" (UniqueName: \"kubernetes.io/projected/4c454206-433b-4b01-92cd-fe229ee6a4de-kube-api-access-jlfwr\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:24:55.602498 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:55.602497 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hnwg5\" (UniqueName: \"kubernetes.io/projected/ff8b6e9e-7dce-41d2-95be-4e487d8fd427-kube-api-access-hnwg5\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:24:56.426980 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:56.426945 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b8bb1e-8bfbcf887-kchch" Oct 31 04:24:56.426980 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:56.426975 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b8bb1e-8bfbcf887-kchch" event={"ID":"ff8b6e9e-7dce-41d2-95be-4e487d8fd427","Type":"ContainerDied","Data":"2cd652c2865d5b82b93449a7d308e253777fd30591f00a3e3ae21e3e0b239884"} Oct 31 04:24:56.427513 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:56.427041 4376 scope.go:122] "RemoveContainer" containerID="9aa9d5588dd8ba1ee593665deea69c189aa565c7108561293ff67d8bdb6b95af" Oct 31 04:24:56.427513 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:56.427156 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b8bb1e-84466f94c-pjkmm" Oct 31 04:24:56.483950 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:56.483912 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b8bb1e-84466f94c-pjkmm"] Oct 31 04:24:56.494604 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:56.494574 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b8bb1e-84466f94c-pjkmm"] Oct 31 04:24:56.521112 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:56.521079 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b8bb1e-8bfbcf887-kchch"] Oct 31 04:24:56.525152 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:56.525126 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b8bb1e-8bfbcf887-kchch"] Oct 31 04:24:58.446807 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:58.446772 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c454206-433b-4b01-92cd-fe229ee6a4de" path="/var/lib/kubelet/pods/4c454206-433b-4b01-92cd-fe229ee6a4de/volumes" Oct 31 04:24:58.447443 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:24:58.447423 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff8b6e9e-7dce-41d2-95be-4e487d8fd427" path="/var/lib/kubelet/pods/ff8b6e9e-7dce-41d2-95be-4e487d8fd427/volumes" Oct 31 04:25:05.597351 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:05.596513 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 04:25:11.004147 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:25:11.004104 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 04:25:15.989244 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:15.989192 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b8bb1e-74c8bd9999-427ss"] Oct 31 04:25:15.991006 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:15.990976 4376 cpu_manager.go:414] "containerMap: removing container" podUID="ff8b6e9e-7dce-41d2-95be-4e487d8fd427" containerName="payload-pre-processing" Oct 31 04:25:15.991174 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:15.991160 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff8b6e9e-7dce-41d2-95be-4e487d8fd427" containerName="payload-pre-processing" Oct 31 04:25:15.991386 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:15.991370 4376 cpu_manager.go:414] "containerMap: removing container" podUID="4c454206-433b-4b01-92cd-fe229ee6a4de" containerName="payload-processing" Oct 31 04:25:15.991508 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:15.991495 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c454206-433b-4b01-92cd-fe229ee6a4de" containerName="payload-processing" Oct 31 04:25:15.991873 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:15.991853 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff8b6e9e-7dce-41d2-95be-4e487d8fd427" containerName="payload-pre-processing" Oct 31 04:25:15.992058 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:15.992043 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c454206-433b-4b01-92cd-fe229ee6a4de" containerName="payload-processing" Oct 31 04:25:16.007813 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:16.007761 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b8bb1e-74c8bd9999-427ss" Oct 31 04:25:16.053020 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:16.052979 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b8bb1e-74c8bd9999-427ss"] Oct 31 04:25:16.084830 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:16.084793 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-b8bb1e\"" Oct 31 04:25:16.085019 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:16.084978 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-b8bb1e-dockercfg-kzdg5\"" Oct 31 04:25:16.130236 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:16.130189 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vk8tq\" (UniqueName: \"kubernetes.io/projected/d27ea922-cf10-4656-8166-3bb39619ae9a-kube-api-access-vk8tq\") pod \"payload-pre-processing-e2e-ipp-b-b8bb1e-74c8bd9999-427ss\" (UID: \"d27ea922-cf10-4656-8166-3bb39619ae9a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b8bb1e-74c8bd9999-427ss" Oct 31 04:25:16.130787 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:16.130765 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d27ea922-cf10-4656-8166-3bb39619ae9a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-b8bb1e-74c8bd9999-427ss\" (UID: \"d27ea922-cf10-4656-8166-3bb39619ae9a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b8bb1e-74c8bd9999-427ss" Oct 31 04:25:16.233543 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:16.233502 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d27ea922-cf10-4656-8166-3bb39619ae9a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-b8bb1e-74c8bd9999-427ss\" (UID: \"d27ea922-cf10-4656-8166-3bb39619ae9a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b8bb1e-74c8bd9999-427ss" Oct 31 04:25:16.234608 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:16.233654 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vk8tq\" (UniqueName: \"kubernetes.io/projected/d27ea922-cf10-4656-8166-3bb39619ae9a-kube-api-access-vk8tq\") pod \"payload-pre-processing-e2e-ipp-b-b8bb1e-74c8bd9999-427ss\" (UID: \"d27ea922-cf10-4656-8166-3bb39619ae9a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b8bb1e-74c8bd9999-427ss" Oct 31 04:25:16.234788 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:16.234762 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d27ea922-cf10-4656-8166-3bb39619ae9a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-b8bb1e-74c8bd9999-427ss\" (UID: \"d27ea922-cf10-4656-8166-3bb39619ae9a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b8bb1e-74c8bd9999-427ss" Oct 31 04:25:16.259394 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:16.259290 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vk8tq\" (UniqueName: \"kubernetes.io/projected/d27ea922-cf10-4656-8166-3bb39619ae9a-kube-api-access-vk8tq\") pod \"payload-pre-processing-e2e-ipp-b-b8bb1e-74c8bd9999-427ss\" (UID: \"d27ea922-cf10-4656-8166-3bb39619ae9a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b8bb1e-74c8bd9999-427ss" Oct 31 04:25:16.403773 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:16.403727 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b8bb1e-74c8bd9999-427ss" Oct 31 04:25:16.798361 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:16.798319 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b8bb1e-74c8bd9999-427ss"] Oct 31 04:25:17.804898 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:17.804865 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b8bb1e-74c8bd9999-427ss" event={"ID":"d27ea922-cf10-4656-8166-3bb39619ae9a","Type":"ContainerStarted","Data":"ceed94c185cdc7020584ae174af089f4244de7f145f06d7a65c8a8d7a431f4b6"} Oct 31 04:25:17.805359 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:17.804916 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b8bb1e-74c8bd9999-427ss" event={"ID":"d27ea922-cf10-4656-8166-3bb39619ae9a","Type":"ContainerStarted","Data":"6efa9bd8b941eb4d396bbc669e3576dce69c2203d8662605b3100572071fee92"} Oct 31 04:25:17.805359 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:17.805083 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b8bb1e-74c8bd9999-427ss" Oct 31 04:25:17.849662 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:17.849606 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b8bb1e-74c8bd9999-427ss" podStartSLOduration=2.849591573 podStartE2EDuration="2.849591573s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:25:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:25:17.846314772 +0000 UTC m=+1265.560649391" watchObservedRunningTime="2026-10-31 04:25:17.849591573 +0000 UTC m=+1265.563926189" Oct 31 04:25:21.887251 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:21.887216 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-5shj7"] Oct 31 04:25:21.887720 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:21.887441 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-5shj7" podUID="982c902b-4689-4eb3-93fb-8930a3136465" containerName="istio-proxy" containerID="cri-o://20a52217f980f305428621e7a04e253804041300b42a6a00ed24f737f1b6a78d" gracePeriod=30 Oct 31 04:25:21.900614 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:21.900584 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-pwfbr"] Oct 31 04:25:21.900884 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:21.900855 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-pwfbr" podUID="91eecd3e-fc45-4cf9-8515-a0f723aa9968" containerName="istio-proxy" containerID="cri-o://88f96685a490995d2b8220307831114d1f9093d62957f8abb16c30127d85ba80" gracePeriod=30 Oct 31 04:25:22.030041 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:22.030006 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86b5bede-5sf9t"] Oct 31 04:25:22.034501 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:22.034472 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86b5bede-5sf9t"] Oct 31 04:25:22.447189 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:22.447155 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7af9e9fd-7429-41b0-a38e-637c94a20fe4" path="/var/lib/kubelet/pods/7af9e9fd-7429-41b0-a38e-637c94a20fe4/volumes" Oct 31 04:25:24.808647 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:24.808605 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7b4185-898d6c9df-fjswd"] Oct 31 04:25:24.842254 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:24.842224 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7b4185-7848cfbf55-jjpxl"] Oct 31 04:25:27.199928 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:27.199897 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-5shj7" Oct 31 04:25:27.249369 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:27.249314 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-pwfbr" Oct 31 04:25:27.332011 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:27.331979 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/982c902b-4689-4eb3-93fb-8930a3136465-istio-token\" (UniqueName: \"kubernetes.io/projected/982c902b-4689-4eb3-93fb-8930a3136465-istio-token\") pod \"982c902b-4689-4eb3-93fb-8930a3136465\" (UID: \"982c902b-4689-4eb3-93fb-8930a3136465\") " Oct 31 04:25:27.332210 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:27.332134 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91eecd3e-fc45-4cf9-8515-a0f723aa9968-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91eecd3e-fc45-4cf9-8515-a0f723aa9968-credential-socket\") pod \"91eecd3e-fc45-4cf9-8515-a0f723aa9968\" (UID: \"91eecd3e-fc45-4cf9-8515-a0f723aa9968\") " Oct 31 04:25:27.332210 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:27.332187 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91eecd3e-fc45-4cf9-8515-a0f723aa9968-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91eecd3e-fc45-4cf9-8515-a0f723aa9968-workload-certs\") pod \"91eecd3e-fc45-4cf9-8515-a0f723aa9968\" (UID: \"91eecd3e-fc45-4cf9-8515-a0f723aa9968\") " Oct 31 04:25:27.332360 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:27.332228 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91eecd3e-fc45-4cf9-8515-a0f723aa9968-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91eecd3e-fc45-4cf9-8515-a0f723aa9968-istio-envoy\") pod \"91eecd3e-fc45-4cf9-8515-a0f723aa9968\" (UID: \"91eecd3e-fc45-4cf9-8515-a0f723aa9968\") " Oct 31 04:25:27.332360 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:27.332272 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/982c902b-4689-4eb3-93fb-8930a3136465-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/982c902b-4689-4eb3-93fb-8930a3136465-istiod-ca-cert\") pod \"982c902b-4689-4eb3-93fb-8930a3136465\" (UID: \"982c902b-4689-4eb3-93fb-8930a3136465\") " Oct 31 04:25:27.332461 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:27.332366 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/982c902b-4689-4eb3-93fb-8930a3136465-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/982c902b-4689-4eb3-93fb-8930a3136465-istio-podinfo\") pod \"982c902b-4689-4eb3-93fb-8930a3136465\" (UID: \"982c902b-4689-4eb3-93fb-8930a3136465\") " Oct 31 04:25:27.332514 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:27.332459 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91eecd3e-fc45-4cf9-8515-a0f723aa9968-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91eecd3e-fc45-4cf9-8515-a0f723aa9968-istiod-ca-cert\") pod \"91eecd3e-fc45-4cf9-8515-a0f723aa9968\" (UID: \"91eecd3e-fc45-4cf9-8515-a0f723aa9968\") " Oct 31 04:25:27.332570 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:27.332514 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/91eecd3e-fc45-4cf9-8515-a0f723aa9968-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91eecd3e-fc45-4cf9-8515-a0f723aa9968-istio-podinfo\") pod \"91eecd3e-fc45-4cf9-8515-a0f723aa9968\" (UID: \"91eecd3e-fc45-4cf9-8515-a0f723aa9968\") " Oct 31 04:25:27.332623 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:27.332589 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/982c902b-4689-4eb3-93fb-8930a3136465-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/982c902b-4689-4eb3-93fb-8930a3136465-credential-socket\") pod \"982c902b-4689-4eb3-93fb-8930a3136465\" (UID: \"982c902b-4689-4eb3-93fb-8930a3136465\") " Oct 31 04:25:27.332807 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:27.332626 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/982c902b-4689-4eb3-93fb-8930a3136465-kube-api-access-glhxj\" (UniqueName: \"kubernetes.io/projected/982c902b-4689-4eb3-93fb-8930a3136465-kube-api-access-glhxj\") pod \"982c902b-4689-4eb3-93fb-8930a3136465\" (UID: \"982c902b-4689-4eb3-93fb-8930a3136465\") " Oct 31 04:25:27.332807 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:27.332660 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91eecd3e-fc45-4cf9-8515-a0f723aa9968-istio-token\" (UniqueName: \"kubernetes.io/projected/91eecd3e-fc45-4cf9-8515-a0f723aa9968-istio-token\") pod \"91eecd3e-fc45-4cf9-8515-a0f723aa9968\" (UID: \"91eecd3e-fc45-4cf9-8515-a0f723aa9968\") " Oct 31 04:25:27.332807 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:27.332722 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91eecd3e-fc45-4cf9-8515-a0f723aa9968-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91eecd3e-fc45-4cf9-8515-a0f723aa9968-workload-socket\") pod \"91eecd3e-fc45-4cf9-8515-a0f723aa9968\" (UID: \"91eecd3e-fc45-4cf9-8515-a0f723aa9968\") " Oct 31 04:25:27.332807 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:27.332756 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/982c902b-4689-4eb3-93fb-8930a3136465-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/982c902b-4689-4eb3-93fb-8930a3136465-workload-certs\") pod \"982c902b-4689-4eb3-93fb-8930a3136465\" (UID: \"982c902b-4689-4eb3-93fb-8930a3136465\") " Oct 31 04:25:27.332807 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:27.332798 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/982c902b-4689-4eb3-93fb-8930a3136465-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/982c902b-4689-4eb3-93fb-8930a3136465-workload-socket\") pod \"982c902b-4689-4eb3-93fb-8930a3136465\" (UID: \"982c902b-4689-4eb3-93fb-8930a3136465\") " Oct 31 04:25:27.333059 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:27.332833 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/982c902b-4689-4eb3-93fb-8930a3136465-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/982c902b-4689-4eb3-93fb-8930a3136465-istio-envoy\") pod \"982c902b-4689-4eb3-93fb-8930a3136465\" (UID: \"982c902b-4689-4eb3-93fb-8930a3136465\") " Oct 31 04:25:27.333059 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:27.332871 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91eecd3e-fc45-4cf9-8515-a0f723aa9968-kube-api-access-wqsjj\" (UniqueName: \"kubernetes.io/projected/91eecd3e-fc45-4cf9-8515-a0f723aa9968-kube-api-access-wqsjj\") pod \"91eecd3e-fc45-4cf9-8515-a0f723aa9968\" (UID: \"91eecd3e-fc45-4cf9-8515-a0f723aa9968\") " Oct 31 04:25:27.333059 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:27.332947 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91eecd3e-fc45-4cf9-8515-a0f723aa9968-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91eecd3e-fc45-4cf9-8515-a0f723aa9968-istio-data\") pod \"91eecd3e-fc45-4cf9-8515-a0f723aa9968\" (UID: \"91eecd3e-fc45-4cf9-8515-a0f723aa9968\") " Oct 31 04:25:27.333059 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:27.332974 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/982c902b-4689-4eb3-93fb-8930a3136465-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/982c902b-4689-4eb3-93fb-8930a3136465-istio-data\") pod \"982c902b-4689-4eb3-93fb-8930a3136465\" (UID: \"982c902b-4689-4eb3-93fb-8930a3136465\") " Oct 31 04:25:27.333059 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:27.333018 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91eecd3e-fc45-4cf9-8515-a0f723aa9968-workload-certs" pod "91eecd3e-fc45-4cf9-8515-a0f723aa9968" (UID: "91eecd3e-fc45-4cf9-8515-a0f723aa9968"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:25:27.333283 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:27.333076 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/982c902b-4689-4eb3-93fb-8930a3136465-istiod-ca-cert" pod "982c902b-4689-4eb3-93fb-8930a3136465" (UID: "982c902b-4689-4eb3-93fb-8930a3136465"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:25:27.333378 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:27.333363 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/982c902b-4689-4eb3-93fb-8930a3136465-workload-certs" pod "982c902b-4689-4eb3-93fb-8930a3136465" (UID: "982c902b-4689-4eb3-93fb-8930a3136465"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:25:27.333471 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:27.333451 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91eecd3e-fc45-4cf9-8515-a0f723aa9968-istiod-ca-cert" pod "91eecd3e-fc45-4cf9-8515-a0f723aa9968" (UID: "91eecd3e-fc45-4cf9-8515-a0f723aa9968"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:25:27.333600 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:27.333581 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/982c902b-4689-4eb3-93fb-8930a3136465-istio-data" pod "982c902b-4689-4eb3-93fb-8930a3136465" (UID: "982c902b-4689-4eb3-93fb-8930a3136465"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:25:27.333917 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:27.333886 4376 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91eecd3e-fc45-4cf9-8515-a0f723aa9968-workload-certs\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:25:27.333917 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:27.333916 4376 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/982c902b-4689-4eb3-93fb-8930a3136465-istiod-ca-cert\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:25:27.334045 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:27.333929 4376 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91eecd3e-fc45-4cf9-8515-a0f723aa9968-istiod-ca-cert\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:25:27.334045 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:27.333941 4376 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/982c902b-4689-4eb3-93fb-8930a3136465-workload-certs\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:25:27.334045 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:27.333954 4376 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/982c902b-4689-4eb3-93fb-8930a3136465-istio-data\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:25:27.334162 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:27.334149 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/982c902b-4689-4eb3-93fb-8930a3136465-credential-socket" pod "982c902b-4689-4eb3-93fb-8930a3136465" (UID: "982c902b-4689-4eb3-93fb-8930a3136465"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:25:27.334904 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:27.334684 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/982c902b-4689-4eb3-93fb-8930a3136465-workload-socket" pod "982c902b-4689-4eb3-93fb-8930a3136465" (UID: "982c902b-4689-4eb3-93fb-8930a3136465"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:25:27.334904 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:27.334741 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91eecd3e-fc45-4cf9-8515-a0f723aa9968-credential-socket" pod "91eecd3e-fc45-4cf9-8515-a0f723aa9968" (UID: "91eecd3e-fc45-4cf9-8515-a0f723aa9968"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:25:27.334904 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:27.334850 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91eecd3e-fc45-4cf9-8515-a0f723aa9968-istio-data" pod "91eecd3e-fc45-4cf9-8515-a0f723aa9968" (UID: "91eecd3e-fc45-4cf9-8515-a0f723aa9968"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:25:27.335197 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:27.335162 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91eecd3e-fc45-4cf9-8515-a0f723aa9968-workload-socket" pod "91eecd3e-fc45-4cf9-8515-a0f723aa9968" (UID: "91eecd3e-fc45-4cf9-8515-a0f723aa9968"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:25:27.336117 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:27.335399 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/982c902b-4689-4eb3-93fb-8930a3136465-istio-podinfo" pod "982c902b-4689-4eb3-93fb-8930a3136465" (UID: "982c902b-4689-4eb3-93fb-8930a3136465"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 04:25:27.336117 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:27.335519 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/982c902b-4689-4eb3-93fb-8930a3136465-istio-token" pod "982c902b-4689-4eb3-93fb-8930a3136465" (UID: "982c902b-4689-4eb3-93fb-8930a3136465"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:25:27.337865 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:27.337840 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91eecd3e-fc45-4cf9-8515-a0f723aa9968-kube-api-access-wqsjj" pod "91eecd3e-fc45-4cf9-8515-a0f723aa9968" (UID: "91eecd3e-fc45-4cf9-8515-a0f723aa9968"). InnerVolumeSpecName "kube-api-access-wqsjj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:25:27.339278 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:27.339244 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91eecd3e-fc45-4cf9-8515-a0f723aa9968-istio-token" pod "91eecd3e-fc45-4cf9-8515-a0f723aa9968" (UID: "91eecd3e-fc45-4cf9-8515-a0f723aa9968"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:25:27.342494 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:27.342465 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/982c902b-4689-4eb3-93fb-8930a3136465-kube-api-access-glhxj" pod "982c902b-4689-4eb3-93fb-8930a3136465" (UID: "982c902b-4689-4eb3-93fb-8930a3136465"). InnerVolumeSpecName "kube-api-access-glhxj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:25:27.344731 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:27.344678 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91eecd3e-fc45-4cf9-8515-a0f723aa9968-istio-envoy" pod "91eecd3e-fc45-4cf9-8515-a0f723aa9968" (UID: "91eecd3e-fc45-4cf9-8515-a0f723aa9968"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:25:27.344731 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:27.344698 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/91eecd3e-fc45-4cf9-8515-a0f723aa9968-istio-podinfo" pod "91eecd3e-fc45-4cf9-8515-a0f723aa9968" (UID: "91eecd3e-fc45-4cf9-8515-a0f723aa9968"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 04:25:27.345890 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:27.345866 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/982c902b-4689-4eb3-93fb-8930a3136465-istio-envoy" pod "982c902b-4689-4eb3-93fb-8930a3136465" (UID: "982c902b-4689-4eb3-93fb-8930a3136465"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:25:27.436292 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:27.436110 4376 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/982c902b-4689-4eb3-93fb-8930a3136465-istio-podinfo\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:25:27.436875 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:27.436846 4376 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91eecd3e-fc45-4cf9-8515-a0f723aa9968-istio-podinfo\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:25:27.437083 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:27.436907 4376 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/982c902b-4689-4eb3-93fb-8930a3136465-credential-socket\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:25:27.437083 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:27.436926 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-glhxj\" (UniqueName: \"kubernetes.io/projected/982c902b-4689-4eb3-93fb-8930a3136465-kube-api-access-glhxj\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:25:27.437083 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:27.436940 4376 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91eecd3e-fc45-4cf9-8515-a0f723aa9968-istio-token\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:25:27.437083 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:27.436954 4376 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91eecd3e-fc45-4cf9-8515-a0f723aa9968-workload-socket\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:25:27.437083 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:27.436987 4376 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/982c902b-4689-4eb3-93fb-8930a3136465-workload-socket\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:25:27.437083 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:27.437000 4376 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/982c902b-4689-4eb3-93fb-8930a3136465-istio-envoy\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:25:27.437083 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:27.437015 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqsjj\" (UniqueName: \"kubernetes.io/projected/91eecd3e-fc45-4cf9-8515-a0f723aa9968-kube-api-access-wqsjj\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:25:27.437083 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:27.437026 4376 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91eecd3e-fc45-4cf9-8515-a0f723aa9968-istio-data\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:25:27.437083 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:27.437058 4376 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/982c902b-4689-4eb3-93fb-8930a3136465-istio-token\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:25:27.437083 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:27.437072 4376 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91eecd3e-fc45-4cf9-8515-a0f723aa9968-credential-socket\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:25:27.437083 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:27.437097 4376 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91eecd3e-fc45-4cf9-8515-a0f723aa9968-istio-envoy\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:25:27.873546 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:27.873508 4376 generic.go:356] "Generic (PLEG): container finished" podID="982c902b-4689-4eb3-93fb-8930a3136465" containerID="20a52217f980f305428621e7a04e253804041300b42a6a00ed24f737f1b6a78d" exitCode=0 Oct 31 04:25:27.873772 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:27.873583 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-5shj7" Oct 31 04:25:27.873772 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:27.873593 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-5shj7" event={"ID":"982c902b-4689-4eb3-93fb-8930a3136465","Type":"ContainerDied","Data":"20a52217f980f305428621e7a04e253804041300b42a6a00ed24f737f1b6a78d"} Oct 31 04:25:27.873772 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:27.873683 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-5shj7" event={"ID":"982c902b-4689-4eb3-93fb-8930a3136465","Type":"ContainerDied","Data":"e73841b8a77074b8e5b28ac1630b8b6effee89b66fb20ed0db393ed9c99c8b04"} Oct 31 04:25:27.873772 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:27.873708 4376 scope.go:122] "RemoveContainer" containerID="20a52217f980f305428621e7a04e253804041300b42a6a00ed24f737f1b6a78d" Oct 31 04:25:27.875221 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:27.875195 4376 generic.go:356] "Generic (PLEG): container finished" podID="91eecd3e-fc45-4cf9-8515-a0f723aa9968" containerID="88f96685a490995d2b8220307831114d1f9093d62957f8abb16c30127d85ba80" exitCode=0 Oct 31 04:25:27.875364 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:27.875271 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-pwfbr" Oct 31 04:25:27.875364 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:27.875300 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-pwfbr" event={"ID":"91eecd3e-fc45-4cf9-8515-a0f723aa9968","Type":"ContainerDied","Data":"88f96685a490995d2b8220307831114d1f9093d62957f8abb16c30127d85ba80"} Oct 31 04:25:27.875460 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:27.875377 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-pwfbr" event={"ID":"91eecd3e-fc45-4cf9-8515-a0f723aa9968","Type":"ContainerDied","Data":"2874989d93930c463e4c261d5fb43d77818c3ab82b7fd26df628847e05e7adc3"} Oct 31 04:25:27.886653 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:27.886634 4376 scope.go:122] "RemoveContainer" containerID="20a52217f980f305428621e7a04e253804041300b42a6a00ed24f737f1b6a78d" Oct 31 04:25:27.887139 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:25:27.887110 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20a52217f980f305428621e7a04e253804041300b42a6a00ed24f737f1b6a78d\": container with ID starting with 20a52217f980f305428621e7a04e253804041300b42a6a00ed24f737f1b6a78d not found: ID does not exist" containerID="20a52217f980f305428621e7a04e253804041300b42a6a00ed24f737f1b6a78d" Oct 31 04:25:27.887214 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:27.887143 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20a52217f980f305428621e7a04e253804041300b42a6a00ed24f737f1b6a78d"} err="failed to get container status \"20a52217f980f305428621e7a04e253804041300b42a6a00ed24f737f1b6a78d\": rpc error: code = NotFound desc = could not find container \"20a52217f980f305428621e7a04e253804041300b42a6a00ed24f737f1b6a78d\": container with ID starting with 20a52217f980f305428621e7a04e253804041300b42a6a00ed24f737f1b6a78d not found: ID does not exist" Oct 31 04:25:27.887214 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:27.887165 4376 scope.go:122] "RemoveContainer" containerID="88f96685a490995d2b8220307831114d1f9093d62957f8abb16c30127d85ba80" Oct 31 04:25:27.898806 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:27.898785 4376 scope.go:122] "RemoveContainer" containerID="88f96685a490995d2b8220307831114d1f9093d62957f8abb16c30127d85ba80" Oct 31 04:25:27.899127 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:25:27.899105 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88f96685a490995d2b8220307831114d1f9093d62957f8abb16c30127d85ba80\": container with ID starting with 88f96685a490995d2b8220307831114d1f9093d62957f8abb16c30127d85ba80 not found: ID does not exist" containerID="88f96685a490995d2b8220307831114d1f9093d62957f8abb16c30127d85ba80" Oct 31 04:25:27.899204 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:27.899139 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88f96685a490995d2b8220307831114d1f9093d62957f8abb16c30127d85ba80"} err="failed to get container status \"88f96685a490995d2b8220307831114d1f9093d62957f8abb16c30127d85ba80\": rpc error: code = NotFound desc = could not find container \"88f96685a490995d2b8220307831114d1f9093d62957f8abb16c30127d85ba80\": container with ID starting with 88f96685a490995d2b8220307831114d1f9093d62957f8abb16c30127d85ba80 not found: ID does not exist" Oct 31 04:25:27.944187 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:27.944155 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-pwfbr"] Oct 31 04:25:27.955965 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:27.955930 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-pwfbr"] Oct 31 04:25:27.984730 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:27.984695 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-5shj7"] Oct 31 04:25:27.988588 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:27.988561 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-5shj7"] Oct 31 04:25:28.445308 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:28.445273 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91eecd3e-fc45-4cf9-8515-a0f723aa9968" path="/var/lib/kubelet/pods/91eecd3e-fc45-4cf9-8515-a0f723aa9968/volumes" Oct 31 04:25:28.445905 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:28.445889 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="982c902b-4689-4eb3-93fb-8930a3136465" path="/var/lib/kubelet/pods/982c902b-4689-4eb3-93fb-8930a3136465/volumes" Oct 31 04:25:28.814931 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:28.814847 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b8bb1e-74c8bd9999-427ss" Oct 31 04:25:29.809288 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:29.809234 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-7b4185-898d6c9df-fjswd" podUID="83a9eaed-ab99-4fd4-9ffe-a1d863a85cf8" containerName="payload-processing" containerID="cri-o://177ffa7257564a79bf161becd0d8491a0eb1dc62d94a2feb9a3d35c26c731974" gracePeriod=25 Oct 31 04:25:29.843472 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:29.843411 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7b4185-7848cfbf55-jjpxl" podUID="419996ea-d44d-44f4-b414-583068843613" containerName="payload-pre-processing" containerID="cri-o://555127ce7d1b668d1db820998d034a68709197d76886907a2a6a8c9ffbbfa3fa" gracePeriod=25 Oct 31 04:25:30.038043 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:30.037982 4376 generic.go:356] "Generic (PLEG): container finished" podID="419996ea-d44d-44f4-b414-583068843613" containerID="555127ce7d1b668d1db820998d034a68709197d76886907a2a6a8c9ffbbfa3fa" exitCode=0 Oct 31 04:25:30.038218 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:30.038182 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7b4185-7848cfbf55-jjpxl" event={"ID":"419996ea-d44d-44f4-b414-583068843613","Type":"ContainerDied","Data":"555127ce7d1b668d1db820998d034a68709197d76886907a2a6a8c9ffbbfa3fa"} Oct 31 04:25:30.041263 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:30.040565 4376 generic.go:356] "Generic (PLEG): container finished" podID="83a9eaed-ab99-4fd4-9ffe-a1d863a85cf8" containerID="177ffa7257564a79bf161becd0d8491a0eb1dc62d94a2feb9a3d35c26c731974" exitCode=0 Oct 31 04:25:30.041263 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:30.040680 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-7b4185-898d6c9df-fjswd" event={"ID":"83a9eaed-ab99-4fd4-9ffe-a1d863a85cf8","Type":"ContainerDied","Data":"177ffa7257564a79bf161becd0d8491a0eb1dc62d94a2feb9a3d35c26c731974"} Oct 31 04:25:30.161707 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:30.161683 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-7b4185-898d6c9df-fjswd" Oct 31 04:25:30.165373 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:30.165348 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7b4185-7848cfbf55-jjpxl" Oct 31 04:25:30.339447 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:30.339414 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/419996ea-d44d-44f4-b414-583068843613-kube-api-access-vb4gq\" (UniqueName: \"kubernetes.io/projected/419996ea-d44d-44f4-b414-583068843613-kube-api-access-vb4gq\") pod \"419996ea-d44d-44f4-b414-583068843613\" (UID: \"419996ea-d44d-44f4-b414-583068843613\") " Oct 31 04:25:30.339639 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:30.339627 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/419996ea-d44d-44f4-b414-583068843613-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/419996ea-d44d-44f4-b414-583068843613-plugins-config-volume\") pod \"419996ea-d44d-44f4-b414-583068843613\" (UID: \"419996ea-d44d-44f4-b414-583068843613\") " Oct 31 04:25:30.339691 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:30.339656 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83a9eaed-ab99-4fd4-9ffe-a1d863a85cf8-kube-api-access-bxbzf\" (UniqueName: \"kubernetes.io/projected/83a9eaed-ab99-4fd4-9ffe-a1d863a85cf8-kube-api-access-bxbzf\") pod \"83a9eaed-ab99-4fd4-9ffe-a1d863a85cf8\" (UID: \"83a9eaed-ab99-4fd4-9ffe-a1d863a85cf8\") " Oct 31 04:25:30.339737 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:30.339726 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83a9eaed-ab99-4fd4-9ffe-a1d863a85cf8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83a9eaed-ab99-4fd4-9ffe-a1d863a85cf8-plugins-config-volume\") pod \"83a9eaed-ab99-4fd4-9ffe-a1d863a85cf8\" (UID: \"83a9eaed-ab99-4fd4-9ffe-a1d863a85cf8\") " Oct 31 04:25:30.340088 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:30.340055 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/419996ea-d44d-44f4-b414-583068843613-plugins-config-volume" pod "419996ea-d44d-44f4-b414-583068843613" (UID: "419996ea-d44d-44f4-b414-583068843613"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:25:30.340217 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:30.340134 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83a9eaed-ab99-4fd4-9ffe-a1d863a85cf8-plugins-config-volume" pod "83a9eaed-ab99-4fd4-9ffe-a1d863a85cf8" (UID: "83a9eaed-ab99-4fd4-9ffe-a1d863a85cf8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:25:30.340496 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:30.340477 4376 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/419996ea-d44d-44f4-b414-583068843613-plugins-config-volume\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:25:30.340568 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:30.340500 4376 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83a9eaed-ab99-4fd4-9ffe-a1d863a85cf8-plugins-config-volume\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:25:30.341593 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:30.341574 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/419996ea-d44d-44f4-b414-583068843613-kube-api-access-vb4gq" pod "419996ea-d44d-44f4-b414-583068843613" (UID: "419996ea-d44d-44f4-b414-583068843613"). InnerVolumeSpecName "kube-api-access-vb4gq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:25:30.341821 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:30.341800 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83a9eaed-ab99-4fd4-9ffe-a1d863a85cf8-kube-api-access-bxbzf" pod "83a9eaed-ab99-4fd4-9ffe-a1d863a85cf8" (UID: "83a9eaed-ab99-4fd4-9ffe-a1d863a85cf8"). InnerVolumeSpecName "kube-api-access-bxbzf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:25:30.442549 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:30.442471 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxbzf\" (UniqueName: \"kubernetes.io/projected/83a9eaed-ab99-4fd4-9ffe-a1d863a85cf8-kube-api-access-bxbzf\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:25:30.442549 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:30.442505 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vb4gq\" (UniqueName: \"kubernetes.io/projected/419996ea-d44d-44f4-b414-583068843613-kube-api-access-vb4gq\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:25:31.049604 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:31.049575 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-7b4185-898d6c9df-fjswd" Oct 31 04:25:31.050091 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:31.049611 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-7b4185-898d6c9df-fjswd" event={"ID":"83a9eaed-ab99-4fd4-9ffe-a1d863a85cf8","Type":"ContainerDied","Data":"3fe574e047d003a78ffa28ca072ee9f35c6ac3b26ba7f21707b64bb7a515f2c5"} Oct 31 04:25:31.050091 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:31.049679 4376 scope.go:122] "RemoveContainer" containerID="177ffa7257564a79bf161becd0d8491a0eb1dc62d94a2feb9a3d35c26c731974" Oct 31 04:25:31.051606 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:31.051570 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7b4185-7848cfbf55-jjpxl" event={"ID":"419996ea-d44d-44f4-b414-583068843613","Type":"ContainerDied","Data":"88cb932bca2789dfa336a3e95dc2c9d27860c1de9e4148f75a8dc4e85f930bf2"} Oct 31 04:25:31.051606 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:31.051599 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7b4185-7848cfbf55-jjpxl" Oct 31 04:25:31.060553 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:31.060532 4376 scope.go:122] "RemoveContainer" containerID="555127ce7d1b668d1db820998d034a68709197d76886907a2a6a8c9ffbbfa3fa" Oct 31 04:25:31.104067 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:31.103988 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7b4185-7848cfbf55-jjpxl"] Oct 31 04:25:31.116096 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:31.116064 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7b4185-7848cfbf55-jjpxl"] Oct 31 04:25:31.142050 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:31.142014 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7b4185-898d6c9df-fjswd"] Oct 31 04:25:31.145407 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:31.145386 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7b4185-898d6c9df-fjswd"] Oct 31 04:25:32.245357 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:32.236644 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 04:25:32.509371 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:32.509286 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="419996ea-d44d-44f4-b414-583068843613" path="/var/lib/kubelet/pods/419996ea-d44d-44f4-b414-583068843613/volumes" Oct 31 04:25:32.509771 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:32.509757 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83a9eaed-ab99-4fd4-9ffe-a1d863a85cf8" path="/var/lib/kubelet/pods/83a9eaed-ab99-4fd4-9ffe-a1d863a85cf8/volumes" Oct 31 04:25:35.677013 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:35.676970 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-bzbxb"] Oct 31 04:25:35.677610 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:35.677226 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-bzbxb" podUID="1287989d-3ca3-48c0-bfa1-3e6e27b7f06d" containerName="istio-proxy" containerID="cri-o://2d6d4ae8e4be174ccecadedfe719fbf0a903a33f1f647bdd228ba3a74dcd8aa3" gracePeriod=30 Oct 31 04:25:35.690446 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:35.690412 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-cqvwg"] Oct 31 04:25:35.690667 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:35.690644 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-cqvwg" podUID="4b871815-e7d5-42e1-a7b7-4777c82df055" containerName="istio-proxy" containerID="cri-o://5f22b0d5ad7e86b40c8950f28b78f1537b10c187b38651b9b3faff39b698d58b" gracePeriod=30 Oct 31 04:25:41.012392 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:25:41.012006 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 04:25:41.015894 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:41.015868 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-bzbxb" Oct 31 04:25:41.176850 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:41.176812 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1287989d-3ca3-48c0-bfa1-3e6e27b7f06d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1287989d-3ca3-48c0-bfa1-3e6e27b7f06d-workload-socket\") pod \"1287989d-3ca3-48c0-bfa1-3e6e27b7f06d\" (UID: \"1287989d-3ca3-48c0-bfa1-3e6e27b7f06d\") " Oct 31 04:25:41.177068 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:41.176963 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1287989d-3ca3-48c0-bfa1-3e6e27b7f06d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1287989d-3ca3-48c0-bfa1-3e6e27b7f06d-credential-socket\") pod \"1287989d-3ca3-48c0-bfa1-3e6e27b7f06d\" (UID: \"1287989d-3ca3-48c0-bfa1-3e6e27b7f06d\") " Oct 31 04:25:41.177068 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:41.177027 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1287989d-3ca3-48c0-bfa1-3e6e27b7f06d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1287989d-3ca3-48c0-bfa1-3e6e27b7f06d-workload-certs\") pod \"1287989d-3ca3-48c0-bfa1-3e6e27b7f06d\" (UID: \"1287989d-3ca3-48c0-bfa1-3e6e27b7f06d\") " Oct 31 04:25:41.177189 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:41.177071 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1287989d-3ca3-48c0-bfa1-3e6e27b7f06d-workload-socket" pod "1287989d-3ca3-48c0-bfa1-3e6e27b7f06d" (UID: "1287989d-3ca3-48c0-bfa1-3e6e27b7f06d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:25:41.177189 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:41.177107 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1287989d-3ca3-48c0-bfa1-3e6e27b7f06d-kube-api-access-zg5gl\" (UniqueName: \"kubernetes.io/projected/1287989d-3ca3-48c0-bfa1-3e6e27b7f06d-kube-api-access-zg5gl\") pod \"1287989d-3ca3-48c0-bfa1-3e6e27b7f06d\" (UID: \"1287989d-3ca3-48c0-bfa1-3e6e27b7f06d\") " Oct 31 04:25:41.177189 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:41.177142 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1287989d-3ca3-48c0-bfa1-3e6e27b7f06d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1287989d-3ca3-48c0-bfa1-3e6e27b7f06d-istio-data\") pod \"1287989d-3ca3-48c0-bfa1-3e6e27b7f06d\" (UID: \"1287989d-3ca3-48c0-bfa1-3e6e27b7f06d\") " Oct 31 04:25:41.177361 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:41.177238 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1287989d-3ca3-48c0-bfa1-3e6e27b7f06d-istio-token\" (UniqueName: \"kubernetes.io/projected/1287989d-3ca3-48c0-bfa1-3e6e27b7f06d-istio-token\") pod \"1287989d-3ca3-48c0-bfa1-3e6e27b7f06d\" (UID: \"1287989d-3ca3-48c0-bfa1-3e6e27b7f06d\") " Oct 31 04:25:41.177361 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:41.177298 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1287989d-3ca3-48c0-bfa1-3e6e27b7f06d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1287989d-3ca3-48c0-bfa1-3e6e27b7f06d-istio-envoy\") pod \"1287989d-3ca3-48c0-bfa1-3e6e27b7f06d\" (UID: \"1287989d-3ca3-48c0-bfa1-3e6e27b7f06d\") " Oct 31 04:25:41.177469 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:41.177451 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1287989d-3ca3-48c0-bfa1-3e6e27b7f06d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1287989d-3ca3-48c0-bfa1-3e6e27b7f06d-istiod-ca-cert\") pod \"1287989d-3ca3-48c0-bfa1-3e6e27b7f06d\" (UID: \"1287989d-3ca3-48c0-bfa1-3e6e27b7f06d\") " Oct 31 04:25:41.177524 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:41.177482 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1287989d-3ca3-48c0-bfa1-3e6e27b7f06d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1287989d-3ca3-48c0-bfa1-3e6e27b7f06d-istio-podinfo\") pod \"1287989d-3ca3-48c0-bfa1-3e6e27b7f06d\" (UID: \"1287989d-3ca3-48c0-bfa1-3e6e27b7f06d\") " Oct 31 04:25:41.177982 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:41.177961 4376 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1287989d-3ca3-48c0-bfa1-3e6e27b7f06d-workload-socket\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:25:41.178835 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:41.178808 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1287989d-3ca3-48c0-bfa1-3e6e27b7f06d-workload-certs" pod "1287989d-3ca3-48c0-bfa1-3e6e27b7f06d" (UID: "1287989d-3ca3-48c0-bfa1-3e6e27b7f06d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:25:41.179005 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:41.178985 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1287989d-3ca3-48c0-bfa1-3e6e27b7f06d-credential-socket" pod "1287989d-3ca3-48c0-bfa1-3e6e27b7f06d" (UID: "1287989d-3ca3-48c0-bfa1-3e6e27b7f06d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:25:41.179500 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:41.179476 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1287989d-3ca3-48c0-bfa1-3e6e27b7f06d-istio-data" pod "1287989d-3ca3-48c0-bfa1-3e6e27b7f06d" (UID: "1287989d-3ca3-48c0-bfa1-3e6e27b7f06d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:25:41.181502 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:41.181474 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1287989d-3ca3-48c0-bfa1-3e6e27b7f06d-istiod-ca-cert" pod "1287989d-3ca3-48c0-bfa1-3e6e27b7f06d" (UID: "1287989d-3ca3-48c0-bfa1-3e6e27b7f06d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:25:41.182567 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:41.182541 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1287989d-3ca3-48c0-bfa1-3e6e27b7f06d-istio-token" pod "1287989d-3ca3-48c0-bfa1-3e6e27b7f06d" (UID: "1287989d-3ca3-48c0-bfa1-3e6e27b7f06d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:25:41.183092 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:41.183064 4376 generic.go:356] "Generic (PLEG): container finished" podID="4b871815-e7d5-42e1-a7b7-4777c82df055" containerID="5f22b0d5ad7e86b40c8950f28b78f1537b10c187b38651b9b3faff39b698d58b" exitCode=0 Oct 31 04:25:41.183527 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:41.183316 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-cqvwg" event={"ID":"4b871815-e7d5-42e1-a7b7-4777c82df055","Type":"ContainerDied","Data":"5f22b0d5ad7e86b40c8950f28b78f1537b10c187b38651b9b3faff39b698d58b"} Oct 31 04:25:41.183639 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:41.183617 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-cqvwg" event={"ID":"4b871815-e7d5-42e1-a7b7-4777c82df055","Type":"ContainerDied","Data":"2ee2e6604bbf7c8ee7d5c2f548c618dc0f4c1eef5d3e27b6a5a6193e5b55f5dc"} Oct 31 04:25:41.183696 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:41.183659 4376 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2ee2e6604bbf7c8ee7d5c2f548c618dc0f4c1eef5d3e27b6a5a6193e5b55f5dc" Oct 31 04:25:41.184666 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:41.184528 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1287989d-3ca3-48c0-bfa1-3e6e27b7f06d-kube-api-access-zg5gl" pod "1287989d-3ca3-48c0-bfa1-3e6e27b7f06d" (UID: "1287989d-3ca3-48c0-bfa1-3e6e27b7f06d"). InnerVolumeSpecName "kube-api-access-zg5gl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:25:41.185699 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:41.185678 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-cqvwg" Oct 31 04:25:41.185806 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:41.185779 4376 generic.go:356] "Generic (PLEG): container finished" podID="1287989d-3ca3-48c0-bfa1-3e6e27b7f06d" containerID="2d6d4ae8e4be174ccecadedfe719fbf0a903a33f1f647bdd228ba3a74dcd8aa3" exitCode=0 Oct 31 04:25:41.185882 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:41.185865 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-bzbxb" event={"ID":"1287989d-3ca3-48c0-bfa1-3e6e27b7f06d","Type":"ContainerDied","Data":"2d6d4ae8e4be174ccecadedfe719fbf0a903a33f1f647bdd228ba3a74dcd8aa3"} Oct 31 04:25:41.185926 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:41.185913 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-bzbxb" event={"ID":"1287989d-3ca3-48c0-bfa1-3e6e27b7f06d","Type":"ContainerDied","Data":"fcf7203220e74f43754e1b30b33b98729c16438dafaaf273569bcb14f1671136"} Oct 31 04:25:41.185961 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:41.185946 4376 scope.go:122] "RemoveContainer" containerID="2d6d4ae8e4be174ccecadedfe719fbf0a903a33f1f647bdd228ba3a74dcd8aa3" Oct 31 04:25:41.186074 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:41.186059 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-bzbxb" Oct 31 04:25:41.186686 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:41.186644 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1287989d-3ca3-48c0-bfa1-3e6e27b7f06d-istio-podinfo" pod "1287989d-3ca3-48c0-bfa1-3e6e27b7f06d" (UID: "1287989d-3ca3-48c0-bfa1-3e6e27b7f06d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 04:25:41.188583 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:41.188558 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1287989d-3ca3-48c0-bfa1-3e6e27b7f06d-istio-envoy" pod "1287989d-3ca3-48c0-bfa1-3e6e27b7f06d" (UID: "1287989d-3ca3-48c0-bfa1-3e6e27b7f06d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:25:41.211014 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:41.210674 4376 scope.go:122] "RemoveContainer" containerID="2d6d4ae8e4be174ccecadedfe719fbf0a903a33f1f647bdd228ba3a74dcd8aa3" Oct 31 04:25:41.211709 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:25:41.211237 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d6d4ae8e4be174ccecadedfe719fbf0a903a33f1f647bdd228ba3a74dcd8aa3\": container with ID starting with 2d6d4ae8e4be174ccecadedfe719fbf0a903a33f1f647bdd228ba3a74dcd8aa3 not found: ID does not exist" containerID="2d6d4ae8e4be174ccecadedfe719fbf0a903a33f1f647bdd228ba3a74dcd8aa3" Oct 31 04:25:41.211709 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:41.211278 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d6d4ae8e4be174ccecadedfe719fbf0a903a33f1f647bdd228ba3a74dcd8aa3"} err="failed to get container status \"2d6d4ae8e4be174ccecadedfe719fbf0a903a33f1f647bdd228ba3a74dcd8aa3\": rpc error: code = NotFound desc = could not find container \"2d6d4ae8e4be174ccecadedfe719fbf0a903a33f1f647bdd228ba3a74dcd8aa3\": container with ID starting with 2d6d4ae8e4be174ccecadedfe719fbf0a903a33f1f647bdd228ba3a74dcd8aa3 not found: ID does not exist" Oct 31 04:25:41.278631 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:41.278508 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b871815-e7d5-42e1-a7b7-4777c82df055-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b871815-e7d5-42e1-a7b7-4777c82df055-istiod-ca-cert\") pod \"4b871815-e7d5-42e1-a7b7-4777c82df055\" (UID: \"4b871815-e7d5-42e1-a7b7-4777c82df055\") " Oct 31 04:25:41.278805 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:41.278650 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b871815-e7d5-42e1-a7b7-4777c82df055-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b871815-e7d5-42e1-a7b7-4777c82df055-workload-socket\") pod \"4b871815-e7d5-42e1-a7b7-4777c82df055\" (UID: \"4b871815-e7d5-42e1-a7b7-4777c82df055\") " Oct 31 04:25:41.278952 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:41.278923 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b871815-e7d5-42e1-a7b7-4777c82df055-istiod-ca-cert" pod "4b871815-e7d5-42e1-a7b7-4777c82df055" (UID: "4b871815-e7d5-42e1-a7b7-4777c82df055"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:25:41.279031 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:41.278943 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b871815-e7d5-42e1-a7b7-4777c82df055-workload-socket" pod "4b871815-e7d5-42e1-a7b7-4777c82df055" (UID: "4b871815-e7d5-42e1-a7b7-4777c82df055"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:25:41.279112 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:41.279092 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b871815-e7d5-42e1-a7b7-4777c82df055-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b871815-e7d5-42e1-a7b7-4777c82df055-istio-data\") pod \"4b871815-e7d5-42e1-a7b7-4777c82df055\" (UID: \"4b871815-e7d5-42e1-a7b7-4777c82df055\") " Oct 31 04:25:41.279362 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:41.279318 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b871815-e7d5-42e1-a7b7-4777c82df055-istio-data" pod "4b871815-e7d5-42e1-a7b7-4777c82df055" (UID: "4b871815-e7d5-42e1-a7b7-4777c82df055"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:25:41.279430 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:41.279403 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b871815-e7d5-42e1-a7b7-4777c82df055-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b871815-e7d5-42e1-a7b7-4777c82df055-credential-socket\") pod \"4b871815-e7d5-42e1-a7b7-4777c82df055\" (UID: \"4b871815-e7d5-42e1-a7b7-4777c82df055\") " Oct 31 04:25:41.279502 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:41.279450 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b871815-e7d5-42e1-a7b7-4777c82df055-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b871815-e7d5-42e1-a7b7-4777c82df055-istio-envoy\") pod \"4b871815-e7d5-42e1-a7b7-4777c82df055\" (UID: \"4b871815-e7d5-42e1-a7b7-4777c82df055\") " Oct 31 04:25:41.279560 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:41.279521 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4b871815-e7d5-42e1-a7b7-4777c82df055-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b871815-e7d5-42e1-a7b7-4777c82df055-istio-podinfo\") pod \"4b871815-e7d5-42e1-a7b7-4777c82df055\" (UID: \"4b871815-e7d5-42e1-a7b7-4777c82df055\") " Oct 31 04:25:41.279613 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:41.279602 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b871815-e7d5-42e1-a7b7-4777c82df055-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b871815-e7d5-42e1-a7b7-4777c82df055-workload-certs\") pod \"4b871815-e7d5-42e1-a7b7-4777c82df055\" (UID: \"4b871815-e7d5-42e1-a7b7-4777c82df055\") " Oct 31 04:25:41.279665 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:41.279647 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b871815-e7d5-42e1-a7b7-4777c82df055-istio-token\" (UniqueName: \"kubernetes.io/projected/4b871815-e7d5-42e1-a7b7-4777c82df055-istio-token\") pod \"4b871815-e7d5-42e1-a7b7-4777c82df055\" (UID: \"4b871815-e7d5-42e1-a7b7-4777c82df055\") " Oct 31 04:25:41.279730 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:41.279714 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b871815-e7d5-42e1-a7b7-4777c82df055-kube-api-access-k976b\" (UniqueName: \"kubernetes.io/projected/4b871815-e7d5-42e1-a7b7-4777c82df055-kube-api-access-k976b\") pod \"4b871815-e7d5-42e1-a7b7-4777c82df055\" (UID: \"4b871815-e7d5-42e1-a7b7-4777c82df055\") " Oct 31 04:25:41.280799 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:41.280385 4376 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b871815-e7d5-42e1-a7b7-4777c82df055-istiod-ca-cert\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:25:41.280799 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:41.280412 4376 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1287989d-3ca3-48c0-bfa1-3e6e27b7f06d-istiod-ca-cert\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:25:41.280799 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:41.280425 4376 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1287989d-3ca3-48c0-bfa1-3e6e27b7f06d-istio-podinfo\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:25:41.280799 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:41.280438 4376 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b871815-e7d5-42e1-a7b7-4777c82df055-workload-socket\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:25:41.280799 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:41.280452 4376 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b871815-e7d5-42e1-a7b7-4777c82df055-istio-data\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:25:41.280799 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:41.280465 4376 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1287989d-3ca3-48c0-bfa1-3e6e27b7f06d-credential-socket\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:25:41.280799 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:41.280472 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b871815-e7d5-42e1-a7b7-4777c82df055-workload-certs" pod "4b871815-e7d5-42e1-a7b7-4777c82df055" (UID: "4b871815-e7d5-42e1-a7b7-4777c82df055"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:25:41.280799 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:41.280479 4376 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1287989d-3ca3-48c0-bfa1-3e6e27b7f06d-workload-certs\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:25:41.280799 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:41.280516 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zg5gl\" (UniqueName: \"kubernetes.io/projected/1287989d-3ca3-48c0-bfa1-3e6e27b7f06d-kube-api-access-zg5gl\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:25:41.280799 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:41.280534 4376 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1287989d-3ca3-48c0-bfa1-3e6e27b7f06d-istio-data\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:25:41.280799 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:41.280548 4376 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1287989d-3ca3-48c0-bfa1-3e6e27b7f06d-istio-token\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:25:41.280799 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:41.280561 4376 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1287989d-3ca3-48c0-bfa1-3e6e27b7f06d-istio-envoy\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:25:41.281546 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:41.281037 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b871815-e7d5-42e1-a7b7-4777c82df055-credential-socket" pod "4b871815-e7d5-42e1-a7b7-4777c82df055" (UID: "4b871815-e7d5-42e1-a7b7-4777c82df055"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:25:41.282394 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:41.282359 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4b871815-e7d5-42e1-a7b7-4777c82df055-istio-podinfo" pod "4b871815-e7d5-42e1-a7b7-4777c82df055" (UID: "4b871815-e7d5-42e1-a7b7-4777c82df055"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 04:25:41.282749 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:41.282715 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b871815-e7d5-42e1-a7b7-4777c82df055-kube-api-access-k976b" pod "4b871815-e7d5-42e1-a7b7-4777c82df055" (UID: "4b871815-e7d5-42e1-a7b7-4777c82df055"). InnerVolumeSpecName "kube-api-access-k976b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:25:41.283381 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:41.283355 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b871815-e7d5-42e1-a7b7-4777c82df055-istio-envoy" pod "4b871815-e7d5-42e1-a7b7-4777c82df055" (UID: "4b871815-e7d5-42e1-a7b7-4777c82df055"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:25:41.283810 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:41.283789 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b871815-e7d5-42e1-a7b7-4777c82df055-istio-token" pod "4b871815-e7d5-42e1-a7b7-4777c82df055" (UID: "4b871815-e7d5-42e1-a7b7-4777c82df055"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:25:41.381469 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:41.381429 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k976b\" (UniqueName: \"kubernetes.io/projected/4b871815-e7d5-42e1-a7b7-4777c82df055-kube-api-access-k976b\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:25:41.381469 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:41.381461 4376 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b871815-e7d5-42e1-a7b7-4777c82df055-credential-socket\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:25:41.381469 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:41.381472 4376 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b871815-e7d5-42e1-a7b7-4777c82df055-istio-envoy\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:25:41.381469 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:41.381481 4376 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b871815-e7d5-42e1-a7b7-4777c82df055-istio-podinfo\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:25:41.381749 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:41.381489 4376 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b871815-e7d5-42e1-a7b7-4777c82df055-workload-certs\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:25:41.381749 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:41.381497 4376 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b871815-e7d5-42e1-a7b7-4777c82df055-istio-token\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:25:41.535500 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:41.535428 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-bzbxb"] Oct 31 04:25:41.557817 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:41.554037 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-bzbxb"] Oct 31 04:25:42.192986 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:42.192951 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-cqvwg" Oct 31 04:25:42.271922 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:42.271889 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-cqvwg"] Oct 31 04:25:42.284208 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:42.284176 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-cqvwg"] Oct 31 04:25:42.469457 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:42.469388 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1287989d-3ca3-48c0-bfa1-3e6e27b7f06d" path="/var/lib/kubelet/pods/1287989d-3ca3-48c0-bfa1-3e6e27b7f06d/volumes" Oct 31 04:25:42.469888 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:42.469874 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b871815-e7d5-42e1-a7b7-4777c82df055" path="/var/lib/kubelet/pods/4b871815-e7d5-42e1-a7b7-4777c82df055/volumes" Oct 31 04:25:49.274984 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:49.274951 4376 generic.go:356] "Generic (PLEG): container finished" podID="1a7c4b33-999a-449b-bea8-9c55c838df0a" containerID="d97124d31fee309b2b61051e27f2842e7c617becb5efaaa5d5b2133566921f90" exitCode=0 Oct 31 04:25:49.275437 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:49.275026 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-9154a6-openshift-default-7f74d4f9bc-v6fc7" event={"ID":"1a7c4b33-999a-449b-bea8-9c55c838df0a","Type":"ContainerDied","Data":"d97124d31fee309b2b61051e27f2842e7c617becb5efaaa5d5b2133566921f90"} Oct 31 04:25:49.275532 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:49.275515 4376 scope.go:122] "RemoveContainer" containerID="d97124d31fee309b2b61051e27f2842e7c617becb5efaaa5d5b2133566921f90" Oct 31 04:25:49.277144 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:49.277111 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:25:49.277253 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:49.277183 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:25:49.277253 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:49.277216 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:25:50.093867 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:50.093832 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-801218-5c988b64cf-w2d4l"] Oct 31 04:25:50.136231 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:50.136200 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-801218-7bd7574847-2fgt5"] Oct 31 04:25:50.293591 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:50.293559 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-9154a6-openshift-default-7f74d4f9bc-v6fc7" event={"ID":"1a7c4b33-999a-449b-bea8-9c55c838df0a","Type":"ContainerStarted","Data":"bc57377f8ebefa898aab1950c5d8528ef2d3a92f98fd7a8b356990c7c6635a2f"} Oct 31 04:25:50.601296 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:50.601261 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-9154a6-openshift-default-7f74d4f9bc-v6fc7" Oct 31 04:25:50.602372 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:50.602310 4376 patch_prober.go:29] interesting pod/e2e-api-b-9154a6-openshift-default-7f74d4f9bc-v6fc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body= Oct 31 04:25:50.602531 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:50.602416 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-9154a6-openshift-default-7f74d4f9bc-v6fc7" podUID="1a7c4b33-999a-449b-bea8-9c55c838df0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" Oct 31 04:25:50.698009 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:50.697971 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-9154a6-openshift-default-7f74d4f9bc-v6fc7" Oct 31 04:25:51.601884 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:51.601851 4376 patch_prober.go:29] interesting pod/e2e-api-b-9154a6-openshift-default-7f74d4f9bc-v6fc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body= Oct 31 04:25:51.602278 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:51.601899 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-9154a6-openshift-default-7f74d4f9bc-v6fc7" podUID="1a7c4b33-999a-449b-bea8-9c55c838df0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" Oct 31 04:25:52.611354 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:52.608952 4376 patch_prober.go:29] interesting pod/e2e-api-b-9154a6-openshift-default-7f74d4f9bc-v6fc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body= Oct 31 04:25:52.611354 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:52.609012 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-9154a6-openshift-default-7f74d4f9bc-v6fc7" podUID="1a7c4b33-999a-449b-bea8-9c55c838df0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" Oct 31 04:25:52.624405 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:52.619382 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7b4185-7c659d956c-jllqw"] Oct 31 04:25:52.624405 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:52.620284 4376 cpu_manager.go:414] "containerMap: removing container" podUID="419996ea-d44d-44f4-b414-583068843613" containerName="payload-pre-processing" Oct 31 04:25:52.624405 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:52.620302 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="419996ea-d44d-44f4-b414-583068843613" containerName="payload-pre-processing" Oct 31 04:25:52.624405 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:52.620340 4376 cpu_manager.go:414] "containerMap: removing container" podUID="1287989d-3ca3-48c0-bfa1-3e6e27b7f06d" containerName="istio-proxy" Oct 31 04:25:52.624405 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:52.620349 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1287989d-3ca3-48c0-bfa1-3e6e27b7f06d" containerName="istio-proxy" Oct 31 04:25:52.624405 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:52.620372 4376 cpu_manager.go:414] "containerMap: removing container" podUID="91eecd3e-fc45-4cf9-8515-a0f723aa9968" containerName="istio-proxy" Oct 31 04:25:52.624405 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:52.620381 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91eecd3e-fc45-4cf9-8515-a0f723aa9968" containerName="istio-proxy" Oct 31 04:25:52.624405 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:52.620419 4376 cpu_manager.go:414] "containerMap: removing container" podUID="982c902b-4689-4eb3-93fb-8930a3136465" containerName="istio-proxy" Oct 31 04:25:52.624405 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:52.620426 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="982c902b-4689-4eb3-93fb-8930a3136465" containerName="istio-proxy" Oct 31 04:25:52.624405 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:52.620449 4376 cpu_manager.go:414] "containerMap: removing container" podUID="83a9eaed-ab99-4fd4-9ffe-a1d863a85cf8" containerName="payload-processing" Oct 31 04:25:52.624405 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:52.620456 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83a9eaed-ab99-4fd4-9ffe-a1d863a85cf8" containerName="payload-processing" Oct 31 04:25:52.624405 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:52.620488 4376 cpu_manager.go:414] "containerMap: removing container" podUID="4b871815-e7d5-42e1-a7b7-4777c82df055" containerName="istio-proxy" Oct 31 04:25:52.624405 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:52.620494 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b871815-e7d5-42e1-a7b7-4777c82df055" containerName="istio-proxy" Oct 31 04:25:52.624405 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:52.620630 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="83a9eaed-ab99-4fd4-9ffe-a1d863a85cf8" containerName="payload-processing" Oct 31 04:25:52.624405 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:52.620645 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b871815-e7d5-42e1-a7b7-4777c82df055" containerName="istio-proxy" Oct 31 04:25:52.624405 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:52.620655 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="419996ea-d44d-44f4-b414-583068843613" containerName="payload-pre-processing" Oct 31 04:25:52.624405 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:52.620666 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="1287989d-3ca3-48c0-bfa1-3e6e27b7f06d" containerName="istio-proxy" Oct 31 04:25:52.624405 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:52.620676 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="91eecd3e-fc45-4cf9-8515-a0f723aa9968" containerName="istio-proxy" Oct 31 04:25:52.624405 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:52.620690 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="982c902b-4689-4eb3-93fb-8930a3136465" containerName="istio-proxy" Oct 31 04:25:52.666349 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:52.661857 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7b4185-7c659d956c-jllqw"] Oct 31 04:25:52.666349 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:52.662020 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7b4185-7c659d956c-jllqw" Oct 31 04:25:52.705697 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:52.705582 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-7b4185-dockercfg-ttn68\"" Oct 31 04:25:52.705697 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:52.705665 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-7b4185\"" Oct 31 04:25:52.710239 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:52.709709 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgjgx\" (UniqueName: \"kubernetes.io/projected/0d8c3260-faea-47ce-aebf-3bad9dc8fcf8-kube-api-access-kgjgx\") pod \"payload-pre-processing-e2e-ipp-b-7b4185-7c659d956c-jllqw\" (UID: \"0d8c3260-faea-47ce-aebf-3bad9dc8fcf8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7b4185-7c659d956c-jllqw" Oct 31 04:25:52.710239 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:52.709969 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d8c3260-faea-47ce-aebf-3bad9dc8fcf8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-7b4185-7c659d956c-jllqw\" (UID: \"0d8c3260-faea-47ce-aebf-3bad9dc8fcf8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7b4185-7c659d956c-jllqw" Oct 31 04:25:52.812036 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:52.811989 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d8c3260-faea-47ce-aebf-3bad9dc8fcf8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-7b4185-7c659d956c-jllqw\" (UID: \"0d8c3260-faea-47ce-aebf-3bad9dc8fcf8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7b4185-7c659d956c-jllqw" Oct 31 04:25:52.812227 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:52.812196 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgjgx\" (UniqueName: \"kubernetes.io/projected/0d8c3260-faea-47ce-aebf-3bad9dc8fcf8-kube-api-access-kgjgx\") pod \"payload-pre-processing-e2e-ipp-b-7b4185-7c659d956c-jllqw\" (UID: \"0d8c3260-faea-47ce-aebf-3bad9dc8fcf8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7b4185-7c659d956c-jllqw" Oct 31 04:25:52.813531 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:52.813498 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d8c3260-faea-47ce-aebf-3bad9dc8fcf8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-7b4185-7c659d956c-jllqw\" (UID: \"0d8c3260-faea-47ce-aebf-3bad9dc8fcf8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7b4185-7c659d956c-jllqw" Oct 31 04:25:52.835913 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:52.835891 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgjgx\" (UniqueName: \"kubernetes.io/projected/0d8c3260-faea-47ce-aebf-3bad9dc8fcf8-kube-api-access-kgjgx\") pod \"payload-pre-processing-e2e-ipp-b-7b4185-7c659d956c-jllqw\" (UID: \"0d8c3260-faea-47ce-aebf-3bad9dc8fcf8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7b4185-7c659d956c-jllqw" Oct 31 04:25:52.983735 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:52.983700 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7b4185-cfcb87d66-l8fzb"] Oct 31 04:25:52.988228 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:52.988182 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-7b4185-cfcb87d66-l8fzb" Oct 31 04:25:53.006121 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:53.006097 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7b4185-cfcb87d66-l8fzb"] Oct 31 04:25:53.111311 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:53.111273 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7b4185-7c659d956c-jllqw" Oct 31 04:25:53.116414 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:53.116385 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b26cw\" (UniqueName: \"kubernetes.io/projected/0f6ed41c-02f0-44a1-9055-0a95c3321143-kube-api-access-b26cw\") pod \"payload-processing-e2e-ipp-b-7b4185-cfcb87d66-l8fzb\" (UID: \"0f6ed41c-02f0-44a1-9055-0a95c3321143\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7b4185-cfcb87d66-l8fzb" Oct 31 04:25:53.116695 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:53.116673 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f6ed41c-02f0-44a1-9055-0a95c3321143-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-7b4185-cfcb87d66-l8fzb\" (UID: \"0f6ed41c-02f0-44a1-9055-0a95c3321143\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7b4185-cfcb87d66-l8fzb" Oct 31 04:25:53.218867 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:53.218796 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f6ed41c-02f0-44a1-9055-0a95c3321143-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-7b4185-cfcb87d66-l8fzb\" (UID: \"0f6ed41c-02f0-44a1-9055-0a95c3321143\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7b4185-cfcb87d66-l8fzb" Oct 31 04:25:53.219003 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:53.218903 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b26cw\" (UniqueName: \"kubernetes.io/projected/0f6ed41c-02f0-44a1-9055-0a95c3321143-kube-api-access-b26cw\") pod \"payload-processing-e2e-ipp-b-7b4185-cfcb87d66-l8fzb\" (UID: \"0f6ed41c-02f0-44a1-9055-0a95c3321143\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7b4185-cfcb87d66-l8fzb" Oct 31 04:25:53.225071 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:53.220475 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f6ed41c-02f0-44a1-9055-0a95c3321143-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-7b4185-cfcb87d66-l8fzb\" (UID: \"0f6ed41c-02f0-44a1-9055-0a95c3321143\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7b4185-cfcb87d66-l8fzb" Oct 31 04:25:53.266946 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:53.266863 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b26cw\" (UniqueName: \"kubernetes.io/projected/0f6ed41c-02f0-44a1-9055-0a95c3321143-kube-api-access-b26cw\") pod \"payload-processing-e2e-ipp-b-7b4185-cfcb87d66-l8fzb\" (UID: \"0f6ed41c-02f0-44a1-9055-0a95c3321143\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7b4185-cfcb87d66-l8fzb" Oct 31 04:25:53.305449 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:53.305418 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-7b4185-cfcb87d66-l8fzb" Oct 31 04:25:53.482573 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:53.482494 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7b4185-7c659d956c-jllqw"] Oct 31 04:25:53.491684 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:25:53.491636 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d8c3260_faea_47ce_aebf_3bad9dc8fcf8.slice/crio-a34cf8b2a4bb82ac5e540af60a07214f9d814e87ab1054e438903c8df8c4226f WatchSource:0}: Error finding container a34cf8b2a4bb82ac5e540af60a07214f9d814e87ab1054e438903c8df8c4226f: Status 404 returned error can't find the container with id a34cf8b2a4bb82ac5e540af60a07214f9d814e87ab1054e438903c8df8c4226f Oct 31 04:25:53.603457 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:53.601571 4376 patch_prober.go:29] interesting pod/e2e-api-b-9154a6-openshift-default-7f74d4f9bc-v6fc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body= Oct 31 04:25:53.603457 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:53.601629 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-9154a6-openshift-default-7f74d4f9bc-v6fc7" podUID="1a7c4b33-999a-449b-bea8-9c55c838df0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" Oct 31 04:25:53.676055 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:53.676027 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7b4185-cfcb87d66-l8fzb"] Oct 31 04:25:53.677373 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:25:53.677318 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f6ed41c_02f0_44a1_9055_0a95c3321143.slice/crio-c04bb90d07fd0994ae74fff5d2265cbbf8bbcbfcfde9be6fbbad63640bb490dc WatchSource:0}: Error finding container c04bb90d07fd0994ae74fff5d2265cbbf8bbcbfcfde9be6fbbad63640bb490dc: Status 404 returned error can't find the container with id c04bb90d07fd0994ae74fff5d2265cbbf8bbcbfcfde9be6fbbad63640bb490dc Oct 31 04:25:54.378727 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:54.378692 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7b4185-7c659d956c-jllqw" event={"ID":"0d8c3260-faea-47ce-aebf-3bad9dc8fcf8","Type":"ContainerStarted","Data":"8fe3fbbe94798b3be7487758d7e213d08b1d4cb87ad2790df1b7bb95ef3725ad"} Oct 31 04:25:54.378948 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:54.378749 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7b4185-7c659d956c-jllqw" event={"ID":"0d8c3260-faea-47ce-aebf-3bad9dc8fcf8","Type":"ContainerStarted","Data":"a34cf8b2a4bb82ac5e540af60a07214f9d814e87ab1054e438903c8df8c4226f"} Oct 31 04:25:54.379096 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:54.379067 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7b4185-7c659d956c-jllqw" Oct 31 04:25:54.380913 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:54.380885 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-7b4185-cfcb87d66-l8fzb" event={"ID":"0f6ed41c-02f0-44a1-9055-0a95c3321143","Type":"ContainerStarted","Data":"3deeb73675a951250d8d81dabea0dd656da1396cd9987e20d2e4b6805c20afd5"} Oct 31 04:25:54.381049 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:54.380935 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-7b4185-cfcb87d66-l8fzb" event={"ID":"0f6ed41c-02f0-44a1-9055-0a95c3321143","Type":"ContainerStarted","Data":"c04bb90d07fd0994ae74fff5d2265cbbf8bbcbfcfde9be6fbbad63640bb490dc"} Oct 31 04:25:54.381049 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:54.380967 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-7b4185-cfcb87d66-l8fzb" Oct 31 04:25:54.422773 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:54.422724 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7b4185-7c659d956c-jllqw" podStartSLOduration=2.422711484 podStartE2EDuration="2.422711484s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:25:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:25:54.419794468 +0000 UTC m=+1302.134129084" watchObservedRunningTime="2026-10-31 04:25:54.422711484 +0000 UTC m=+1302.137046099" Oct 31 04:25:54.471208 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:54.471147 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-7b4185-cfcb87d66-l8fzb" podStartSLOduration=2.471129501 podStartE2EDuration="2.471129501s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:25:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:25:54.466554659 +0000 UTC m=+1302.180889275" watchObservedRunningTime="2026-10-31 04:25:54.471129501 +0000 UTC m=+1302.185464117" Oct 31 04:25:54.608356 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:54.606624 4376 patch_prober.go:29] interesting pod/e2e-api-b-9154a6-openshift-default-7f74d4f9bc-v6fc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body= Oct 31 04:25:54.608356 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:54.606683 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-9154a6-openshift-default-7f74d4f9bc-v6fc7" podUID="1a7c4b33-999a-449b-bea8-9c55c838df0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" Oct 31 04:25:55.094900 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:55.094850 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-801218-5c988b64cf-w2d4l" podUID="11f03146-3b0e-4cb0-b42c-0cb1d1952997" containerName="payload-processing" containerID="cri-o://f236204cd75a5a857b553c26d7df30ca7371a70b348c3fbb40483bb99ed98491" gracePeriod=25 Oct 31 04:25:55.159856 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:55.137250 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-801218-7bd7574847-2fgt5" podUID="862d48b9-2fae-4584-a58b-8dbd9d985857" containerName="payload-pre-processing" containerID="cri-o://871924ac16310975ea9771c3314f78ccc7012c627b3380ca8ed70ad8e0e06458" gracePeriod=25 Oct 31 04:25:55.489957 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:55.489696 4376 generic.go:356] "Generic (PLEG): container finished" podID="862d48b9-2fae-4584-a58b-8dbd9d985857" containerID="871924ac16310975ea9771c3314f78ccc7012c627b3380ca8ed70ad8e0e06458" exitCode=0 Oct 31 04:25:55.489957 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:55.489840 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-801218-7bd7574847-2fgt5" event={"ID":"862d48b9-2fae-4584-a58b-8dbd9d985857","Type":"ContainerDied","Data":"871924ac16310975ea9771c3314f78ccc7012c627b3380ca8ed70ad8e0e06458"} Oct 31 04:25:55.492949 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:55.492694 4376 generic.go:356] "Generic (PLEG): container finished" podID="11f03146-3b0e-4cb0-b42c-0cb1d1952997" containerID="f236204cd75a5a857b553c26d7df30ca7371a70b348c3fbb40483bb99ed98491" exitCode=0 Oct 31 04:25:55.494545 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:55.494445 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-801218-5c988b64cf-w2d4l" event={"ID":"11f03146-3b0e-4cb0-b42c-0cb1d1952997","Type":"ContainerDied","Data":"f236204cd75a5a857b553c26d7df30ca7371a70b348c3fbb40483bb99ed98491"} Oct 31 04:25:55.649386 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:55.625106 4376 patch_prober.go:29] interesting pod/e2e-api-b-9154a6-openshift-default-7f74d4f9bc-v6fc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body= Oct 31 04:25:55.649386 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:55.625166 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-9154a6-openshift-default-7f74d4f9bc-v6fc7" podUID="1a7c4b33-999a-449b-bea8-9c55c838df0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" Oct 31 04:25:55.992347 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:55.981278 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-801218-5c988b64cf-w2d4l" Oct 31 04:25:56.019587 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:56.015407 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11f03146-3b0e-4cb0-b42c-0cb1d1952997-kube-api-access-mdw4m\" (UniqueName: \"kubernetes.io/projected/11f03146-3b0e-4cb0-b42c-0cb1d1952997-kube-api-access-mdw4m\") pod \"11f03146-3b0e-4cb0-b42c-0cb1d1952997\" (UID: \"11f03146-3b0e-4cb0-b42c-0cb1d1952997\") " Oct 31 04:25:56.019587 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:56.015542 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11f03146-3b0e-4cb0-b42c-0cb1d1952997-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11f03146-3b0e-4cb0-b42c-0cb1d1952997-plugins-config-volume\") pod \"11f03146-3b0e-4cb0-b42c-0cb1d1952997\" (UID: \"11f03146-3b0e-4cb0-b42c-0cb1d1952997\") " Oct 31 04:25:56.019872 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:56.019604 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11f03146-3b0e-4cb0-b42c-0cb1d1952997-plugins-config-volume" pod "11f03146-3b0e-4cb0-b42c-0cb1d1952997" (UID: "11f03146-3b0e-4cb0-b42c-0cb1d1952997"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:25:56.023357 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:56.021612 4376 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11f03146-3b0e-4cb0-b42c-0cb1d1952997-plugins-config-volume\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:25:56.025237 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:56.025004 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11f03146-3b0e-4cb0-b42c-0cb1d1952997-kube-api-access-mdw4m" pod "11f03146-3b0e-4cb0-b42c-0cb1d1952997" (UID: "11f03146-3b0e-4cb0-b42c-0cb1d1952997"). InnerVolumeSpecName "kube-api-access-mdw4m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:25:56.035955 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:56.035891 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-801218-7bd7574847-2fgt5" Oct 31 04:25:56.135857 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:56.135821 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/862d48b9-2fae-4584-a58b-8dbd9d985857-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/862d48b9-2fae-4584-a58b-8dbd9d985857-plugins-config-volume\") pod \"862d48b9-2fae-4584-a58b-8dbd9d985857\" (UID: \"862d48b9-2fae-4584-a58b-8dbd9d985857\") " Oct 31 04:25:56.136285 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:56.136113 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/862d48b9-2fae-4584-a58b-8dbd9d985857-kube-api-access-xfj48\" (UniqueName: \"kubernetes.io/projected/862d48b9-2fae-4584-a58b-8dbd9d985857-kube-api-access-xfj48\") pod \"862d48b9-2fae-4584-a58b-8dbd9d985857\" (UID: \"862d48b9-2fae-4584-a58b-8dbd9d985857\") " Oct 31 04:25:56.136447 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:56.136422 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/862d48b9-2fae-4584-a58b-8dbd9d985857-plugins-config-volume" pod "862d48b9-2fae-4584-a58b-8dbd9d985857" (UID: "862d48b9-2fae-4584-a58b-8dbd9d985857"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:25:56.136815 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:56.136791 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdw4m\" (UniqueName: \"kubernetes.io/projected/11f03146-3b0e-4cb0-b42c-0cb1d1952997-kube-api-access-mdw4m\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:25:56.136920 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:56.136818 4376 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/862d48b9-2fae-4584-a58b-8dbd9d985857-plugins-config-volume\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:25:56.144988 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:56.143043 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/862d48b9-2fae-4584-a58b-8dbd9d985857-kube-api-access-xfj48" pod "862d48b9-2fae-4584-a58b-8dbd9d985857" (UID: "862d48b9-2fae-4584-a58b-8dbd9d985857"). InnerVolumeSpecName "kube-api-access-xfj48". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:25:56.269930 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:56.257877 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xfj48\" (UniqueName: \"kubernetes.io/projected/862d48b9-2fae-4584-a58b-8dbd9d985857-kube-api-access-xfj48\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:25:56.523365 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:56.522631 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-801218-7bd7574847-2fgt5" event={"ID":"862d48b9-2fae-4584-a58b-8dbd9d985857","Type":"ContainerDied","Data":"85d85d81caa126370ada75269fe800fe88d1d738d5cfa45202207c5fca5a7b41"} Oct 31 04:25:56.523365 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:56.522710 4376 scope.go:122] "RemoveContainer" containerID="871924ac16310975ea9771c3314f78ccc7012c627b3380ca8ed70ad8e0e06458" Oct 31 04:25:56.523365 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:56.522876 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-801218-7bd7574847-2fgt5" Oct 31 04:25:56.533317 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:56.532418 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-801218-5c988b64cf-w2d4l" event={"ID":"11f03146-3b0e-4cb0-b42c-0cb1d1952997","Type":"ContainerDied","Data":"aabb405cf91f5b79356f5cd081a3ccdb4d91b52cd370984b9a755fd03de034d3"} Oct 31 04:25:56.533317 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:56.532553 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-801218-5c988b64cf-w2d4l" Oct 31 04:25:56.568224 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:56.568123 4376 scope.go:122] "RemoveContainer" containerID="f236204cd75a5a857b553c26d7df30ca7371a70b348c3fbb40483bb99ed98491" Oct 31 04:25:56.642461 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:56.642402 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-801218-7bd7574847-2fgt5"] Oct 31 04:25:56.688485 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:56.688443 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-801218-7bd7574847-2fgt5"] Oct 31 04:25:56.692063 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:56.692002 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-801218-5c988b64cf-w2d4l"] Oct 31 04:25:56.698820 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:56.698687 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-801218-5c988b64cf-w2d4l"] Oct 31 04:25:56.811084 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:56.811002 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-9154a6-openshift-default-7f74d4f9bc-v6fc7" Oct 31 04:25:56.812393 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:56.812369 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-9154a6-openshift-default-7f74d4f9bc-v6fc7" Oct 31 04:25:58.444964 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:58.444934 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11f03146-3b0e-4cb0-b42c-0cb1d1952997" path="/var/lib/kubelet/pods/11f03146-3b0e-4cb0-b42c-0cb1d1952997/volumes" Oct 31 04:25:58.445401 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:58.445386 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="862d48b9-2fae-4584-a58b-8dbd9d985857" path="/var/lib/kubelet/pods/862d48b9-2fae-4584-a58b-8dbd9d985857/volumes" Oct 31 04:25:59.182289 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:59.182259 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-801218-openshift-default-58b4b8685c-8gpjq"] Oct 31 04:25:59.182608 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:59.182581 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-801218-openshift-default-58b4b8685c-8gpjq" podUID="e3f3e12d-e06e-4b4e-a23a-784e49d5cbaf" containerName="istio-proxy" containerID="cri-o://05fb4b4a647bbba5071604f9fa08cb58fe5b76ede194d21403f43d1d9745942f" gracePeriod=30 Oct 31 04:25:59.823028 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:59.822995 4376 patch_prober.go:29] interesting pod/e2e-api-a-801218-openshift-default-58b4b8685c-8gpjq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 04:25:59.823502 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:25:59.823047 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-801218-openshift-default-58b4b8685c-8gpjq" podUID="e3f3e12d-e06e-4b4e-a23a-784e49d5cbaf" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 04:26:04.473533 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:04.473469 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-801218-openshift-default-58b4b8685c-8gpjq" Oct 31 04:26:04.590027 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:04.589997 4376 generic.go:356] "Generic (PLEG): container finished" podID="e3f3e12d-e06e-4b4e-a23a-784e49d5cbaf" containerID="05fb4b4a647bbba5071604f9fa08cb58fe5b76ede194d21403f43d1d9745942f" exitCode=0 Oct 31 04:26:04.590187 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:04.590044 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-801218-openshift-default-58b4b8685c-8gpjq" Oct 31 04:26:04.590187 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:04.590078 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-801218-openshift-default-58b4b8685c-8gpjq" event={"ID":"e3f3e12d-e06e-4b4e-a23a-784e49d5cbaf","Type":"ContainerDied","Data":"05fb4b4a647bbba5071604f9fa08cb58fe5b76ede194d21403f43d1d9745942f"} Oct 31 04:26:04.590187 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:04.590135 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-801218-openshift-default-58b4b8685c-8gpjq" event={"ID":"e3f3e12d-e06e-4b4e-a23a-784e49d5cbaf","Type":"ContainerDied","Data":"8e349ce805ed9f8488d1261d495f1025b0c0afbe260b0d5933641c07fea5ce86"} Oct 31 04:26:04.590187 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:04.590157 4376 scope.go:122] "RemoveContainer" containerID="05fb4b4a647bbba5071604f9fa08cb58fe5b76ede194d21403f43d1d9745942f" Oct 31 04:26:04.597637 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:04.597609 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3f3e12d-e06e-4b4e-a23a-784e49d5cbaf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3f3e12d-e06e-4b4e-a23a-784e49d5cbaf-istio-envoy\") pod \"e3f3e12d-e06e-4b4e-a23a-784e49d5cbaf\" (UID: \"e3f3e12d-e06e-4b4e-a23a-784e49d5cbaf\") " Oct 31 04:26:04.597754 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:04.597696 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3f3e12d-e06e-4b4e-a23a-784e49d5cbaf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3f3e12d-e06e-4b4e-a23a-784e49d5cbaf-istiod-ca-cert\") pod \"e3f3e12d-e06e-4b4e-a23a-784e49d5cbaf\" (UID: \"e3f3e12d-e06e-4b4e-a23a-784e49d5cbaf\") " Oct 31 04:26:04.597754 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:04.597740 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3f3e12d-e06e-4b4e-a23a-784e49d5cbaf-kube-api-access-vm88k\" (UniqueName: \"kubernetes.io/projected/e3f3e12d-e06e-4b4e-a23a-784e49d5cbaf-kube-api-access-vm88k\") pod \"e3f3e12d-e06e-4b4e-a23a-784e49d5cbaf\" (UID: \"e3f3e12d-e06e-4b4e-a23a-784e49d5cbaf\") " Oct 31 04:26:04.597864 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:04.597786 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3f3e12d-e06e-4b4e-a23a-784e49d5cbaf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3f3e12d-e06e-4b4e-a23a-784e49d5cbaf-workload-socket\") pod \"e3f3e12d-e06e-4b4e-a23a-784e49d5cbaf\" (UID: \"e3f3e12d-e06e-4b4e-a23a-784e49d5cbaf\") " Oct 31 04:26:04.597864 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:04.597832 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3f3e12d-e06e-4b4e-a23a-784e49d5cbaf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3f3e12d-e06e-4b4e-a23a-784e49d5cbaf-istio-data\") pod \"e3f3e12d-e06e-4b4e-a23a-784e49d5cbaf\" (UID: \"e3f3e12d-e06e-4b4e-a23a-784e49d5cbaf\") " Oct 31 04:26:04.597987 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:04.597877 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3f3e12d-e06e-4b4e-a23a-784e49d5cbaf-istio-token\" (UniqueName: \"kubernetes.io/projected/e3f3e12d-e06e-4b4e-a23a-784e49d5cbaf-istio-token\") pod \"e3f3e12d-e06e-4b4e-a23a-784e49d5cbaf\" (UID: \"e3f3e12d-e06e-4b4e-a23a-784e49d5cbaf\") " Oct 31 04:26:04.597987 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:04.597976 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3f3e12d-e06e-4b4e-a23a-784e49d5cbaf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3f3e12d-e06e-4b4e-a23a-784e49d5cbaf-workload-certs\") pod \"e3f3e12d-e06e-4b4e-a23a-784e49d5cbaf\" (UID: \"e3f3e12d-e06e-4b4e-a23a-784e49d5cbaf\") " Oct 31 04:26:04.598089 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:04.598037 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3f3e12d-e06e-4b4e-a23a-784e49d5cbaf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3f3e12d-e06e-4b4e-a23a-784e49d5cbaf-credential-socket\") pod \"e3f3e12d-e06e-4b4e-a23a-784e49d5cbaf\" (UID: \"e3f3e12d-e06e-4b4e-a23a-784e49d5cbaf\") " Oct 31 04:26:04.598089 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:04.598082 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e3f3e12d-e06e-4b4e-a23a-784e49d5cbaf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3f3e12d-e06e-4b4e-a23a-784e49d5cbaf-istio-podinfo\") pod \"e3f3e12d-e06e-4b4e-a23a-784e49d5cbaf\" (UID: \"e3f3e12d-e06e-4b4e-a23a-784e49d5cbaf\") " Oct 31 04:26:04.598198 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:04.598164 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3f3e12d-e06e-4b4e-a23a-784e49d5cbaf-istiod-ca-cert" pod "e3f3e12d-e06e-4b4e-a23a-784e49d5cbaf" (UID: "e3f3e12d-e06e-4b4e-a23a-784e49d5cbaf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:26:04.598266 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:04.598243 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3f3e12d-e06e-4b4e-a23a-784e49d5cbaf-workload-socket" pod "e3f3e12d-e06e-4b4e-a23a-784e49d5cbaf" (UID: "e3f3e12d-e06e-4b4e-a23a-784e49d5cbaf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:26:04.598461 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:04.598435 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3f3e12d-e06e-4b4e-a23a-784e49d5cbaf-workload-certs" pod "e3f3e12d-e06e-4b4e-a23a-784e49d5cbaf" (UID: "e3f3e12d-e06e-4b4e-a23a-784e49d5cbaf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:26:04.598461 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:04.598453 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3f3e12d-e06e-4b4e-a23a-784e49d5cbaf-credential-socket" pod "e3f3e12d-e06e-4b4e-a23a-784e49d5cbaf" (UID: "e3f3e12d-e06e-4b4e-a23a-784e49d5cbaf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:26:04.598705 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:04.598691 4376 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3f3e12d-e06e-4b4e-a23a-784e49d5cbaf-workload-socket\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:26:04.598705 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:04.598709 4376 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3f3e12d-e06e-4b4e-a23a-784e49d5cbaf-workload-certs\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:26:04.598818 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:04.598718 4376 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3f3e12d-e06e-4b4e-a23a-784e49d5cbaf-credential-socket\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:26:04.598818 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:04.598727 4376 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3f3e12d-e06e-4b4e-a23a-784e49d5cbaf-istiod-ca-cert\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:26:04.598918 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:04.598834 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3f3e12d-e06e-4b4e-a23a-784e49d5cbaf-istio-data" pod "e3f3e12d-e06e-4b4e-a23a-784e49d5cbaf" (UID: "e3f3e12d-e06e-4b4e-a23a-784e49d5cbaf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:26:04.600159 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:04.600088 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3f3e12d-e06e-4b4e-a23a-784e49d5cbaf-istio-envoy" pod "e3f3e12d-e06e-4b4e-a23a-784e49d5cbaf" (UID: "e3f3e12d-e06e-4b4e-a23a-784e49d5cbaf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:26:04.600222 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:04.600205 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3f3e12d-e06e-4b4e-a23a-784e49d5cbaf-kube-api-access-vm88k" pod "e3f3e12d-e06e-4b4e-a23a-784e49d5cbaf" (UID: "e3f3e12d-e06e-4b4e-a23a-784e49d5cbaf"). InnerVolumeSpecName "kube-api-access-vm88k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:26:04.600269 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:04.600206 4376 scope.go:122] "RemoveContainer" containerID="f9cbfef45650c56699306fde4739e979d5b6d74a078474c21b5a6d2d92557af2" Oct 31 04:26:04.600482 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:04.600410 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e3f3e12d-e06e-4b4e-a23a-784e49d5cbaf-istio-podinfo" pod "e3f3e12d-e06e-4b4e-a23a-784e49d5cbaf" (UID: "e3f3e12d-e06e-4b4e-a23a-784e49d5cbaf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 04:26:04.600569 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:04.600543 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3f3e12d-e06e-4b4e-a23a-784e49d5cbaf-istio-token" pod "e3f3e12d-e06e-4b4e-a23a-784e49d5cbaf" (UID: "e3f3e12d-e06e-4b4e-a23a-784e49d5cbaf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:26:04.627068 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:04.627042 4376 scope.go:122] "RemoveContainer" containerID="05fb4b4a647bbba5071604f9fa08cb58fe5b76ede194d21403f43d1d9745942f" Oct 31 04:26:04.627323 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:26:04.627301 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05fb4b4a647bbba5071604f9fa08cb58fe5b76ede194d21403f43d1d9745942f\": container with ID starting with 05fb4b4a647bbba5071604f9fa08cb58fe5b76ede194d21403f43d1d9745942f not found: ID does not exist" containerID="05fb4b4a647bbba5071604f9fa08cb58fe5b76ede194d21403f43d1d9745942f" Oct 31 04:26:04.627426 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:04.627348 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05fb4b4a647bbba5071604f9fa08cb58fe5b76ede194d21403f43d1d9745942f"} err="failed to get container status \"05fb4b4a647bbba5071604f9fa08cb58fe5b76ede194d21403f43d1d9745942f\": rpc error: code = NotFound desc = could not find container \"05fb4b4a647bbba5071604f9fa08cb58fe5b76ede194d21403f43d1d9745942f\": container with ID starting with 05fb4b4a647bbba5071604f9fa08cb58fe5b76ede194d21403f43d1d9745942f not found: ID does not exist" Oct 31 04:26:04.627426 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:04.627372 4376 scope.go:122] "RemoveContainer" containerID="f9cbfef45650c56699306fde4739e979d5b6d74a078474c21b5a6d2d92557af2" Oct 31 04:26:04.627631 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:26:04.627613 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9cbfef45650c56699306fde4739e979d5b6d74a078474c21b5a6d2d92557af2\": container with ID starting with f9cbfef45650c56699306fde4739e979d5b6d74a078474c21b5a6d2d92557af2 not found: ID does not exist" containerID="f9cbfef45650c56699306fde4739e979d5b6d74a078474c21b5a6d2d92557af2" Oct 31 04:26:04.627675 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:04.627638 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9cbfef45650c56699306fde4739e979d5b6d74a078474c21b5a6d2d92557af2"} err="failed to get container status \"f9cbfef45650c56699306fde4739e979d5b6d74a078474c21b5a6d2d92557af2\": rpc error: code = NotFound desc = could not find container \"f9cbfef45650c56699306fde4739e979d5b6d74a078474c21b5a6d2d92557af2\": container with ID starting with f9cbfef45650c56699306fde4739e979d5b6d74a078474c21b5a6d2d92557af2 not found: ID does not exist" Oct 31 04:26:04.699805 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:04.699776 4376 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3f3e12d-e06e-4b4e-a23a-784e49d5cbaf-istio-data\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:26:04.699805 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:04.699799 4376 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3f3e12d-e06e-4b4e-a23a-784e49d5cbaf-istio-token\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:26:04.699805 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:04.699808 4376 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3f3e12d-e06e-4b4e-a23a-784e49d5cbaf-istio-podinfo\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:26:04.699990 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:04.699816 4376 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3f3e12d-e06e-4b4e-a23a-784e49d5cbaf-istio-envoy\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:26:04.699990 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:04.699825 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vm88k\" (UniqueName: \"kubernetes.io/projected/e3f3e12d-e06e-4b4e-a23a-784e49d5cbaf-kube-api-access-vm88k\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:26:04.938748 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:04.938717 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-801218-openshift-default-58b4b8685c-8gpjq"] Oct 31 04:26:04.952588 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:04.952556 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-801218-openshift-default-58b4b8685c-8gpjq"] Oct 31 04:26:05.498638 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:05.498611 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-7b4185-cfcb87d66-l8fzb" Oct 31 04:26:05.499107 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:05.499028 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7b4185-7c659d956c-jllqw" Oct 31 04:26:06.165672 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:06.165638 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-9154a6-85c9f9448c-d4pt4"] Oct 31 04:26:06.464408 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:06.463940 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3f3e12d-e06e-4b4e-a23a-784e49d5cbaf" path="/var/lib/kubelet/pods/e3f3e12d-e06e-4b4e-a23a-784e49d5cbaf/volumes" Oct 31 04:26:08.001066 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:08.001033 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a9b573-6c5c49445d-j6qkq"] Oct 31 04:26:08.001722 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:08.001705 4376 cpu_manager.go:414] "containerMap: removing container" podUID="e3f3e12d-e06e-4b4e-a23a-784e49d5cbaf" containerName="istio-proxy" Oct 31 04:26:08.001773 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:08.001723 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3f3e12d-e06e-4b4e-a23a-784e49d5cbaf" containerName="istio-proxy" Oct 31 04:26:08.001773 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:08.001743 4376 cpu_manager.go:414] "containerMap: removing container" podUID="862d48b9-2fae-4584-a58b-8dbd9d985857" containerName="payload-pre-processing" Oct 31 04:26:08.001773 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:08.001749 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="862d48b9-2fae-4584-a58b-8dbd9d985857" containerName="payload-pre-processing" Oct 31 04:26:08.001773 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:08.001760 4376 cpu_manager.go:414] "containerMap: removing container" podUID="11f03146-3b0e-4cb0-b42c-0cb1d1952997" containerName="payload-processing" Oct 31 04:26:08.001773 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:08.001766 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11f03146-3b0e-4cb0-b42c-0cb1d1952997" containerName="payload-processing" Oct 31 04:26:08.001942 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:08.001866 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="11f03146-3b0e-4cb0-b42c-0cb1d1952997" containerName="payload-processing" Oct 31 04:26:08.001942 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:08.001880 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3f3e12d-e06e-4b4e-a23a-784e49d5cbaf" containerName="istio-proxy" Oct 31 04:26:08.001942 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:08.001888 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3f3e12d-e06e-4b4e-a23a-784e49d5cbaf" containerName="istio-proxy" Oct 31 04:26:08.001942 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:08.001895 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="862d48b9-2fae-4584-a58b-8dbd9d985857" containerName="payload-pre-processing" Oct 31 04:26:08.006893 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:08.006865 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a9b573-6c5c49445d-j6qkq" Oct 31 04:26:08.024085 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:08.024055 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a9b573-6c5c49445d-j6qkq"] Oct 31 04:26:08.031711 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:08.031651 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-a9b573-dockercfg-6hh9c\"" Oct 31 04:26:08.035305 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:08.035279 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-a9b573\"" Oct 31 04:26:08.073145 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:08.073108 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab9e79a2-c585-4d4e-a26d-9422cf1138ea-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a9b573-6c5c49445d-j6qkq\" (UID: \"ab9e79a2-c585-4d4e-a26d-9422cf1138ea\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a9b573-6c5c49445d-j6qkq" Oct 31 04:26:08.073351 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:08.073231 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6qg2\" (UniqueName: \"kubernetes.io/projected/ab9e79a2-c585-4d4e-a26d-9422cf1138ea-kube-api-access-f6qg2\") pod \"payload-processing-e2e-ipp-b-a9b573-6c5c49445d-j6qkq\" (UID: \"ab9e79a2-c585-4d4e-a26d-9422cf1138ea\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a9b573-6c5c49445d-j6qkq" Oct 31 04:26:08.174740 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:08.174709 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab9e79a2-c585-4d4e-a26d-9422cf1138ea-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a9b573-6c5c49445d-j6qkq\" (UID: \"ab9e79a2-c585-4d4e-a26d-9422cf1138ea\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a9b573-6c5c49445d-j6qkq" Oct 31 04:26:08.174901 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:08.174864 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6qg2\" (UniqueName: \"kubernetes.io/projected/ab9e79a2-c585-4d4e-a26d-9422cf1138ea-kube-api-access-f6qg2\") pod \"payload-processing-e2e-ipp-b-a9b573-6c5c49445d-j6qkq\" (UID: \"ab9e79a2-c585-4d4e-a26d-9422cf1138ea\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a9b573-6c5c49445d-j6qkq" Oct 31 04:26:08.175699 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:08.175675 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab9e79a2-c585-4d4e-a26d-9422cf1138ea-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a9b573-6c5c49445d-j6qkq\" (UID: \"ab9e79a2-c585-4d4e-a26d-9422cf1138ea\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a9b573-6c5c49445d-j6qkq" Oct 31 04:26:08.198868 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:08.198837 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6qg2\" (UniqueName: \"kubernetes.io/projected/ab9e79a2-c585-4d4e-a26d-9422cf1138ea-kube-api-access-f6qg2\") pod \"payload-processing-e2e-ipp-b-a9b573-6c5c49445d-j6qkq\" (UID: \"ab9e79a2-c585-4d4e-a26d-9422cf1138ea\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a9b573-6c5c49445d-j6qkq" Oct 31 04:26:08.321231 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:08.321156 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a9b573-6c5c49445d-j6qkq" Oct 31 04:26:08.912974 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:26:08.912944 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab9e79a2_c585_4d4e_a26d_9422cf1138ea.slice/crio-8a2a2164fb30f2d6eebdca849d1c5092b798d104bbd7ca528038724e08a37019 WatchSource:0}: Error finding container 8a2a2164fb30f2d6eebdca849d1c5092b798d104bbd7ca528038724e08a37019: Status 404 returned error can't find the container with id 8a2a2164fb30f2d6eebdca849d1c5092b798d104bbd7ca528038724e08a37019 Oct 31 04:26:08.914677 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:08.914573 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a9b573-6c5c49445d-j6qkq"] Oct 31 04:26:09.635004 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:09.634971 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a9b573-6c5c49445d-j6qkq" event={"ID":"ab9e79a2-c585-4d4e-a26d-9422cf1138ea","Type":"ContainerStarted","Data":"8181be2463a1bfa52eba3d6fbb2e6b77441d142e33c50a4dcdd857b16d1d9786"} Oct 31 04:26:09.635438 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:09.635020 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a9b573-6c5c49445d-j6qkq" event={"ID":"ab9e79a2-c585-4d4e-a26d-9422cf1138ea","Type":"ContainerStarted","Data":"8a2a2164fb30f2d6eebdca849d1c5092b798d104bbd7ca528038724e08a37019"} Oct 31 04:26:09.635438 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:09.635163 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-a9b573-6c5c49445d-j6qkq" Oct 31 04:26:09.681464 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:09.681408 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-a9b573-6c5c49445d-j6qkq" podStartSLOduration=2.681393074 podStartE2EDuration="2.681393074s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:26:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:26:09.677814398 +0000 UTC m=+1317.392149041" watchObservedRunningTime="2026-10-31 04:26:09.681393074 +0000 UTC m=+1317.395727694" Oct 31 04:26:11.000848 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:26:11.000813 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 04:26:11.166768 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:11.166725 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9154a6-85c9f9448c-d4pt4" podUID="097d0283-9c56-4738-b296-3255c3367cd8" containerName="payload-pre-processing" containerID="cri-o://c1a4d54b05678e409b431b08e933d0abc0df1ec303686498a3937c6a0f74a505" gracePeriod=25 Oct 31 04:26:11.435552 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:11.435526 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9154a6-85c9f9448c-d4pt4" Oct 31 04:26:11.560026 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:11.559935 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/097d0283-9c56-4738-b296-3255c3367cd8-kube-api-access-947fp\" (UniqueName: \"kubernetes.io/projected/097d0283-9c56-4738-b296-3255c3367cd8-kube-api-access-947fp\") pod \"097d0283-9c56-4738-b296-3255c3367cd8\" (UID: \"097d0283-9c56-4738-b296-3255c3367cd8\") " Oct 31 04:26:11.560179 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:11.560116 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/097d0283-9c56-4738-b296-3255c3367cd8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/097d0283-9c56-4738-b296-3255c3367cd8-plugins-config-volume\") pod \"097d0283-9c56-4738-b296-3255c3367cd8\" (UID: \"097d0283-9c56-4738-b296-3255c3367cd8\") " Oct 31 04:26:11.562461 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:11.560810 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/097d0283-9c56-4738-b296-3255c3367cd8-plugins-config-volume" pod "097d0283-9c56-4738-b296-3255c3367cd8" (UID: "097d0283-9c56-4738-b296-3255c3367cd8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:26:11.562461 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:11.561828 4376 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/097d0283-9c56-4738-b296-3255c3367cd8-plugins-config-volume\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:26:11.562673 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:11.562509 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/097d0283-9c56-4738-b296-3255c3367cd8-kube-api-access-947fp" pod "097d0283-9c56-4738-b296-3255c3367cd8" (UID: "097d0283-9c56-4738-b296-3255c3367cd8"). InnerVolumeSpecName "kube-api-access-947fp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:26:11.649859 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:11.649819 4376 generic.go:356] "Generic (PLEG): container finished" podID="097d0283-9c56-4738-b296-3255c3367cd8" containerID="c1a4d54b05678e409b431b08e933d0abc0df1ec303686498a3937c6a0f74a505" exitCode=0 Oct 31 04:26:11.650034 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:11.649884 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9154a6-85c9f9448c-d4pt4" Oct 31 04:26:11.650034 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:11.649906 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9154a6-85c9f9448c-d4pt4" event={"ID":"097d0283-9c56-4738-b296-3255c3367cd8","Type":"ContainerDied","Data":"c1a4d54b05678e409b431b08e933d0abc0df1ec303686498a3937c6a0f74a505"} Oct 31 04:26:11.650034 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:11.650002 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9154a6-85c9f9448c-d4pt4" event={"ID":"097d0283-9c56-4738-b296-3255c3367cd8","Type":"ContainerDied","Data":"273d1f93632615947158cce1b8e072a3628894794326e256c5fc71292d6bd27f"} Oct 31 04:26:11.650034 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:11.650026 4376 scope.go:122] "RemoveContainer" containerID="c1a4d54b05678e409b431b08e933d0abc0df1ec303686498a3937c6a0f74a505" Oct 31 04:26:11.660598 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:11.660576 4376 scope.go:122] "RemoveContainer" containerID="c1a4d54b05678e409b431b08e933d0abc0df1ec303686498a3937c6a0f74a505" Oct 31 04:26:11.660913 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:26:11.660892 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1a4d54b05678e409b431b08e933d0abc0df1ec303686498a3937c6a0f74a505\": container with ID starting with c1a4d54b05678e409b431b08e933d0abc0df1ec303686498a3937c6a0f74a505 not found: ID does not exist" containerID="c1a4d54b05678e409b431b08e933d0abc0df1ec303686498a3937c6a0f74a505" Oct 31 04:26:11.660977 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:11.660922 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1a4d54b05678e409b431b08e933d0abc0df1ec303686498a3937c6a0f74a505"} err="failed to get container status \"c1a4d54b05678e409b431b08e933d0abc0df1ec303686498a3937c6a0f74a505\": rpc error: code = NotFound desc = could not find container \"c1a4d54b05678e409b431b08e933d0abc0df1ec303686498a3937c6a0f74a505\": container with ID starting with c1a4d54b05678e409b431b08e933d0abc0df1ec303686498a3937c6a0f74a505 not found: ID does not exist" Oct 31 04:26:11.662952 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:11.662929 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-947fp\" (UniqueName: \"kubernetes.io/projected/097d0283-9c56-4738-b296-3255c3367cd8-kube-api-access-947fp\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:26:11.704091 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:11.704052 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-9154a6-85c9f9448c-d4pt4"] Oct 31 04:26:11.707957 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:11.707930 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-9154a6-85c9f9448c-d4pt4"] Oct 31 04:26:12.446170 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:12.446135 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="097d0283-9c56-4738-b296-3255c3367cd8" path="/var/lib/kubelet/pods/097d0283-9c56-4738-b296-3255c3367cd8/volumes" Oct 31 04:26:13.705686 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:13.705653 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-9154a6-openshift-default-7f74d4f9bc-v6fc7"] Oct 31 04:26:13.706456 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:13.706421 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-9154a6-openshift-default-7f74d4f9bc-v6fc7" podUID="1a7c4b33-999a-449b-bea8-9c55c838df0a" containerName="istio-proxy" containerID="cri-o://bc57377f8ebefa898aab1950c5d8528ef2d3a92f98fd7a8b356990c7c6635a2f" gracePeriod=30 Oct 31 04:26:15.577142 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:15.577106 4376 scope.go:122] "RemoveContainer" containerID="5f22b0d5ad7e86b40c8950f28b78f1537b10c187b38651b9b3faff39b698d58b" Oct 31 04:26:18.976832 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:18.976771 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-9154a6-openshift-default-7f74d4f9bc-v6fc7" Oct 31 04:26:19.038679 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:19.038636 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a7c4b33-999a-449b-bea8-9c55c838df0a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a7c4b33-999a-449b-bea8-9c55c838df0a-workload-certs\") pod \"1a7c4b33-999a-449b-bea8-9c55c838df0a\" (UID: \"1a7c4b33-999a-449b-bea8-9c55c838df0a\") " Oct 31 04:26:19.038867 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:19.038699 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a7c4b33-999a-449b-bea8-9c55c838df0a-kube-api-access-jpkrn\" (UniqueName: \"kubernetes.io/projected/1a7c4b33-999a-449b-bea8-9c55c838df0a-kube-api-access-jpkrn\") pod \"1a7c4b33-999a-449b-bea8-9c55c838df0a\" (UID: \"1a7c4b33-999a-449b-bea8-9c55c838df0a\") " Oct 31 04:26:19.038867 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:19.038816 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a7c4b33-999a-449b-bea8-9c55c838df0a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a7c4b33-999a-449b-bea8-9c55c838df0a-istiod-ca-cert\") pod \"1a7c4b33-999a-449b-bea8-9c55c838df0a\" (UID: \"1a7c4b33-999a-449b-bea8-9c55c838df0a\") " Oct 31 04:26:19.038999 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:19.038940 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a7c4b33-999a-449b-bea8-9c55c838df0a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a7c4b33-999a-449b-bea8-9c55c838df0a-istio-envoy\") pod \"1a7c4b33-999a-449b-bea8-9c55c838df0a\" (UID: \"1a7c4b33-999a-449b-bea8-9c55c838df0a\") " Oct 31 04:26:19.039058 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:19.039008 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a7c4b33-999a-449b-bea8-9c55c838df0a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a7c4b33-999a-449b-bea8-9c55c838df0a-istio-data\") pod \"1a7c4b33-999a-449b-bea8-9c55c838df0a\" (UID: \"1a7c4b33-999a-449b-bea8-9c55c838df0a\") " Oct 31 04:26:19.039058 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:19.039034 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a7c4b33-999a-449b-bea8-9c55c838df0a-workload-certs" pod "1a7c4b33-999a-449b-bea8-9c55c838df0a" (UID: "1a7c4b33-999a-449b-bea8-9c55c838df0a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:26:19.039175 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:19.039076 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a7c4b33-999a-449b-bea8-9c55c838df0a-istio-token\" (UniqueName: \"kubernetes.io/projected/1a7c4b33-999a-449b-bea8-9c55c838df0a-istio-token\") pod \"1a7c4b33-999a-449b-bea8-9c55c838df0a\" (UID: \"1a7c4b33-999a-449b-bea8-9c55c838df0a\") " Oct 31 04:26:19.039234 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:19.039190 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a7c4b33-999a-449b-bea8-9c55c838df0a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a7c4b33-999a-449b-bea8-9c55c838df0a-workload-socket\") pod \"1a7c4b33-999a-449b-bea8-9c55c838df0a\" (UID: \"1a7c4b33-999a-449b-bea8-9c55c838df0a\") " Oct 31 04:26:19.039291 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:19.039239 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a7c4b33-999a-449b-bea8-9c55c838df0a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a7c4b33-999a-449b-bea8-9c55c838df0a-credential-socket\") pod \"1a7c4b33-999a-449b-bea8-9c55c838df0a\" (UID: \"1a7c4b33-999a-449b-bea8-9c55c838df0a\") " Oct 31 04:26:19.039382 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:19.039287 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1a7c4b33-999a-449b-bea8-9c55c838df0a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a7c4b33-999a-449b-bea8-9c55c838df0a-istio-podinfo\") pod \"1a7c4b33-999a-449b-bea8-9c55c838df0a\" (UID: \"1a7c4b33-999a-449b-bea8-9c55c838df0a\") " Oct 31 04:26:19.039382 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:19.039301 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a7c4b33-999a-449b-bea8-9c55c838df0a-istiod-ca-cert" pod "1a7c4b33-999a-449b-bea8-9c55c838df0a" (UID: "1a7c4b33-999a-449b-bea8-9c55c838df0a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:26:19.039961 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:19.039803 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a7c4b33-999a-449b-bea8-9c55c838df0a-workload-socket" pod "1a7c4b33-999a-449b-bea8-9c55c838df0a" (UID: "1a7c4b33-999a-449b-bea8-9c55c838df0a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:26:19.039961 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:19.039920 4376 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a7c4b33-999a-449b-bea8-9c55c838df0a-workload-certs\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:26:19.039961 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:19.039941 4376 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a7c4b33-999a-449b-bea8-9c55c838df0a-istiod-ca-cert\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:26:19.039961 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:19.039954 4376 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a7c4b33-999a-449b-bea8-9c55c838df0a-workload-socket\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:26:19.040217 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:19.040144 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a7c4b33-999a-449b-bea8-9c55c838df0a-credential-socket" pod "1a7c4b33-999a-449b-bea8-9c55c838df0a" (UID: "1a7c4b33-999a-449b-bea8-9c55c838df0a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:26:19.040378 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:19.040340 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a7c4b33-999a-449b-bea8-9c55c838df0a-istio-data" pod "1a7c4b33-999a-449b-bea8-9c55c838df0a" (UID: "1a7c4b33-999a-449b-bea8-9c55c838df0a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:26:19.042512 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:19.042480 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a7c4b33-999a-449b-bea8-9c55c838df0a-istio-token" pod "1a7c4b33-999a-449b-bea8-9c55c838df0a" (UID: "1a7c4b33-999a-449b-bea8-9c55c838df0a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:26:19.042618 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:19.042536 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a7c4b33-999a-449b-bea8-9c55c838df0a-istio-envoy" pod "1a7c4b33-999a-449b-bea8-9c55c838df0a" (UID: "1a7c4b33-999a-449b-bea8-9c55c838df0a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:26:19.042618 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:19.042602 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1a7c4b33-999a-449b-bea8-9c55c838df0a-istio-podinfo" pod "1a7c4b33-999a-449b-bea8-9c55c838df0a" (UID: "1a7c4b33-999a-449b-bea8-9c55c838df0a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 04:26:19.043397 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:19.043375 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a7c4b33-999a-449b-bea8-9c55c838df0a-kube-api-access-jpkrn" pod "1a7c4b33-999a-449b-bea8-9c55c838df0a" (UID: "1a7c4b33-999a-449b-bea8-9c55c838df0a"). InnerVolumeSpecName "kube-api-access-jpkrn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:26:19.141521 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:19.141490 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jpkrn\" (UniqueName: \"kubernetes.io/projected/1a7c4b33-999a-449b-bea8-9c55c838df0a-kube-api-access-jpkrn\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:26:19.141521 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:19.141515 4376 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a7c4b33-999a-449b-bea8-9c55c838df0a-istio-envoy\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:26:19.141521 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:19.141524 4376 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a7c4b33-999a-449b-bea8-9c55c838df0a-istio-data\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:26:19.141751 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:19.141531 4376 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a7c4b33-999a-449b-bea8-9c55c838df0a-istio-token\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:26:19.141751 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:19.141540 4376 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a7c4b33-999a-449b-bea8-9c55c838df0a-credential-socket\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:26:19.141751 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:19.141548 4376 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a7c4b33-999a-449b-bea8-9c55c838df0a-istio-podinfo\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:26:19.728174 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:19.728135 4376 generic.go:356] "Generic (PLEG): container finished" podID="1a7c4b33-999a-449b-bea8-9c55c838df0a" containerID="bc57377f8ebefa898aab1950c5d8528ef2d3a92f98fd7a8b356990c7c6635a2f" exitCode=0 Oct 31 04:26:19.728394 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:19.728187 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-9154a6-openshift-default-7f74d4f9bc-v6fc7" Oct 31 04:26:19.728394 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:19.728210 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-9154a6-openshift-default-7f74d4f9bc-v6fc7" event={"ID":"1a7c4b33-999a-449b-bea8-9c55c838df0a","Type":"ContainerDied","Data":"bc57377f8ebefa898aab1950c5d8528ef2d3a92f98fd7a8b356990c7c6635a2f"} Oct 31 04:26:19.728394 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:19.728307 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-9154a6-openshift-default-7f74d4f9bc-v6fc7" event={"ID":"1a7c4b33-999a-449b-bea8-9c55c838df0a","Type":"ContainerDied","Data":"d36093d5520f0f5d2c9fce56ead55f790432267f2a465f9f74a555bfa424b58b"} Oct 31 04:26:19.728394 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:19.728354 4376 scope.go:122] "RemoveContainer" containerID="bc57377f8ebefa898aab1950c5d8528ef2d3a92f98fd7a8b356990c7c6635a2f" Oct 31 04:26:19.740105 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:19.740086 4376 scope.go:122] "RemoveContainer" containerID="d97124d31fee309b2b61051e27f2842e7c617becb5efaaa5d5b2133566921f90" Oct 31 04:26:19.749673 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:19.749650 4376 scope.go:122] "RemoveContainer" containerID="bc57377f8ebefa898aab1950c5d8528ef2d3a92f98fd7a8b356990c7c6635a2f" Oct 31 04:26:19.749972 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:26:19.749940 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc57377f8ebefa898aab1950c5d8528ef2d3a92f98fd7a8b356990c7c6635a2f\": container with ID starting with bc57377f8ebefa898aab1950c5d8528ef2d3a92f98fd7a8b356990c7c6635a2f not found: ID does not exist" containerID="bc57377f8ebefa898aab1950c5d8528ef2d3a92f98fd7a8b356990c7c6635a2f" Oct 31 04:26:19.750014 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:19.749983 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc57377f8ebefa898aab1950c5d8528ef2d3a92f98fd7a8b356990c7c6635a2f"} err="failed to get container status \"bc57377f8ebefa898aab1950c5d8528ef2d3a92f98fd7a8b356990c7c6635a2f\": rpc error: code = NotFound desc = could not find container \"bc57377f8ebefa898aab1950c5d8528ef2d3a92f98fd7a8b356990c7c6635a2f\": container with ID starting with bc57377f8ebefa898aab1950c5d8528ef2d3a92f98fd7a8b356990c7c6635a2f not found: ID does not exist" Oct 31 04:26:19.750014 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:19.750004 4376 scope.go:122] "RemoveContainer" containerID="d97124d31fee309b2b61051e27f2842e7c617becb5efaaa5d5b2133566921f90" Oct 31 04:26:19.750254 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:26:19.750238 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d97124d31fee309b2b61051e27f2842e7c617becb5efaaa5d5b2133566921f90\": container with ID starting with d97124d31fee309b2b61051e27f2842e7c617becb5efaaa5d5b2133566921f90 not found: ID does not exist" containerID="d97124d31fee309b2b61051e27f2842e7c617becb5efaaa5d5b2133566921f90" Oct 31 04:26:19.750322 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:19.750257 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d97124d31fee309b2b61051e27f2842e7c617becb5efaaa5d5b2133566921f90"} err="failed to get container status \"d97124d31fee309b2b61051e27f2842e7c617becb5efaaa5d5b2133566921f90\": rpc error: code = NotFound desc = could not find container \"d97124d31fee309b2b61051e27f2842e7c617becb5efaaa5d5b2133566921f90\": container with ID starting with d97124d31fee309b2b61051e27f2842e7c617becb5efaaa5d5b2133566921f90 not found: ID does not exist" Oct 31 04:26:19.799793 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:19.799762 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-9154a6-openshift-default-7f74d4f9bc-v6fc7"] Oct 31 04:26:19.811541 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:19.811512 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-9154a6-openshift-default-7f74d4f9bc-v6fc7"] Oct 31 04:26:19.851910 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:19.851873 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 04:26:20.444719 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:20.444685 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a7c4b33-999a-449b-bea8-9c55c838df0a" path="/var/lib/kubelet/pods/1a7c4b33-999a-449b-bea8-9c55c838df0a/volumes" Oct 31 04:26:20.658353 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:20.654497 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-a9b573-6c5c49445d-j6qkq" Oct 31 04:26:21.611400 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:21.611359 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ff43e3-576df466d7-4pqsr"] Oct 31 04:26:21.612341 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:21.612300 4376 cpu_manager.go:414] "containerMap: removing container" podUID="e3f3e12d-e06e-4b4e-a23a-784e49d5cbaf" containerName="istio-proxy" Oct 31 04:26:21.643411 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:21.612345 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3f3e12d-e06e-4b4e-a23a-784e49d5cbaf" containerName="istio-proxy" Oct 31 04:26:21.643411 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:21.632465 4376 cpu_manager.go:414] "containerMap: removing container" podUID="1a7c4b33-999a-449b-bea8-9c55c838df0a" containerName="istio-proxy" Oct 31 04:26:21.643411 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:21.632481 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a7c4b33-999a-449b-bea8-9c55c838df0a" containerName="istio-proxy" Oct 31 04:26:21.643411 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:21.632511 4376 cpu_manager.go:414] "containerMap: removing container" podUID="1a7c4b33-999a-449b-bea8-9c55c838df0a" containerName="istio-proxy" Oct 31 04:26:21.643411 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:21.632518 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a7c4b33-999a-449b-bea8-9c55c838df0a" containerName="istio-proxy" Oct 31 04:26:21.643411 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:21.632535 4376 cpu_manager.go:414] "containerMap: removing container" podUID="097d0283-9c56-4738-b296-3255c3367cd8" containerName="payload-pre-processing" Oct 31 04:26:21.643411 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:21.632543 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="097d0283-9c56-4738-b296-3255c3367cd8" containerName="payload-pre-processing" Oct 31 04:26:21.643411 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:21.642586 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a7c4b33-999a-449b-bea8-9c55c838df0a" containerName="istio-proxy" Oct 31 04:26:21.643411 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:21.642646 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a7c4b33-999a-449b-bea8-9c55c838df0a" containerName="istio-proxy" Oct 31 04:26:21.643411 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:21.642660 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="097d0283-9c56-4738-b296-3255c3367cd8" containerName="payload-pre-processing" Oct 31 04:26:21.658569 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:21.658530 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ff43e3-576df466d7-4pqsr"] Oct 31 04:26:21.658904 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:21.658883 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-ff43e3-576df466d7-4pqsr" Oct 31 04:26:21.774156 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:21.774121 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-ff43e3\"" Oct 31 04:26:21.775173 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:21.774745 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-ff43e3-dockercfg-jhhdh\"" Oct 31 04:26:21.878789 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:21.878748 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2cjg\" (UniqueName: \"kubernetes.io/projected/151dc5b7-7589-4707-8586-8f70ab9b8d66-kube-api-access-v2cjg\") pod \"payload-processing-e2e-ipp-a-ff43e3-576df466d7-4pqsr\" (UID: \"151dc5b7-7589-4707-8586-8f70ab9b8d66\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ff43e3-576df466d7-4pqsr" Oct 31 04:26:21.878979 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:21.878829 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/151dc5b7-7589-4707-8586-8f70ab9b8d66-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-ff43e3-576df466d7-4pqsr\" (UID: \"151dc5b7-7589-4707-8586-8f70ab9b8d66\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ff43e3-576df466d7-4pqsr" Oct 31 04:26:21.980408 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:21.980376 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2cjg\" (UniqueName: \"kubernetes.io/projected/151dc5b7-7589-4707-8586-8f70ab9b8d66-kube-api-access-v2cjg\") pod \"payload-processing-e2e-ipp-a-ff43e3-576df466d7-4pqsr\" (UID: \"151dc5b7-7589-4707-8586-8f70ab9b8d66\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ff43e3-576df466d7-4pqsr" Oct 31 04:26:21.980604 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:21.980468 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/151dc5b7-7589-4707-8586-8f70ab9b8d66-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-ff43e3-576df466d7-4pqsr\" (UID: \"151dc5b7-7589-4707-8586-8f70ab9b8d66\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ff43e3-576df466d7-4pqsr" Oct 31 04:26:21.981179 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:21.981158 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/151dc5b7-7589-4707-8586-8f70ab9b8d66-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-ff43e3-576df466d7-4pqsr\" (UID: \"151dc5b7-7589-4707-8586-8f70ab9b8d66\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ff43e3-576df466d7-4pqsr" Oct 31 04:26:22.004082 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:22.004053 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2cjg\" (UniqueName: \"kubernetes.io/projected/151dc5b7-7589-4707-8586-8f70ab9b8d66-kube-api-access-v2cjg\") pod \"payload-processing-e2e-ipp-a-ff43e3-576df466d7-4pqsr\" (UID: \"151dc5b7-7589-4707-8586-8f70ab9b8d66\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ff43e3-576df466d7-4pqsr" Oct 31 04:26:22.133063 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:22.132988 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-ff43e3-576df466d7-4pqsr" Oct 31 04:26:22.435097 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:26:22.435055 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod151dc5b7_7589_4707_8586_8f70ab9b8d66.slice/crio-4077cab580769dc16fd0c4abe0ca0ae37aab3b7b16052f1d320509c1630a8e45 WatchSource:0}: Error finding container 4077cab580769dc16fd0c4abe0ca0ae37aab3b7b16052f1d320509c1630a8e45: Status 404 returned error can't find the container with id 4077cab580769dc16fd0c4abe0ca0ae37aab3b7b16052f1d320509c1630a8e45 Oct 31 04:26:22.448566 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:22.448533 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ff43e3-576df466d7-4pqsr"] Oct 31 04:26:22.842824 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:22.842737 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-ff43e3-576df466d7-4pqsr" event={"ID":"151dc5b7-7589-4707-8586-8f70ab9b8d66","Type":"ContainerStarted","Data":"8c854abddec1682d9dac4d9ac1cb368880ff0b5b4f1ae8cbf82cd73db1102616"} Oct 31 04:26:22.843195 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:22.842840 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-ff43e3-576df466d7-4pqsr" Oct 31 04:26:22.843195 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:22.842867 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-ff43e3-576df466d7-4pqsr" event={"ID":"151dc5b7-7589-4707-8586-8f70ab9b8d66","Type":"ContainerStarted","Data":"4077cab580769dc16fd0c4abe0ca0ae37aab3b7b16052f1d320509c1630a8e45"} Oct 31 04:26:22.886564 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:22.886512 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-ff43e3-576df466d7-4pqsr" podStartSLOduration=1.886496129 podStartE2EDuration="1.886496129s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:26:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:26:22.884688207 +0000 UTC m=+1330.599022824" watchObservedRunningTime="2026-10-31 04:26:22.886496129 +0000 UTC m=+1330.600830744" Oct 31 04:26:24.926432 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:24.926395 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3430f6-774fd99d88-fqpg2"] Oct 31 04:26:24.927058 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:24.926611 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3430f6-774fd99d88-fqpg2" podUID="6a62e709-216f-42e4-ac81-0c6712e61f31" containerName="maas-api" containerID="cri-o://ef3dcd284ba302f855f8e6377b6e79f8e4f82e34185b96e76b422847f4aa7e97" gracePeriod=30 Oct 31 04:26:25.109140 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:25.109105 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3430f6-84fd4c6898-7rzjt"] Oct 31 04:26:25.109402 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:25.109373 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-3430f6-84fd4c6898-7rzjt" podUID="38a822e5-1a32-4126-b19a-1264d09c66cc" containerName="payload-processing" containerID="cri-o://8bdd819bc76dcc8338813f4c7f4153ecefb2c92aa76ad06296ade5a02922efd1" gracePeriod=30 Oct 31 04:26:25.246430 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:25.246400 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3430f6-774fd99d88-fqpg2" Oct 31 04:26:25.331935 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:25.331901 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a62e709-216f-42e4-ac81-0c6712e61f31-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a62e709-216f-42e4-ac81-0c6712e61f31-maas-api-tls\") pod \"6a62e709-216f-42e4-ac81-0c6712e61f31\" (UID: \"6a62e709-216f-42e4-ac81-0c6712e61f31\") " Oct 31 04:26:25.332112 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:25.331950 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a62e709-216f-42e4-ac81-0c6712e61f31-kube-api-access-f8pbn\" (UniqueName: \"kubernetes.io/projected/6a62e709-216f-42e4-ac81-0c6712e61f31-kube-api-access-f8pbn\") pod \"6a62e709-216f-42e4-ac81-0c6712e61f31\" (UID: \"6a62e709-216f-42e4-ac81-0c6712e61f31\") " Oct 31 04:26:25.332112 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:25.331984 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a62e709-216f-42e4-ac81-0c6712e61f31-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a62e709-216f-42e4-ac81-0c6712e61f31-openshift-service-ca\") pod \"6a62e709-216f-42e4-ac81-0c6712e61f31\" (UID: \"6a62e709-216f-42e4-ac81-0c6712e61f31\") " Oct 31 04:26:25.332676 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:25.332641 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a62e709-216f-42e4-ac81-0c6712e61f31-openshift-service-ca" pod "6a62e709-216f-42e4-ac81-0c6712e61f31" (UID: "6a62e709-216f-42e4-ac81-0c6712e61f31"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:26:25.334160 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:25.334133 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a62e709-216f-42e4-ac81-0c6712e61f31-maas-api-tls" pod "6a62e709-216f-42e4-ac81-0c6712e61f31" (UID: "6a62e709-216f-42e4-ac81-0c6712e61f31"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:26:25.334504 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:25.334475 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a62e709-216f-42e4-ac81-0c6712e61f31-kube-api-access-f8pbn" pod "6a62e709-216f-42e4-ac81-0c6712e61f31" (UID: "6a62e709-216f-42e4-ac81-0c6712e61f31"). InnerVolumeSpecName "kube-api-access-f8pbn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:26:25.398190 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:25.398162 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3430f6-84fd4c6898-7rzjt" Oct 31 04:26:25.448349 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:25.444785 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a62e709-216f-42e4-ac81-0c6712e61f31-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a62e709-216f-42e4-ac81-0c6712e61f31-metrics-certs\") pod \"6a62e709-216f-42e4-ac81-0c6712e61f31\" (UID: \"6a62e709-216f-42e4-ac81-0c6712e61f31\") " Oct 31 04:26:25.448349 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:25.445553 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8pbn\" (UniqueName: \"kubernetes.io/projected/6a62e709-216f-42e4-ac81-0c6712e61f31-kube-api-access-f8pbn\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:26:25.448349 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:25.445575 4376 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a62e709-216f-42e4-ac81-0c6712e61f31-openshift-service-ca\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:26:25.448349 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:25.445590 4376 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a62e709-216f-42e4-ac81-0c6712e61f31-maas-api-tls\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:26:25.515404 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:25.513496 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a62e709-216f-42e4-ac81-0c6712e61f31-metrics-certs" pod "6a62e709-216f-42e4-ac81-0c6712e61f31" (UID: "6a62e709-216f-42e4-ac81-0c6712e61f31"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:26:25.551357 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:25.550421 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38a822e5-1a32-4126-b19a-1264d09c66cc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38a822e5-1a32-4126-b19a-1264d09c66cc-plugins-config-volume\") pod \"38a822e5-1a32-4126-b19a-1264d09c66cc\" (UID: \"38a822e5-1a32-4126-b19a-1264d09c66cc\") " Oct 31 04:26:25.551357 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:25.550622 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38a822e5-1a32-4126-b19a-1264d09c66cc-kube-api-access-597d9\" (UniqueName: \"kubernetes.io/projected/38a822e5-1a32-4126-b19a-1264d09c66cc-kube-api-access-597d9\") pod \"38a822e5-1a32-4126-b19a-1264d09c66cc\" (UID: \"38a822e5-1a32-4126-b19a-1264d09c66cc\") " Oct 31 04:26:25.551357 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:25.551251 4376 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a62e709-216f-42e4-ac81-0c6712e61f31-metrics-certs\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:26:25.579849 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:25.579811 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38a822e5-1a32-4126-b19a-1264d09c66cc-plugins-config-volume" pod "38a822e5-1a32-4126-b19a-1264d09c66cc" (UID: "38a822e5-1a32-4126-b19a-1264d09c66cc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:26:25.582766 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:25.582737 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38a822e5-1a32-4126-b19a-1264d09c66cc-kube-api-access-597d9" pod "38a822e5-1a32-4126-b19a-1264d09c66cc" (UID: "38a822e5-1a32-4126-b19a-1264d09c66cc"). InnerVolumeSpecName "kube-api-access-597d9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:26:25.659848 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:25.659809 4376 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38a822e5-1a32-4126-b19a-1264d09c66cc-plugins-config-volume\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:26:25.660085 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:25.660070 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-597d9\" (UniqueName: \"kubernetes.io/projected/38a822e5-1a32-4126-b19a-1264d09c66cc-kube-api-access-597d9\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:26:25.863944 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:25.863861 4376 generic.go:356] "Generic (PLEG): container finished" podID="38a822e5-1a32-4126-b19a-1264d09c66cc" containerID="8bdd819bc76dcc8338813f4c7f4153ecefb2c92aa76ad06296ade5a02922efd1" exitCode=0 Oct 31 04:26:25.863944 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:25.863901 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3430f6-84fd4c6898-7rzjt" Oct 31 04:26:25.864152 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:25.863949 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3430f6-84fd4c6898-7rzjt" event={"ID":"38a822e5-1a32-4126-b19a-1264d09c66cc","Type":"ContainerDied","Data":"8bdd819bc76dcc8338813f4c7f4153ecefb2c92aa76ad06296ade5a02922efd1"} Oct 31 04:26:25.864152 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:25.864012 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3430f6-84fd4c6898-7rzjt" event={"ID":"38a822e5-1a32-4126-b19a-1264d09c66cc","Type":"ContainerDied","Data":"b754c242bb4e6883e7992105c3037d8f3f1506af9c8fac0749fadabbc32dd7b0"} Oct 31 04:26:25.864152 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:25.864045 4376 scope.go:122] "RemoveContainer" containerID="8bdd819bc76dcc8338813f4c7f4153ecefb2c92aa76ad06296ade5a02922efd1" Oct 31 04:26:25.865451 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:25.865425 4376 generic.go:356] "Generic (PLEG): container finished" podID="6a62e709-216f-42e4-ac81-0c6712e61f31" containerID="ef3dcd284ba302f855f8e6377b6e79f8e4f82e34185b96e76b422847f4aa7e97" exitCode=0 Oct 31 04:26:25.865529 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:25.865471 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3430f6-774fd99d88-fqpg2" event={"ID":"6a62e709-216f-42e4-ac81-0c6712e61f31","Type":"ContainerDied","Data":"ef3dcd284ba302f855f8e6377b6e79f8e4f82e34185b96e76b422847f4aa7e97"} Oct 31 04:26:25.865529 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:25.865482 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3430f6-774fd99d88-fqpg2" Oct 31 04:26:25.865529 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:25.865505 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3430f6-774fd99d88-fqpg2" event={"ID":"6a62e709-216f-42e4-ac81-0c6712e61f31","Type":"ContainerDied","Data":"c7dc1de2bd85d9e990c84275778b894cd04161739fe0ea2e0d108c8584872654"} Oct 31 04:26:25.876181 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:25.876165 4376 scope.go:122] "RemoveContainer" containerID="8bdd819bc76dcc8338813f4c7f4153ecefb2c92aa76ad06296ade5a02922efd1" Oct 31 04:26:25.876503 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:26:25.876478 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8bdd819bc76dcc8338813f4c7f4153ecefb2c92aa76ad06296ade5a02922efd1\": container with ID starting with 8bdd819bc76dcc8338813f4c7f4153ecefb2c92aa76ad06296ade5a02922efd1 not found: ID does not exist" containerID="8bdd819bc76dcc8338813f4c7f4153ecefb2c92aa76ad06296ade5a02922efd1" Oct 31 04:26:25.876588 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:25.876511 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8bdd819bc76dcc8338813f4c7f4153ecefb2c92aa76ad06296ade5a02922efd1"} err="failed to get container status \"8bdd819bc76dcc8338813f4c7f4153ecefb2c92aa76ad06296ade5a02922efd1\": rpc error: code = NotFound desc = could not find container \"8bdd819bc76dcc8338813f4c7f4153ecefb2c92aa76ad06296ade5a02922efd1\": container with ID starting with 8bdd819bc76dcc8338813f4c7f4153ecefb2c92aa76ad06296ade5a02922efd1 not found: ID does not exist" Oct 31 04:26:25.876588 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:25.876535 4376 scope.go:122] "RemoveContainer" containerID="ef3dcd284ba302f855f8e6377b6e79f8e4f82e34185b96e76b422847f4aa7e97" Oct 31 04:26:25.885634 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:25.885613 4376 scope.go:122] "RemoveContainer" containerID="ef3dcd284ba302f855f8e6377b6e79f8e4f82e34185b96e76b422847f4aa7e97" Oct 31 04:26:25.885886 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:26:25.885866 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef3dcd284ba302f855f8e6377b6e79f8e4f82e34185b96e76b422847f4aa7e97\": container with ID starting with ef3dcd284ba302f855f8e6377b6e79f8e4f82e34185b96e76b422847f4aa7e97 not found: ID does not exist" containerID="ef3dcd284ba302f855f8e6377b6e79f8e4f82e34185b96e76b422847f4aa7e97" Oct 31 04:26:25.885948 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:25.885896 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef3dcd284ba302f855f8e6377b6e79f8e4f82e34185b96e76b422847f4aa7e97"} err="failed to get container status \"ef3dcd284ba302f855f8e6377b6e79f8e4f82e34185b96e76b422847f4aa7e97\": rpc error: code = NotFound desc = could not find container \"ef3dcd284ba302f855f8e6377b6e79f8e4f82e34185b96e76b422847f4aa7e97\": container with ID starting with ef3dcd284ba302f855f8e6377b6e79f8e4f82e34185b96e76b422847f4aa7e97 not found: ID does not exist" Oct 31 04:26:25.922173 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:25.922144 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3430f6-774fd99d88-fqpg2"] Oct 31 04:26:25.926770 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:25.926742 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3430f6-774fd99d88-fqpg2"] Oct 31 04:26:25.961631 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:25.961603 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3430f6-84fd4c6898-7rzjt"] Oct 31 04:26:25.967412 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:25.967384 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3430f6-84fd4c6898-7rzjt"] Oct 31 04:26:26.444123 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:26.444094 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38a822e5-1a32-4126-b19a-1264d09c66cc" path="/var/lib/kubelet/pods/38a822e5-1a32-4126-b19a-1264d09c66cc/volumes" Oct 31 04:26:26.444575 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:26.444560 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a62e709-216f-42e4-ac81-0c6712e61f31" path="/var/lib/kubelet/pods/6a62e709-216f-42e4-ac81-0c6712e61f31/volumes" Oct 31 04:26:27.416557 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:27.416531 4376 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-cpb5g_83cfbbfb-a15c-4967-b20c-e102351c6b7b/payload-processing/0.log" Oct 31 04:26:27.645919 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:27.645886 4376 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-cpb5g_83cfbbfb-a15c-4967-b20c-e102351c6b7b/payload-processing/0.log" Oct 31 04:26:32.292602 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:32.292571 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-3430f6-openshift-default-595d4f6b8c-ltm6g"] Oct 31 04:26:32.293122 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:32.292828 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-3430f6-openshift-default-595d4f6b8c-ltm6g" podUID="3ade7461-87d8-42ea-a721-512bae1f5f65" containerName="istio-proxy" containerID="cri-o://b8ee493ca4d59dc205b420f6ff1185cc92bd561067da0ac75193342d13b2504a" gracePeriod=30 Oct 31 04:26:33.585044 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:33.585003 4376 patch_prober.go:29] interesting pod/e2e-ipp-a-3430f6-openshift-default-595d4f6b8c-ltm6g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 04:26:33.585438 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:33.585073 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-3430f6-openshift-default-595d4f6b8c-ltm6g" podUID="3ade7461-87d8-42ea-a721-512bae1f5f65" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 04:26:33.853002 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:33.852917 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-ff43e3-576df466d7-4pqsr" Oct 31 04:26:36.843603 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:36.843567 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ceba51-6c8fc9774c-vk52b"] Oct 31 04:26:36.844311 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:36.844273 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ceba51-6c8fc9774c-vk52b" podUID="c503874b-c276-4ecf-b767-952b44173d94" containerName="payload-pre-processing" containerID="cri-o://f53d83918d3952eae679a0f9744e5d9ef0f06c2cb77f0b740c631e12b7464894" gracePeriod=30 Oct 31 04:26:36.994853 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:36.994819 4376 generic.go:356] "Generic (PLEG): container finished" podID="c503874b-c276-4ecf-b767-952b44173d94" containerID="f53d83918d3952eae679a0f9744e5d9ef0f06c2cb77f0b740c631e12b7464894" exitCode=0 Oct 31 04:26:36.995036 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:36.994873 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ceba51-6c8fc9774c-vk52b" event={"ID":"c503874b-c276-4ecf-b767-952b44173d94","Type":"ContainerDied","Data":"f53d83918d3952eae679a0f9744e5d9ef0f06c2cb77f0b740c631e12b7464894"} Oct 31 04:26:37.241350 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:37.233505 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ceba51-6c8fc9774c-vk52b" Oct 31 04:26:37.402345 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:37.402237 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c503874b-c276-4ecf-b767-952b44173d94-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c503874b-c276-4ecf-b767-952b44173d94-plugins-config-volume\") pod \"c503874b-c276-4ecf-b767-952b44173d94\" (UID: \"c503874b-c276-4ecf-b767-952b44173d94\") " Oct 31 04:26:37.402561 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:37.402538 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c503874b-c276-4ecf-b767-952b44173d94-kube-api-access-txzfs\" (UniqueName: \"kubernetes.io/projected/c503874b-c276-4ecf-b767-952b44173d94-kube-api-access-txzfs\") pod \"c503874b-c276-4ecf-b767-952b44173d94\" (UID: \"c503874b-c276-4ecf-b767-952b44173d94\") " Oct 31 04:26:37.402630 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:37.402607 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c503874b-c276-4ecf-b767-952b44173d94-plugins-config-volume" pod "c503874b-c276-4ecf-b767-952b44173d94" (UID: "c503874b-c276-4ecf-b767-952b44173d94"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:26:37.403173 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:37.403149 4376 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c503874b-c276-4ecf-b767-952b44173d94-plugins-config-volume\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:26:37.404669 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:37.404643 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c503874b-c276-4ecf-b767-952b44173d94-kube-api-access-txzfs" pod "c503874b-c276-4ecf-b767-952b44173d94" (UID: "c503874b-c276-4ecf-b767-952b44173d94"). InnerVolumeSpecName "kube-api-access-txzfs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:26:37.504698 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:37.504656 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-txzfs\" (UniqueName: \"kubernetes.io/projected/c503874b-c276-4ecf-b767-952b44173d94-kube-api-access-txzfs\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:26:37.564841 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:37.564813 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3430f6-openshift-default-595d4f6b8c-ltm6g" Oct 31 04:26:37.707295 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:37.707206 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ade7461-87d8-42ea-a721-512bae1f5f65-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ade7461-87d8-42ea-a721-512bae1f5f65-workload-socket\") pod \"3ade7461-87d8-42ea-a721-512bae1f5f65\" (UID: \"3ade7461-87d8-42ea-a721-512bae1f5f65\") " Oct 31 04:26:37.707496 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:37.707304 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3ade7461-87d8-42ea-a721-512bae1f5f65-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ade7461-87d8-42ea-a721-512bae1f5f65-istio-podinfo\") pod \"3ade7461-87d8-42ea-a721-512bae1f5f65\" (UID: \"3ade7461-87d8-42ea-a721-512bae1f5f65\") " Oct 31 04:26:37.707496 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:37.707376 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ade7461-87d8-42ea-a721-512bae1f5f65-istio-token\" (UniqueName: \"kubernetes.io/projected/3ade7461-87d8-42ea-a721-512bae1f5f65-istio-token\") pod \"3ade7461-87d8-42ea-a721-512bae1f5f65\" (UID: \"3ade7461-87d8-42ea-a721-512bae1f5f65\") " Oct 31 04:26:37.707496 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:37.707406 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ade7461-87d8-42ea-a721-512bae1f5f65-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ade7461-87d8-42ea-a721-512bae1f5f65-credential-socket\") pod \"3ade7461-87d8-42ea-a721-512bae1f5f65\" (UID: \"3ade7461-87d8-42ea-a721-512bae1f5f65\") " Oct 31 04:26:37.707649 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:37.707512 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ade7461-87d8-42ea-a721-512bae1f5f65-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ade7461-87d8-42ea-a721-512bae1f5f65-istio-envoy\") pod \"3ade7461-87d8-42ea-a721-512bae1f5f65\" (UID: \"3ade7461-87d8-42ea-a721-512bae1f5f65\") " Oct 31 04:26:37.707649 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:37.707521 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ade7461-87d8-42ea-a721-512bae1f5f65-workload-socket" pod "3ade7461-87d8-42ea-a721-512bae1f5f65" (UID: "3ade7461-87d8-42ea-a721-512bae1f5f65"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:26:37.707649 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:37.707543 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ade7461-87d8-42ea-a721-512bae1f5f65-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ade7461-87d8-42ea-a721-512bae1f5f65-workload-certs\") pod \"3ade7461-87d8-42ea-a721-512bae1f5f65\" (UID: \"3ade7461-87d8-42ea-a721-512bae1f5f65\") " Oct 31 04:26:37.707649 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:37.707615 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ade7461-87d8-42ea-a721-512bae1f5f65-kube-api-access-bxmq8\" (UniqueName: \"kubernetes.io/projected/3ade7461-87d8-42ea-a721-512bae1f5f65-kube-api-access-bxmq8\") pod \"3ade7461-87d8-42ea-a721-512bae1f5f65\" (UID: \"3ade7461-87d8-42ea-a721-512bae1f5f65\") " Oct 31 04:26:37.707835 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:37.707710 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ade7461-87d8-42ea-a721-512bae1f5f65-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ade7461-87d8-42ea-a721-512bae1f5f65-istio-data\") pod \"3ade7461-87d8-42ea-a721-512bae1f5f65\" (UID: \"3ade7461-87d8-42ea-a721-512bae1f5f65\") " Oct 31 04:26:37.707835 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:37.707741 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ade7461-87d8-42ea-a721-512bae1f5f65-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ade7461-87d8-42ea-a721-512bae1f5f65-istiod-ca-cert\") pod \"3ade7461-87d8-42ea-a721-512bae1f5f65\" (UID: \"3ade7461-87d8-42ea-a721-512bae1f5f65\") " Oct 31 04:26:37.707835 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:37.707752 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ade7461-87d8-42ea-a721-512bae1f5f65-credential-socket" pod "3ade7461-87d8-42ea-a721-512bae1f5f65" (UID: "3ade7461-87d8-42ea-a721-512bae1f5f65"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:26:37.707963 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:37.707833 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ade7461-87d8-42ea-a721-512bae1f5f65-workload-certs" pod "3ade7461-87d8-42ea-a721-512bae1f5f65" (UID: "3ade7461-87d8-42ea-a721-512bae1f5f65"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:26:37.708014 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:37.707972 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ade7461-87d8-42ea-a721-512bae1f5f65-istio-data" pod "3ade7461-87d8-42ea-a721-512bae1f5f65" (UID: "3ade7461-87d8-42ea-a721-512bae1f5f65"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:26:37.708284 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:37.708260 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ade7461-87d8-42ea-a721-512bae1f5f65-istiod-ca-cert" pod "3ade7461-87d8-42ea-a721-512bae1f5f65" (UID: "3ade7461-87d8-42ea-a721-512bae1f5f65"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:26:37.708459 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:37.708440 4376 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ade7461-87d8-42ea-a721-512bae1f5f65-credential-socket\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:26:37.708501 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:37.708466 4376 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ade7461-87d8-42ea-a721-512bae1f5f65-workload-certs\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:26:37.708501 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:37.708482 4376 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ade7461-87d8-42ea-a721-512bae1f5f65-istio-data\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:26:37.708501 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:37.708494 4376 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ade7461-87d8-42ea-a721-512bae1f5f65-istiod-ca-cert\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:26:37.708594 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:37.708506 4376 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ade7461-87d8-42ea-a721-512bae1f5f65-workload-socket\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:26:37.709813 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:37.709783 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ade7461-87d8-42ea-a721-512bae1f5f65-kube-api-access-bxmq8" pod "3ade7461-87d8-42ea-a721-512bae1f5f65" (UID: "3ade7461-87d8-42ea-a721-512bae1f5f65"). InnerVolumeSpecName "kube-api-access-bxmq8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:26:37.709813 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:37.709800 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ade7461-87d8-42ea-a721-512bae1f5f65-istio-token" pod "3ade7461-87d8-42ea-a721-512bae1f5f65" (UID: "3ade7461-87d8-42ea-a721-512bae1f5f65"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:26:37.709961 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:37.709941 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3ade7461-87d8-42ea-a721-512bae1f5f65-istio-podinfo" pod "3ade7461-87d8-42ea-a721-512bae1f5f65" (UID: "3ade7461-87d8-42ea-a721-512bae1f5f65"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 04:26:37.710012 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:37.709939 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ade7461-87d8-42ea-a721-512bae1f5f65-istio-envoy" pod "3ade7461-87d8-42ea-a721-512bae1f5f65" (UID: "3ade7461-87d8-42ea-a721-512bae1f5f65"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:26:37.810222 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:37.810192 4376 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ade7461-87d8-42ea-a721-512bae1f5f65-istio-envoy\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:26:37.810222 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:37.810215 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxmq8\" (UniqueName: \"kubernetes.io/projected/3ade7461-87d8-42ea-a721-512bae1f5f65-kube-api-access-bxmq8\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:26:37.810222 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:37.810226 4376 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ade7461-87d8-42ea-a721-512bae1f5f65-istio-podinfo\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:26:37.810452 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:37.810235 4376 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ade7461-87d8-42ea-a721-512bae1f5f65-istio-token\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:26:38.004011 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:38.003923 4376 generic.go:356] "Generic (PLEG): container finished" podID="3ade7461-87d8-42ea-a721-512bae1f5f65" containerID="b8ee493ca4d59dc205b420f6ff1185cc92bd561067da0ac75193342d13b2504a" exitCode=0 Oct 31 04:26:38.004011 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:38.003966 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3430f6-openshift-default-595d4f6b8c-ltm6g" Oct 31 04:26:38.004497 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:38.004016 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3430f6-openshift-default-595d4f6b8c-ltm6g" event={"ID":"3ade7461-87d8-42ea-a721-512bae1f5f65","Type":"ContainerDied","Data":"b8ee493ca4d59dc205b420f6ff1185cc92bd561067da0ac75193342d13b2504a"} Oct 31 04:26:38.004497 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:38.004068 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3430f6-openshift-default-595d4f6b8c-ltm6g" event={"ID":"3ade7461-87d8-42ea-a721-512bae1f5f65","Type":"ContainerDied","Data":"d39c85cdb688145cbe9d5d675d15b8f6207cc93c82f7da029d6255464fee18ce"} Oct 31 04:26:38.004497 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:38.004099 4376 scope.go:122] "RemoveContainer" containerID="b8ee493ca4d59dc205b420f6ff1185cc92bd561067da0ac75193342d13b2504a" Oct 31 04:26:38.005792 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:38.005633 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ceba51-6c8fc9774c-vk52b" event={"ID":"c503874b-c276-4ecf-b767-952b44173d94","Type":"ContainerDied","Data":"6aeb1a9a7190a5b2a66fbedac89e61ecc8f3035750a140552ac301456df4087e"} Oct 31 04:26:38.005792 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:38.005643 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ceba51-6c8fc9774c-vk52b" Oct 31 04:26:38.015226 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:38.015019 4376 scope.go:122] "RemoveContainer" containerID="137eadc48924c8f862ed0e8bb4ad99217a5739ed1579e0b78d0f8cdfab1423d3" Oct 31 04:26:38.023834 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:38.023818 4376 scope.go:122] "RemoveContainer" containerID="b8ee493ca4d59dc205b420f6ff1185cc92bd561067da0ac75193342d13b2504a" Oct 31 04:26:38.024123 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:26:38.024102 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8ee493ca4d59dc205b420f6ff1185cc92bd561067da0ac75193342d13b2504a\": container with ID starting with b8ee493ca4d59dc205b420f6ff1185cc92bd561067da0ac75193342d13b2504a not found: ID does not exist" containerID="b8ee493ca4d59dc205b420f6ff1185cc92bd561067da0ac75193342d13b2504a" Oct 31 04:26:38.024183 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:38.024132 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8ee493ca4d59dc205b420f6ff1185cc92bd561067da0ac75193342d13b2504a"} err="failed to get container status \"b8ee493ca4d59dc205b420f6ff1185cc92bd561067da0ac75193342d13b2504a\": rpc error: code = NotFound desc = could not find container \"b8ee493ca4d59dc205b420f6ff1185cc92bd561067da0ac75193342d13b2504a\": container with ID starting with b8ee493ca4d59dc205b420f6ff1185cc92bd561067da0ac75193342d13b2504a not found: ID does not exist" Oct 31 04:26:38.024183 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:38.024149 4376 scope.go:122] "RemoveContainer" containerID="137eadc48924c8f862ed0e8bb4ad99217a5739ed1579e0b78d0f8cdfab1423d3" Oct 31 04:26:38.024519 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:26:38.024500 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"137eadc48924c8f862ed0e8bb4ad99217a5739ed1579e0b78d0f8cdfab1423d3\": container with ID starting with 137eadc48924c8f862ed0e8bb4ad99217a5739ed1579e0b78d0f8cdfab1423d3 not found: ID does not exist" containerID="137eadc48924c8f862ed0e8bb4ad99217a5739ed1579e0b78d0f8cdfab1423d3" Oct 31 04:26:38.024569 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:38.024525 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"137eadc48924c8f862ed0e8bb4ad99217a5739ed1579e0b78d0f8cdfab1423d3"} err="failed to get container status \"137eadc48924c8f862ed0e8bb4ad99217a5739ed1579e0b78d0f8cdfab1423d3\": rpc error: code = NotFound desc = could not find container \"137eadc48924c8f862ed0e8bb4ad99217a5739ed1579e0b78d0f8cdfab1423d3\": container with ID starting with 137eadc48924c8f862ed0e8bb4ad99217a5739ed1579e0b78d0f8cdfab1423d3 not found: ID does not exist" Oct 31 04:26:38.024569 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:38.024540 4376 scope.go:122] "RemoveContainer" containerID="f53d83918d3952eae679a0f9744e5d9ef0f06c2cb77f0b740c631e12b7464894" Oct 31 04:26:38.057366 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:38.057316 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ceba51-6c8fc9774c-vk52b"] Oct 31 04:26:38.062956 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:38.062936 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ceba51-6c8fc9774c-vk52b"] Oct 31 04:26:38.102107 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:38.102087 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-3430f6-openshift-default-595d4f6b8c-ltm6g"] Oct 31 04:26:38.107149 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:38.107128 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-3430f6-openshift-default-595d4f6b8c-ltm6g"] Oct 31 04:26:38.444003 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:38.443966 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ade7461-87d8-42ea-a721-512bae1f5f65" path="/var/lib/kubelet/pods/3ade7461-87d8-42ea-a721-512bae1f5f65/volumes" Oct 31 04:26:38.444436 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:38.444419 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c503874b-c276-4ecf-b767-952b44173d94" path="/var/lib/kubelet/pods/c503874b-c276-4ecf-b767-952b44173d94/volumes" Oct 31 04:26:40.999786 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:26:40.999741 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 04:26:42.223689 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:42.223657 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-ff43e3-6cc76d6dc7-ndh7f"] Oct 31 04:26:42.224211 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:42.224197 4376 cpu_manager.go:414] "containerMap: removing container" podUID="3ade7461-87d8-42ea-a721-512bae1f5f65" containerName="istio-proxy" Oct 31 04:26:42.224262 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:42.224211 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ade7461-87d8-42ea-a721-512bae1f5f65" containerName="istio-proxy" Oct 31 04:26:42.224262 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:42.224235 4376 cpu_manager.go:414] "containerMap: removing container" podUID="3ade7461-87d8-42ea-a721-512bae1f5f65" containerName="istio-proxy" Oct 31 04:26:42.224262 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:42.224240 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ade7461-87d8-42ea-a721-512bae1f5f65" containerName="istio-proxy" Oct 31 04:26:42.224262 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:42.224255 4376 cpu_manager.go:414] "containerMap: removing container" podUID="6a62e709-216f-42e4-ac81-0c6712e61f31" containerName="maas-api" Oct 31 04:26:42.224262 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:42.224260 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a62e709-216f-42e4-ac81-0c6712e61f31" containerName="maas-api" Oct 31 04:26:42.224441 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:42.224268 4376 cpu_manager.go:414] "containerMap: removing container" podUID="c503874b-c276-4ecf-b767-952b44173d94" containerName="payload-pre-processing" Oct 31 04:26:42.224441 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:42.224273 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c503874b-c276-4ecf-b767-952b44173d94" containerName="payload-pre-processing" Oct 31 04:26:42.224441 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:42.224285 4376 cpu_manager.go:414] "containerMap: removing container" podUID="38a822e5-1a32-4126-b19a-1264d09c66cc" containerName="payload-processing" Oct 31 04:26:42.224441 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:42.224290 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38a822e5-1a32-4126-b19a-1264d09c66cc" containerName="payload-processing" Oct 31 04:26:42.224441 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:42.224393 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ade7461-87d8-42ea-a721-512bae1f5f65" containerName="istio-proxy" Oct 31 04:26:42.224441 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:42.224404 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ade7461-87d8-42ea-a721-512bae1f5f65" containerName="istio-proxy" Oct 31 04:26:42.224441 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:42.224412 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a62e709-216f-42e4-ac81-0c6712e61f31" containerName="maas-api" Oct 31 04:26:42.224441 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:42.224418 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="c503874b-c276-4ecf-b767-952b44173d94" containerName="payload-pre-processing" Oct 31 04:26:42.224441 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:42.224425 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="38a822e5-1a32-4126-b19a-1264d09c66cc" containerName="payload-processing" Oct 31 04:26:42.229189 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:42.229166 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ff43e3-6cc76d6dc7-ndh7f" Oct 31 04:26:42.247838 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:42.247812 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-ff43e3-6cc76d6dc7-ndh7f"] Oct 31 04:26:42.255965 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:42.255932 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-ff43e3-serving-cert\"" Oct 31 04:26:42.268899 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:42.268879 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab191222-92d2-4794-8317-e2ce18b8dcc8-maas-api-tls\") pod \"maas-api-e2e-ipp-a-ff43e3-6cc76d6dc7-ndh7f\" (UID: \"ab191222-92d2-4794-8317-e2ce18b8dcc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ff43e3-6cc76d6dc7-ndh7f" Oct 31 04:26:42.269018 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:42.268918 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpw4c\" (UniqueName: \"kubernetes.io/projected/ab191222-92d2-4794-8317-e2ce18b8dcc8-kube-api-access-kpw4c\") pod \"maas-api-e2e-ipp-a-ff43e3-6cc76d6dc7-ndh7f\" (UID: \"ab191222-92d2-4794-8317-e2ce18b8dcc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ff43e3-6cc76d6dc7-ndh7f" Oct 31 04:26:42.269195 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:42.269175 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab191222-92d2-4794-8317-e2ce18b8dcc8-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-ff43e3-6cc76d6dc7-ndh7f\" (UID: \"ab191222-92d2-4794-8317-e2ce18b8dcc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ff43e3-6cc76d6dc7-ndh7f" Oct 31 04:26:42.269264 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:42.269214 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab191222-92d2-4794-8317-e2ce18b8dcc8-metrics-certs\") pod \"maas-api-e2e-ipp-a-ff43e3-6cc76d6dc7-ndh7f\" (UID: \"ab191222-92d2-4794-8317-e2ce18b8dcc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ff43e3-6cc76d6dc7-ndh7f" Oct 31 04:26:42.370687 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:42.370658 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab191222-92d2-4794-8317-e2ce18b8dcc8-maas-api-tls\") pod \"maas-api-e2e-ipp-a-ff43e3-6cc76d6dc7-ndh7f\" (UID: \"ab191222-92d2-4794-8317-e2ce18b8dcc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ff43e3-6cc76d6dc7-ndh7f" Oct 31 04:26:42.370791 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:42.370701 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpw4c\" (UniqueName: \"kubernetes.io/projected/ab191222-92d2-4794-8317-e2ce18b8dcc8-kube-api-access-kpw4c\") pod \"maas-api-e2e-ipp-a-ff43e3-6cc76d6dc7-ndh7f\" (UID: \"ab191222-92d2-4794-8317-e2ce18b8dcc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ff43e3-6cc76d6dc7-ndh7f" Oct 31 04:26:42.370845 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:26:42.370808 4376 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-ff43e3-serving-cert: secret "maas-api-e2e-ipp-a-ff43e3-serving-cert" not found Oct 31 04:26:42.370884 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:26:42.370873 4376 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab191222-92d2-4794-8317-e2ce18b8dcc8-maas-api-tls podName:ab191222-92d2-4794-8317-e2ce18b8dcc8 nodeName:}" failed. No retries permitted until 2026-10-31 04:26:42.870852429 +0000 UTC m=+1350.585187039 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ab191222-92d2-4794-8317-e2ce18b8dcc8-maas-api-tls") pod "maas-api-e2e-ipp-a-ff43e3-6cc76d6dc7-ndh7f" (UID: "ab191222-92d2-4794-8317-e2ce18b8dcc8") : secret "maas-api-e2e-ipp-a-ff43e3-serving-cert" not found Oct 31 04:26:42.370923 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:42.370891 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab191222-92d2-4794-8317-e2ce18b8dcc8-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-ff43e3-6cc76d6dc7-ndh7f\" (UID: \"ab191222-92d2-4794-8317-e2ce18b8dcc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ff43e3-6cc76d6dc7-ndh7f" Oct 31 04:26:42.370956 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:42.370918 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab191222-92d2-4794-8317-e2ce18b8dcc8-metrics-certs\") pod \"maas-api-e2e-ipp-a-ff43e3-6cc76d6dc7-ndh7f\" (UID: \"ab191222-92d2-4794-8317-e2ce18b8dcc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ff43e3-6cc76d6dc7-ndh7f" Oct 31 04:26:42.371551 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:42.371533 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab191222-92d2-4794-8317-e2ce18b8dcc8-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-ff43e3-6cc76d6dc7-ndh7f\" (UID: \"ab191222-92d2-4794-8317-e2ce18b8dcc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ff43e3-6cc76d6dc7-ndh7f" Oct 31 04:26:42.373513 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:42.373492 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab191222-92d2-4794-8317-e2ce18b8dcc8-metrics-certs\") pod \"maas-api-e2e-ipp-a-ff43e3-6cc76d6dc7-ndh7f\" (UID: \"ab191222-92d2-4794-8317-e2ce18b8dcc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ff43e3-6cc76d6dc7-ndh7f" Oct 31 04:26:42.394626 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:42.394605 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpw4c\" (UniqueName: \"kubernetes.io/projected/ab191222-92d2-4794-8317-e2ce18b8dcc8-kube-api-access-kpw4c\") pod \"maas-api-e2e-ipp-a-ff43e3-6cc76d6dc7-ndh7f\" (UID: \"ab191222-92d2-4794-8317-e2ce18b8dcc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ff43e3-6cc76d6dc7-ndh7f" Oct 31 04:26:42.879818 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:42.879783 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab191222-92d2-4794-8317-e2ce18b8dcc8-maas-api-tls\") pod \"maas-api-e2e-ipp-a-ff43e3-6cc76d6dc7-ndh7f\" (UID: \"ab191222-92d2-4794-8317-e2ce18b8dcc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ff43e3-6cc76d6dc7-ndh7f" Oct 31 04:26:42.882104 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:42.882079 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab191222-92d2-4794-8317-e2ce18b8dcc8-maas-api-tls\") pod \"maas-api-e2e-ipp-a-ff43e3-6cc76d6dc7-ndh7f\" (UID: \"ab191222-92d2-4794-8317-e2ce18b8dcc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ff43e3-6cc76d6dc7-ndh7f" Oct 31 04:26:43.049585 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:43.049554 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 04:26:43.143278 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:43.143183 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ff43e3-6cc76d6dc7-ndh7f" Oct 31 04:26:43.473920 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:43.473692 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-ff43e3-6cc76d6dc7-ndh7f"] Oct 31 04:26:43.502213 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:43.502172 4376 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 04:26:44.063576 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:44.063548 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ff43e3-6cc76d6dc7-ndh7f" event={"ID":"ab191222-92d2-4794-8317-e2ce18b8dcc8","Type":"ContainerStarted","Data":"bf56b40a2e3240ee92e7962573d200431013d3ab93d5478f4a73656d157c93ee"} Oct 31 04:26:45.072924 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:45.072885 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ff43e3-6cc76d6dc7-ndh7f" event={"ID":"ab191222-92d2-4794-8317-e2ce18b8dcc8","Type":"ContainerStarted","Data":"6fb530781c01089ebb4f2e166275f21635f6375bc13ac0183b36c77225ad1d4f"} Oct 31 04:26:45.073398 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:45.072977 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ff43e3-6cc76d6dc7-ndh7f" Oct 31 04:26:45.128393 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:45.128342 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ff43e3-6cc76d6dc7-ndh7f" podStartSLOduration=2.596473691 podStartE2EDuration="3.128313841s" totalImagesPullingTime="531.84015ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:26:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:26:45.125186068 +0000 UTC m=+1352.839520684" watchObservedRunningTime="2026-10-31 04:26:45.128313841 +0000 UTC m=+1352.842648455" Oct 31 04:26:45.908073 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:45.908041 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7b4185-cfcb87d66-l8fzb"] Oct 31 04:26:45.908293 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:45.908271 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-7b4185-cfcb87d66-l8fzb" podUID="0f6ed41c-02f0-44a1-9055-0a95c3321143" containerName="payload-processing" containerID="cri-o://3deeb73675a951250d8d81dabea0dd656da1396cd9987e20d2e4b6805c20afd5" gracePeriod=30 Oct 31 04:26:45.929469 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:45.929208 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7b4185-7c659d956c-jllqw"] Oct 31 04:26:45.929469 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:45.929443 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7b4185-7c659d956c-jllqw" podUID="0d8c3260-faea-47ce-aebf-3bad9dc8fcf8" containerName="payload-pre-processing" containerID="cri-o://8fe3fbbe94798b3be7487758d7e213d08b1d4cb87ad2790df1b7bb95ef3725ad" gracePeriod=30 Oct 31 04:26:46.083888 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:46.083849 4376 generic.go:356] "Generic (PLEG): container finished" podID="0d8c3260-faea-47ce-aebf-3bad9dc8fcf8" containerID="8fe3fbbe94798b3be7487758d7e213d08b1d4cb87ad2790df1b7bb95ef3725ad" exitCode=0 Oct 31 04:26:46.084488 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:46.084023 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7b4185-7c659d956c-jllqw" event={"ID":"0d8c3260-faea-47ce-aebf-3bad9dc8fcf8","Type":"ContainerDied","Data":"8fe3fbbe94798b3be7487758d7e213d08b1d4cb87ad2790df1b7bb95ef3725ad"} Oct 31 04:26:46.086413 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:46.086385 4376 generic.go:356] "Generic (PLEG): container finished" podID="0f6ed41c-02f0-44a1-9055-0a95c3321143" containerID="3deeb73675a951250d8d81dabea0dd656da1396cd9987e20d2e4b6805c20afd5" exitCode=0 Oct 31 04:26:46.087316 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:46.087289 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-7b4185-cfcb87d66-l8fzb" event={"ID":"0f6ed41c-02f0-44a1-9055-0a95c3321143","Type":"ContainerDied","Data":"3deeb73675a951250d8d81dabea0dd656da1396cd9987e20d2e4b6805c20afd5"} Oct 31 04:26:46.308354 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:46.302524 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-7b4185-cfcb87d66-l8fzb" Oct 31 04:26:46.357352 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:46.355568 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7b4185-7c659d956c-jllqw" Oct 31 04:26:46.474398 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:46.462418 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f6ed41c-02f0-44a1-9055-0a95c3321143-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f6ed41c-02f0-44a1-9055-0a95c3321143-plugins-config-volume\") pod \"0f6ed41c-02f0-44a1-9055-0a95c3321143\" (UID: \"0f6ed41c-02f0-44a1-9055-0a95c3321143\") " Oct 31 04:26:46.474398 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:46.462640 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f6ed41c-02f0-44a1-9055-0a95c3321143-kube-api-access-b26cw\" (UniqueName: \"kubernetes.io/projected/0f6ed41c-02f0-44a1-9055-0a95c3321143-kube-api-access-b26cw\") pod \"0f6ed41c-02f0-44a1-9055-0a95c3321143\" (UID: \"0f6ed41c-02f0-44a1-9055-0a95c3321143\") " Oct 31 04:26:46.474398 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:46.462753 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d8c3260-faea-47ce-aebf-3bad9dc8fcf8-kube-api-access-kgjgx\" (UniqueName: \"kubernetes.io/projected/0d8c3260-faea-47ce-aebf-3bad9dc8fcf8-kube-api-access-kgjgx\") pod \"0d8c3260-faea-47ce-aebf-3bad9dc8fcf8\" (UID: \"0d8c3260-faea-47ce-aebf-3bad9dc8fcf8\") " Oct 31 04:26:46.474398 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:46.462836 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f6ed41c-02f0-44a1-9055-0a95c3321143-plugins-config-volume" pod "0f6ed41c-02f0-44a1-9055-0a95c3321143" (UID: "0f6ed41c-02f0-44a1-9055-0a95c3321143"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:26:46.474398 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:46.462848 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d8c3260-faea-47ce-aebf-3bad9dc8fcf8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d8c3260-faea-47ce-aebf-3bad9dc8fcf8-plugins-config-volume\") pod \"0d8c3260-faea-47ce-aebf-3bad9dc8fcf8\" (UID: \"0d8c3260-faea-47ce-aebf-3bad9dc8fcf8\") " Oct 31 04:26:46.474398 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:46.463518 4376 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f6ed41c-02f0-44a1-9055-0a95c3321143-plugins-config-volume\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:26:46.474398 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:46.463785 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d8c3260-faea-47ce-aebf-3bad9dc8fcf8-plugins-config-volume" pod "0d8c3260-faea-47ce-aebf-3bad9dc8fcf8" (UID: "0d8c3260-faea-47ce-aebf-3bad9dc8fcf8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:26:46.474398 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:46.473568 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d8c3260-faea-47ce-aebf-3bad9dc8fcf8-kube-api-access-kgjgx" pod "0d8c3260-faea-47ce-aebf-3bad9dc8fcf8" (UID: "0d8c3260-faea-47ce-aebf-3bad9dc8fcf8"). InnerVolumeSpecName "kube-api-access-kgjgx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:26:46.474398 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:46.473622 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f6ed41c-02f0-44a1-9055-0a95c3321143-kube-api-access-b26cw" pod "0f6ed41c-02f0-44a1-9055-0a95c3321143" (UID: "0f6ed41c-02f0-44a1-9055-0a95c3321143"). InnerVolumeSpecName "kube-api-access-b26cw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:26:46.565000 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:46.564970 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b26cw\" (UniqueName: \"kubernetes.io/projected/0f6ed41c-02f0-44a1-9055-0a95c3321143-kube-api-access-b26cw\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:26:46.565000 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:46.564999 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kgjgx\" (UniqueName: \"kubernetes.io/projected/0d8c3260-faea-47ce-aebf-3bad9dc8fcf8-kube-api-access-kgjgx\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:26:46.565186 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:46.565011 4376 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d8c3260-faea-47ce-aebf-3bad9dc8fcf8-plugins-config-volume\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:26:47.094945 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:47.094918 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7b4185-7c659d956c-jllqw" Oct 31 04:26:47.095492 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:47.094948 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7b4185-7c659d956c-jllqw" event={"ID":"0d8c3260-faea-47ce-aebf-3bad9dc8fcf8","Type":"ContainerDied","Data":"a34cf8b2a4bb82ac5e540af60a07214f9d814e87ab1054e438903c8df8c4226f"} Oct 31 04:26:47.095492 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:47.095033 4376 scope.go:122] "RemoveContainer" containerID="8fe3fbbe94798b3be7487758d7e213d08b1d4cb87ad2790df1b7bb95ef3725ad" Oct 31 04:26:47.096422 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:47.096405 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-7b4185-cfcb87d66-l8fzb" Oct 31 04:26:47.096526 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:47.096418 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-7b4185-cfcb87d66-l8fzb" event={"ID":"0f6ed41c-02f0-44a1-9055-0a95c3321143","Type":"ContainerDied","Data":"c04bb90d07fd0994ae74fff5d2265cbbf8bbcbfcfde9be6fbbad63640bb490dc"} Oct 31 04:26:47.105417 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:47.105398 4376 scope.go:122] "RemoveContainer" containerID="3deeb73675a951250d8d81dabea0dd656da1396cd9987e20d2e4b6805c20afd5" Oct 31 04:26:47.148648 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:47.148625 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7b4185-cfcb87d66-l8fzb"] Oct 31 04:26:47.160904 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:47.160879 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7b4185-cfcb87d66-l8fzb"] Oct 31 04:26:47.187049 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:47.187001 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7b4185-7c659d956c-jllqw"] Oct 31 04:26:47.189752 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:47.189733 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7b4185-7c659d956c-jllqw"] Oct 31 04:26:48.445016 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:48.444983 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d8c3260-faea-47ce-aebf-3bad9dc8fcf8" path="/var/lib/kubelet/pods/0d8c3260-faea-47ce-aebf-3bad9dc8fcf8/volumes" Oct 31 04:26:48.445601 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:48.445581 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f6ed41c-02f0-44a1-9055-0a95c3321143" path="/var/lib/kubelet/pods/0f6ed41c-02f0-44a1-9055-0a95c3321143/volumes" Oct 31 04:26:51.092180 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:51.092152 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ff43e3-6cc76d6dc7-ndh7f" Oct 31 04:26:54.662488 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:54.662459 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a9b573-6c5c49445d-j6qkq"] Oct 31 04:26:54.662947 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:54.662654 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-a9b573-6c5c49445d-j6qkq" podUID="ab9e79a2-c585-4d4e-a26d-9422cf1138ea" containerName="payload-processing" containerID="cri-o://8181be2463a1bfa52eba3d6fbb2e6b77441d142e33c50a4dcdd857b16d1d9786" gracePeriod=30 Oct 31 04:26:55.107368 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:55.106880 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a9b573-6c5c49445d-j6qkq" Oct 31 04:26:55.204402 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:55.202909 4376 generic.go:356] "Generic (PLEG): container finished" podID="ab9e79a2-c585-4d4e-a26d-9422cf1138ea" containerID="8181be2463a1bfa52eba3d6fbb2e6b77441d142e33c50a4dcdd857b16d1d9786" exitCode=0 Oct 31 04:26:55.204402 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:55.202986 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a9b573-6c5c49445d-j6qkq" event={"ID":"ab9e79a2-c585-4d4e-a26d-9422cf1138ea","Type":"ContainerDied","Data":"8181be2463a1bfa52eba3d6fbb2e6b77441d142e33c50a4dcdd857b16d1d9786"} Oct 31 04:26:55.204402 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:55.203089 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a9b573-6c5c49445d-j6qkq" event={"ID":"ab9e79a2-c585-4d4e-a26d-9422cf1138ea","Type":"ContainerDied","Data":"8a2a2164fb30f2d6eebdca849d1c5092b798d104bbd7ca528038724e08a37019"} Oct 31 04:26:55.204402 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:55.203138 4376 scope.go:122] "RemoveContainer" containerID="8181be2463a1bfa52eba3d6fbb2e6b77441d142e33c50a4dcdd857b16d1d9786" Oct 31 04:26:55.204402 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:55.203375 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a9b573-6c5c49445d-j6qkq" Oct 31 04:26:55.286358 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:55.269411 4376 scope.go:122] "RemoveContainer" containerID="8181be2463a1bfa52eba3d6fbb2e6b77441d142e33c50a4dcdd857b16d1d9786" Oct 31 04:26:55.286358 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:26:55.284827 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8181be2463a1bfa52eba3d6fbb2e6b77441d142e33c50a4dcdd857b16d1d9786\": container with ID starting with 8181be2463a1bfa52eba3d6fbb2e6b77441d142e33c50a4dcdd857b16d1d9786 not found: ID does not exist" containerID="8181be2463a1bfa52eba3d6fbb2e6b77441d142e33c50a4dcdd857b16d1d9786" Oct 31 04:26:55.286358 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:55.284885 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8181be2463a1bfa52eba3d6fbb2e6b77441d142e33c50a4dcdd857b16d1d9786"} err="failed to get container status \"8181be2463a1bfa52eba3d6fbb2e6b77441d142e33c50a4dcdd857b16d1d9786\": rpc error: code = NotFound desc = could not find container \"8181be2463a1bfa52eba3d6fbb2e6b77441d142e33c50a4dcdd857b16d1d9786\": container with ID starting with 8181be2463a1bfa52eba3d6fbb2e6b77441d142e33c50a4dcdd857b16d1d9786 not found: ID does not exist" Oct 31 04:26:55.308552 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:55.308514 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab9e79a2-c585-4d4e-a26d-9422cf1138ea-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab9e79a2-c585-4d4e-a26d-9422cf1138ea-plugins-config-volume\") pod \"ab9e79a2-c585-4d4e-a26d-9422cf1138ea\" (UID: \"ab9e79a2-c585-4d4e-a26d-9422cf1138ea\") " Oct 31 04:26:55.308713 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:55.308563 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab9e79a2-c585-4d4e-a26d-9422cf1138ea-kube-api-access-f6qg2\" (UniqueName: \"kubernetes.io/projected/ab9e79a2-c585-4d4e-a26d-9422cf1138ea-kube-api-access-f6qg2\") pod \"ab9e79a2-c585-4d4e-a26d-9422cf1138ea\" (UID: \"ab9e79a2-c585-4d4e-a26d-9422cf1138ea\") " Oct 31 04:26:55.309008 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:55.308979 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab9e79a2-c585-4d4e-a26d-9422cf1138ea-plugins-config-volume" pod "ab9e79a2-c585-4d4e-a26d-9422cf1138ea" (UID: "ab9e79a2-c585-4d4e-a26d-9422cf1138ea"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:26:55.309324 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:55.309298 4376 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab9e79a2-c585-4d4e-a26d-9422cf1138ea-plugins-config-volume\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:26:55.310806 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:55.310778 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab9e79a2-c585-4d4e-a26d-9422cf1138ea-kube-api-access-f6qg2" pod "ab9e79a2-c585-4d4e-a26d-9422cf1138ea" (UID: "ab9e79a2-c585-4d4e-a26d-9422cf1138ea"). InnerVolumeSpecName "kube-api-access-f6qg2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:26:55.410306 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:55.410277 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6qg2\" (UniqueName: \"kubernetes.io/projected/ab9e79a2-c585-4d4e-a26d-9422cf1138ea-kube-api-access-f6qg2\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:26:55.554357 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:55.554317 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a9b573-6c5c49445d-j6qkq"] Oct 31 04:26:55.565900 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:55.565879 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a9b573-6c5c49445d-j6qkq"] Oct 31 04:26:56.504355 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:26:56.501118 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab9e79a2-c585-4d4e-a26d-9422cf1138ea" path="/var/lib/kubelet/pods/ab9e79a2-c585-4d4e-a26d-9422cf1138ea/volumes" Oct 31 04:27:05.694576 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:05.694543 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ff43e3-576df466d7-4pqsr"] Oct 31 04:27:07.405260 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:07.362401 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-c6859a-59d59469c9-4r9lt"] Oct 31 04:27:07.405260 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:07.363239 4376 cpu_manager.go:414] "containerMap: removing container" podUID="0d8c3260-faea-47ce-aebf-3bad9dc8fcf8" containerName="payload-pre-processing" Oct 31 04:27:07.405260 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:07.363256 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d8c3260-faea-47ce-aebf-3bad9dc8fcf8" containerName="payload-pre-processing" Oct 31 04:27:07.405260 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:07.363302 4376 cpu_manager.go:414] "containerMap: removing container" podUID="ab9e79a2-c585-4d4e-a26d-9422cf1138ea" containerName="payload-processing" Oct 31 04:27:07.405260 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:07.363309 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab9e79a2-c585-4d4e-a26d-9422cf1138ea" containerName="payload-processing" Oct 31 04:27:07.406773 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:07.363324 4376 cpu_manager.go:414] "containerMap: removing container" podUID="0f6ed41c-02f0-44a1-9055-0a95c3321143" containerName="payload-processing" Oct 31 04:27:07.406773 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:07.406670 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f6ed41c-02f0-44a1-9055-0a95c3321143" containerName="payload-processing" Oct 31 04:27:07.407277 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:07.407213 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d8c3260-faea-47ce-aebf-3bad9dc8fcf8" containerName="payload-pre-processing" Oct 31 04:27:07.407277 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:07.407246 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab9e79a2-c585-4d4e-a26d-9422cf1138ea" containerName="payload-processing" Oct 31 04:27:07.407562 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:07.407489 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f6ed41c-02f0-44a1-9055-0a95c3321143" containerName="payload-processing" Oct 31 04:27:07.423857 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:07.422574 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-c6859a-59d59469c9-4r9lt"] Oct 31 04:27:07.423857 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:07.422706 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c6859a-59d59469c9-4r9lt" Oct 31 04:27:07.439943 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:07.439918 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-c6859a\"" Oct 31 04:27:07.504406 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:07.477744 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-c6859a-dockercfg-nbrk8\"" Oct 31 04:27:07.522491 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:07.506798 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa104be0-8c9b-4345-bcaf-a0cc7d0bebb3-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-c6859a-59d59469c9-4r9lt\" (UID: \"aa104be0-8c9b-4345-bcaf-a0cc7d0bebb3\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c6859a-59d59469c9-4r9lt" Oct 31 04:27:07.522960 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:07.522931 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqshz\" (UniqueName: \"kubernetes.io/projected/aa104be0-8c9b-4345-bcaf-a0cc7d0bebb3-kube-api-access-lqshz\") pod \"payload-pre-processing-e2e-praxis-a-c6859a-59d59469c9-4r9lt\" (UID: \"aa104be0-8c9b-4345-bcaf-a0cc7d0bebb3\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c6859a-59d59469c9-4r9lt" Oct 31 04:27:07.626351 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:07.624479 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa104be0-8c9b-4345-bcaf-a0cc7d0bebb3-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-c6859a-59d59469c9-4r9lt\" (UID: \"aa104be0-8c9b-4345-bcaf-a0cc7d0bebb3\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c6859a-59d59469c9-4r9lt" Oct 31 04:27:07.626351 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:07.624655 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqshz\" (UniqueName: \"kubernetes.io/projected/aa104be0-8c9b-4345-bcaf-a0cc7d0bebb3-kube-api-access-lqshz\") pod \"payload-pre-processing-e2e-praxis-a-c6859a-59d59469c9-4r9lt\" (UID: \"aa104be0-8c9b-4345-bcaf-a0cc7d0bebb3\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c6859a-59d59469c9-4r9lt" Oct 31 04:27:07.634541 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:07.634507 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa104be0-8c9b-4345-bcaf-a0cc7d0bebb3-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-c6859a-59d59469c9-4r9lt\" (UID: \"aa104be0-8c9b-4345-bcaf-a0cc7d0bebb3\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c6859a-59d59469c9-4r9lt" Oct 31 04:27:07.650529 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:07.650498 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-d1d441-openshift-default-c79997b97-84mqf"] Oct 31 04:27:07.662098 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:07.661678 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-d1d441-openshift-default-c79997b97-84mqf"] Oct 31 04:27:07.662098 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:07.661798 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-d1d441-openshift-default-c79997b97-84mqf" Oct 31 04:27:07.662958 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:07.662929 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqshz\" (UniqueName: \"kubernetes.io/projected/aa104be0-8c9b-4345-bcaf-a0cc7d0bebb3-kube-api-access-lqshz\") pod \"payload-pre-processing-e2e-praxis-a-c6859a-59d59469c9-4r9lt\" (UID: \"aa104be0-8c9b-4345-bcaf-a0cc7d0bebb3\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c6859a-59d59469c9-4r9lt" Oct 31 04:27:07.709154 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:07.705532 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-d1d441-openshift-default-dockercfg-vd7l5\"" Oct 31 04:27:07.725865 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:07.725829 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9240e5d-789e-4ebf-8cd3-e7807036e8a7-istio-podinfo\") pod \"e2e-praxis-a-d1d441-openshift-default-c79997b97-84mqf\" (UID: \"c9240e5d-789e-4ebf-8cd3-e7807036e8a7\") " pod="openshift-ingress/e2e-praxis-a-d1d441-openshift-default-c79997b97-84mqf" Oct 31 04:27:07.726059 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:07.725895 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9240e5d-789e-4ebf-8cd3-e7807036e8a7-istio-envoy\") pod \"e2e-praxis-a-d1d441-openshift-default-c79997b97-84mqf\" (UID: \"c9240e5d-789e-4ebf-8cd3-e7807036e8a7\") " pod="openshift-ingress/e2e-praxis-a-d1d441-openshift-default-c79997b97-84mqf" Oct 31 04:27:07.726059 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:07.725928 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9240e5d-789e-4ebf-8cd3-e7807036e8a7-istiod-ca-cert\") pod \"e2e-praxis-a-d1d441-openshift-default-c79997b97-84mqf\" (UID: \"c9240e5d-789e-4ebf-8cd3-e7807036e8a7\") " pod="openshift-ingress/e2e-praxis-a-d1d441-openshift-default-c79997b97-84mqf" Oct 31 04:27:07.726059 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:07.726038 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9240e5d-789e-4ebf-8cd3-e7807036e8a7-istio-data\") pod \"e2e-praxis-a-d1d441-openshift-default-c79997b97-84mqf\" (UID: \"c9240e5d-789e-4ebf-8cd3-e7807036e8a7\") " pod="openshift-ingress/e2e-praxis-a-d1d441-openshift-default-c79997b97-84mqf" Oct 31 04:27:07.726188 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:07.726087 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9240e5d-789e-4ebf-8cd3-e7807036e8a7-workload-socket\") pod \"e2e-praxis-a-d1d441-openshift-default-c79997b97-84mqf\" (UID: \"c9240e5d-789e-4ebf-8cd3-e7807036e8a7\") " pod="openshift-ingress/e2e-praxis-a-d1d441-openshift-default-c79997b97-84mqf" Oct 31 04:27:07.726188 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:07.726173 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9240e5d-789e-4ebf-8cd3-e7807036e8a7-istio-token\") pod \"e2e-praxis-a-d1d441-openshift-default-c79997b97-84mqf\" (UID: \"c9240e5d-789e-4ebf-8cd3-e7807036e8a7\") " pod="openshift-ingress/e2e-praxis-a-d1d441-openshift-default-c79997b97-84mqf" Oct 31 04:27:07.726288 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:07.726258 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9240e5d-789e-4ebf-8cd3-e7807036e8a7-workload-certs\") pod \"e2e-praxis-a-d1d441-openshift-default-c79997b97-84mqf\" (UID: \"c9240e5d-789e-4ebf-8cd3-e7807036e8a7\") " pod="openshift-ingress/e2e-praxis-a-d1d441-openshift-default-c79997b97-84mqf" Oct 31 04:27:07.726388 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:07.726368 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kx2qd\" (UniqueName: \"kubernetes.io/projected/c9240e5d-789e-4ebf-8cd3-e7807036e8a7-kube-api-access-kx2qd\") pod \"e2e-praxis-a-d1d441-openshift-default-c79997b97-84mqf\" (UID: \"c9240e5d-789e-4ebf-8cd3-e7807036e8a7\") " pod="openshift-ingress/e2e-praxis-a-d1d441-openshift-default-c79997b97-84mqf" Oct 31 04:27:07.726446 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:07.726412 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9240e5d-789e-4ebf-8cd3-e7807036e8a7-credential-socket\") pod \"e2e-praxis-a-d1d441-openshift-default-c79997b97-84mqf\" (UID: \"c9240e5d-789e-4ebf-8cd3-e7807036e8a7\") " pod="openshift-ingress/e2e-praxis-a-d1d441-openshift-default-c79997b97-84mqf" Oct 31 04:27:07.842437 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:07.817894 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c6859a-59d59469c9-4r9lt" Oct 31 04:27:07.843409 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:07.843212 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9240e5d-789e-4ebf-8cd3-e7807036e8a7-istio-data\") pod \"e2e-praxis-a-d1d441-openshift-default-c79997b97-84mqf\" (UID: \"c9240e5d-789e-4ebf-8cd3-e7807036e8a7\") " pod="openshift-ingress/e2e-praxis-a-d1d441-openshift-default-c79997b97-84mqf" Oct 31 04:27:07.888355 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:07.873768 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9240e5d-789e-4ebf-8cd3-e7807036e8a7-workload-socket\") pod \"e2e-praxis-a-d1d441-openshift-default-c79997b97-84mqf\" (UID: \"c9240e5d-789e-4ebf-8cd3-e7807036e8a7\") " pod="openshift-ingress/e2e-praxis-a-d1d441-openshift-default-c79997b97-84mqf" Oct 31 04:27:07.888355 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:07.873933 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9240e5d-789e-4ebf-8cd3-e7807036e8a7-istio-token\") pod \"e2e-praxis-a-d1d441-openshift-default-c79997b97-84mqf\" (UID: \"c9240e5d-789e-4ebf-8cd3-e7807036e8a7\") " pod="openshift-ingress/e2e-praxis-a-d1d441-openshift-default-c79997b97-84mqf" Oct 31 04:27:07.888355 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:07.874016 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9240e5d-789e-4ebf-8cd3-e7807036e8a7-workload-certs\") pod \"e2e-praxis-a-d1d441-openshift-default-c79997b97-84mqf\" (UID: \"c9240e5d-789e-4ebf-8cd3-e7807036e8a7\") " pod="openshift-ingress/e2e-praxis-a-d1d441-openshift-default-c79997b97-84mqf" Oct 31 04:27:07.888355 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:07.874117 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kx2qd\" (UniqueName: \"kubernetes.io/projected/c9240e5d-789e-4ebf-8cd3-e7807036e8a7-kube-api-access-kx2qd\") pod \"e2e-praxis-a-d1d441-openshift-default-c79997b97-84mqf\" (UID: \"c9240e5d-789e-4ebf-8cd3-e7807036e8a7\") " pod="openshift-ingress/e2e-praxis-a-d1d441-openshift-default-c79997b97-84mqf" Oct 31 04:27:07.888355 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:07.874174 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9240e5d-789e-4ebf-8cd3-e7807036e8a7-credential-socket\") pod \"e2e-praxis-a-d1d441-openshift-default-c79997b97-84mqf\" (UID: \"c9240e5d-789e-4ebf-8cd3-e7807036e8a7\") " pod="openshift-ingress/e2e-praxis-a-d1d441-openshift-default-c79997b97-84mqf" Oct 31 04:27:07.888355 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:07.874291 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9240e5d-789e-4ebf-8cd3-e7807036e8a7-istio-podinfo\") pod \"e2e-praxis-a-d1d441-openshift-default-c79997b97-84mqf\" (UID: \"c9240e5d-789e-4ebf-8cd3-e7807036e8a7\") " pod="openshift-ingress/e2e-praxis-a-d1d441-openshift-default-c79997b97-84mqf" Oct 31 04:27:07.888355 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:07.845803 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9240e5d-789e-4ebf-8cd3-e7807036e8a7-istio-data\") pod \"e2e-praxis-a-d1d441-openshift-default-c79997b97-84mqf\" (UID: \"c9240e5d-789e-4ebf-8cd3-e7807036e8a7\") " pod="openshift-ingress/e2e-praxis-a-d1d441-openshift-default-c79997b97-84mqf" Oct 31 04:27:07.888355 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:07.883285 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9240e5d-789e-4ebf-8cd3-e7807036e8a7-workload-socket\") pod \"e2e-praxis-a-d1d441-openshift-default-c79997b97-84mqf\" (UID: \"c9240e5d-789e-4ebf-8cd3-e7807036e8a7\") " pod="openshift-ingress/e2e-praxis-a-d1d441-openshift-default-c79997b97-84mqf" Oct 31 04:27:07.888355 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:07.883873 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9240e5d-789e-4ebf-8cd3-e7807036e8a7-workload-certs\") pod \"e2e-praxis-a-d1d441-openshift-default-c79997b97-84mqf\" (UID: \"c9240e5d-789e-4ebf-8cd3-e7807036e8a7\") " pod="openshift-ingress/e2e-praxis-a-d1d441-openshift-default-c79997b97-84mqf" Oct 31 04:27:07.888355 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:07.884472 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9240e5d-789e-4ebf-8cd3-e7807036e8a7-credential-socket\") pod \"e2e-praxis-a-d1d441-openshift-default-c79997b97-84mqf\" (UID: \"c9240e5d-789e-4ebf-8cd3-e7807036e8a7\") " pod="openshift-ingress/e2e-praxis-a-d1d441-openshift-default-c79997b97-84mqf" Oct 31 04:27:07.937590 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:07.919118 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9240e5d-789e-4ebf-8cd3-e7807036e8a7-istio-podinfo\") pod \"e2e-praxis-a-d1d441-openshift-default-c79997b97-84mqf\" (UID: \"c9240e5d-789e-4ebf-8cd3-e7807036e8a7\") " pod="openshift-ingress/e2e-praxis-a-d1d441-openshift-default-c79997b97-84mqf" Oct 31 04:27:07.937590 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:07.928360 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9240e5d-789e-4ebf-8cd3-e7807036e8a7-istio-envoy\") pod \"e2e-praxis-a-d1d441-openshift-default-c79997b97-84mqf\" (UID: \"c9240e5d-789e-4ebf-8cd3-e7807036e8a7\") " pod="openshift-ingress/e2e-praxis-a-d1d441-openshift-default-c79997b97-84mqf" Oct 31 04:27:07.937590 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:07.928417 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9240e5d-789e-4ebf-8cd3-e7807036e8a7-istiod-ca-cert\") pod \"e2e-praxis-a-d1d441-openshift-default-c79997b97-84mqf\" (UID: \"c9240e5d-789e-4ebf-8cd3-e7807036e8a7\") " pod="openshift-ingress/e2e-praxis-a-d1d441-openshift-default-c79997b97-84mqf" Oct 31 04:27:07.937590 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:07.929604 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9240e5d-789e-4ebf-8cd3-e7807036e8a7-istiod-ca-cert\") pod \"e2e-praxis-a-d1d441-openshift-default-c79997b97-84mqf\" (UID: \"c9240e5d-789e-4ebf-8cd3-e7807036e8a7\") " pod="openshift-ingress/e2e-praxis-a-d1d441-openshift-default-c79997b97-84mqf" Oct 31 04:27:07.944675 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:07.942323 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9240e5d-789e-4ebf-8cd3-e7807036e8a7-istio-envoy\") pod \"e2e-praxis-a-d1d441-openshift-default-c79997b97-84mqf\" (UID: \"c9240e5d-789e-4ebf-8cd3-e7807036e8a7\") " pod="openshift-ingress/e2e-praxis-a-d1d441-openshift-default-c79997b97-84mqf" Oct 31 04:27:08.026235 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:08.024124 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9240e5d-789e-4ebf-8cd3-e7807036e8a7-istio-token\") pod \"e2e-praxis-a-d1d441-openshift-default-c79997b97-84mqf\" (UID: \"c9240e5d-789e-4ebf-8cd3-e7807036e8a7\") " pod="openshift-ingress/e2e-praxis-a-d1d441-openshift-default-c79997b97-84mqf" Oct 31 04:27:08.030372 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:08.028045 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kx2qd\" (UniqueName: \"kubernetes.io/projected/c9240e5d-789e-4ebf-8cd3-e7807036e8a7-kube-api-access-kx2qd\") pod \"e2e-praxis-a-d1d441-openshift-default-c79997b97-84mqf\" (UID: \"c9240e5d-789e-4ebf-8cd3-e7807036e8a7\") " pod="openshift-ingress/e2e-praxis-a-d1d441-openshift-default-c79997b97-84mqf" Oct 31 04:27:08.284005 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:27:08.283970 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa104be0_8c9b_4345_bcaf_a0cc7d0bebb3.slice/crio-5d43580ef3d00390123c5ccb963740f494ad271be32e14804715e93276922279 WatchSource:0}: Error finding container 5d43580ef3d00390123c5ccb963740f494ad271be32e14804715e93276922279: Status 404 returned error can't find the container with id 5d43580ef3d00390123c5ccb963740f494ad271be32e14804715e93276922279 Oct 31 04:27:08.285916 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:08.285883 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-c6859a-59d59469c9-4r9lt"] Oct 31 04:27:08.294749 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:08.294720 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-d1d441-openshift-default-c79997b97-84mqf" Oct 31 04:27:08.452381 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:08.452316 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c6859a-59d59469c9-4r9lt" event={"ID":"aa104be0-8c9b-4345-bcaf-a0cc7d0bebb3","Type":"ContainerStarted","Data":"74b8f7d0151c90595af4fe1b255d0d750302623fbc5c8619e106883daeb50c19"} Oct 31 04:27:08.452765 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:08.452409 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c6859a-59d59469c9-4r9lt" event={"ID":"aa104be0-8c9b-4345-bcaf-a0cc7d0bebb3","Type":"ContainerStarted","Data":"5d43580ef3d00390123c5ccb963740f494ad271be32e14804715e93276922279"} Oct 31 04:27:08.452765 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:08.452542 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c6859a-59d59469c9-4r9lt" Oct 31 04:27:08.495759 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:08.495706 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c6859a-59d59469c9-4r9lt" podStartSLOduration=1.495694397 podStartE2EDuration="1.495694397s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:27:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:27:08.494662089 +0000 UTC m=+1376.208996731" watchObservedRunningTime="2026-10-31 04:27:08.495694397 +0000 UTC m=+1376.210029012" Oct 31 04:27:08.646535 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:08.640651 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-d1d441-openshift-default-c79997b97-84mqf"] Oct 31 04:27:08.669978 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:08.668717 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:27:08.669978 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:08.668851 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:27:08.669978 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:08.668904 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:27:09.460652 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:09.460615 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-d1d441-openshift-default-c79997b97-84mqf" event={"ID":"c9240e5d-789e-4ebf-8cd3-e7807036e8a7","Type":"ContainerStarted","Data":"256c492ec07aeb1473d776dac5053a9be1813e76d7b35ba430f04da83e72bb7b"} Oct 31 04:27:09.461063 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:09.460676 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-d1d441-openshift-default-c79997b97-84mqf" event={"ID":"c9240e5d-789e-4ebf-8cd3-e7807036e8a7","Type":"ContainerStarted","Data":"700364b4a230d6083e8da32331a7c22031ed8fbda65d9e399d713529212bfbaf"} Oct 31 04:27:09.532128 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:09.532074 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-d1d441-openshift-default-c79997b97-84mqf" podStartSLOduration=2.532061106 podStartE2EDuration="2.532061106s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:27:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:27:09.519238731 +0000 UTC m=+1377.233573349" watchObservedRunningTime="2026-10-31 04:27:09.532061106 +0000 UTC m=+1377.246395722" Oct 31 04:27:09.848072 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:09.847969 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-n6t9x"] Oct 31 04:27:09.848238 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:09.848196 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-n6t9x" podUID="31c92206-bebe-492e-be35-a008408e1e86" containerName="istio-proxy" containerID="cri-o://379565cdf4008e1cecc097ff692ae676392277493bbda4f557b86a7536232897" gracePeriod=30 Oct 31 04:27:09.851640 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:09.851610 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-k2jb2"] Oct 31 04:27:09.851989 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:09.851957 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-k2jb2" podUID="716d4a99-414f-449d-89c2-fa06e5751c60" containerName="istio-proxy" containerID="cri-o://45d080900be9400444f7c02487989b189b36ce89754d4a8a50599929415695a7" gracePeriod=30 Oct 31 04:27:10.295846 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:10.295810 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-d1d441-openshift-default-c79997b97-84mqf" Oct 31 04:27:10.301365 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:10.301323 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-d1d441-openshift-default-c79997b97-84mqf" Oct 31 04:27:10.524842 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:10.522908 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-d1d441-openshift-default-c79997b97-84mqf" Oct 31 04:27:10.524842 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:10.522982 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-d1d441-openshift-default-c79997b97-84mqf" Oct 31 04:27:10.695363 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:10.695310 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-ff43e3-576df466d7-4pqsr" podUID="151dc5b7-7589-4707-8586-8f70ab9b8d66" containerName="payload-processing" containerID="cri-o://8c854abddec1682d9dac4d9ac1cb368880ff0b5b4f1ae8cbf82cd73db1102616" gracePeriod=25 Oct 31 04:27:10.970899 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:10.970876 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-ff43e3-576df466d7-4pqsr" Oct 31 04:27:10.992792 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:10.992759 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/151dc5b7-7589-4707-8586-8f70ab9b8d66-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/151dc5b7-7589-4707-8586-8f70ab9b8d66-plugins-config-volume\") pod \"151dc5b7-7589-4707-8586-8f70ab9b8d66\" (UID: \"151dc5b7-7589-4707-8586-8f70ab9b8d66\") " Oct 31 04:27:10.992986 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:10.992932 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/151dc5b7-7589-4707-8586-8f70ab9b8d66-kube-api-access-v2cjg\" (UniqueName: \"kubernetes.io/projected/151dc5b7-7589-4707-8586-8f70ab9b8d66-kube-api-access-v2cjg\") pod \"151dc5b7-7589-4707-8586-8f70ab9b8d66\" (UID: \"151dc5b7-7589-4707-8586-8f70ab9b8d66\") " Oct 31 04:27:10.993202 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:10.993169 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/151dc5b7-7589-4707-8586-8f70ab9b8d66-plugins-config-volume" pod "151dc5b7-7589-4707-8586-8f70ab9b8d66" (UID: "151dc5b7-7589-4707-8586-8f70ab9b8d66"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:27:10.993604 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:10.993581 4376 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/151dc5b7-7589-4707-8586-8f70ab9b8d66-plugins-config-volume\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:27:10.995399 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:10.995367 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/151dc5b7-7589-4707-8586-8f70ab9b8d66-kube-api-access-v2cjg" pod "151dc5b7-7589-4707-8586-8f70ab9b8d66" (UID: "151dc5b7-7589-4707-8586-8f70ab9b8d66"). InnerVolumeSpecName "kube-api-access-v2cjg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:27:11.001007 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:27:11.000975 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 04:27:11.112350 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:11.108348 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2cjg\" (UniqueName: \"kubernetes.io/projected/151dc5b7-7589-4707-8586-8f70ab9b8d66-kube-api-access-v2cjg\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:27:11.523508 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:11.519300 4376 generic.go:356] "Generic (PLEG): container finished" podID="151dc5b7-7589-4707-8586-8f70ab9b8d66" containerID="8c854abddec1682d9dac4d9ac1cb368880ff0b5b4f1ae8cbf82cd73db1102616" exitCode=0 Oct 31 04:27:11.523508 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:11.519965 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-ff43e3-576df466d7-4pqsr" event={"ID":"151dc5b7-7589-4707-8586-8f70ab9b8d66","Type":"ContainerDied","Data":"8c854abddec1682d9dac4d9ac1cb368880ff0b5b4f1ae8cbf82cd73db1102616"} Oct 31 04:27:11.523508 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:11.520024 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-ff43e3-576df466d7-4pqsr" Oct 31 04:27:11.523508 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:11.520104 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-ff43e3-576df466d7-4pqsr" event={"ID":"151dc5b7-7589-4707-8586-8f70ab9b8d66","Type":"ContainerDied","Data":"4077cab580769dc16fd0c4abe0ca0ae37aab3b7b16052f1d320509c1630a8e45"} Oct 31 04:27:11.523508 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:11.520148 4376 scope.go:122] "RemoveContainer" containerID="8c854abddec1682d9dac4d9ac1cb368880ff0b5b4f1ae8cbf82cd73db1102616" Oct 31 04:27:11.537093 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:11.537074 4376 scope.go:122] "RemoveContainer" containerID="8c854abddec1682d9dac4d9ac1cb368880ff0b5b4f1ae8cbf82cd73db1102616" Oct 31 04:27:11.537451 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:27:11.537362 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c854abddec1682d9dac4d9ac1cb368880ff0b5b4f1ae8cbf82cd73db1102616\": container with ID starting with 8c854abddec1682d9dac4d9ac1cb368880ff0b5b4f1ae8cbf82cd73db1102616 not found: ID does not exist" containerID="8c854abddec1682d9dac4d9ac1cb368880ff0b5b4f1ae8cbf82cd73db1102616" Oct 31 04:27:11.537451 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:11.537384 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c854abddec1682d9dac4d9ac1cb368880ff0b5b4f1ae8cbf82cd73db1102616"} err="failed to get container status \"8c854abddec1682d9dac4d9ac1cb368880ff0b5b4f1ae8cbf82cd73db1102616\": rpc error: code = NotFound desc = could not find container \"8c854abddec1682d9dac4d9ac1cb368880ff0b5b4f1ae8cbf82cd73db1102616\": container with ID starting with 8c854abddec1682d9dac4d9ac1cb368880ff0b5b4f1ae8cbf82cd73db1102616 not found: ID does not exist" Oct 31 04:27:11.575709 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:11.575677 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ff43e3-576df466d7-4pqsr"] Oct 31 04:27:11.586091 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:11.586065 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ff43e3-576df466d7-4pqsr"] Oct 31 04:27:12.444417 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:12.444386 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="151dc5b7-7589-4707-8586-8f70ab9b8d66" path="/var/lib/kubelet/pods/151dc5b7-7589-4707-8586-8f70ab9b8d66/volumes" Oct 31 04:27:15.224118 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:15.224092 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-n6t9x" Oct 31 04:27:15.264303 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:15.264278 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-k2jb2" Oct 31 04:27:15.325844 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:15.325777 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/716d4a99-414f-449d-89c2-fa06e5751c60-istio-token\" (UniqueName: \"kubernetes.io/projected/716d4a99-414f-449d-89c2-fa06e5751c60-istio-token\") pod \"716d4a99-414f-449d-89c2-fa06e5751c60\" (UID: \"716d4a99-414f-449d-89c2-fa06e5751c60\") " Oct 31 04:27:15.336871 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:15.336431 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/716d4a99-414f-449d-89c2-fa06e5751c60-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/716d4a99-414f-449d-89c2-fa06e5751c60-credential-socket\") pod \"716d4a99-414f-449d-89c2-fa06e5751c60\" (UID: \"716d4a99-414f-449d-89c2-fa06e5751c60\") " Oct 31 04:27:15.336871 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:15.336558 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31c92206-bebe-492e-be35-a008408e1e86-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31c92206-bebe-492e-be35-a008408e1e86-workload-socket\") pod \"31c92206-bebe-492e-be35-a008408e1e86\" (UID: \"31c92206-bebe-492e-be35-a008408e1e86\") " Oct 31 04:27:15.336871 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:15.336608 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31c92206-bebe-492e-be35-a008408e1e86-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31c92206-bebe-492e-be35-a008408e1e86-istiod-ca-cert\") pod \"31c92206-bebe-492e-be35-a008408e1e86\" (UID: \"31c92206-bebe-492e-be35-a008408e1e86\") " Oct 31 04:27:15.336871 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:15.336619 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/716d4a99-414f-449d-89c2-fa06e5751c60-credential-socket" pod "716d4a99-414f-449d-89c2-fa06e5751c60" (UID: "716d4a99-414f-449d-89c2-fa06e5751c60"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:27:15.336871 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:15.336651 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/716d4a99-414f-449d-89c2-fa06e5751c60-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/716d4a99-414f-449d-89c2-fa06e5751c60-workload-certs\") pod \"716d4a99-414f-449d-89c2-fa06e5751c60\" (UID: \"716d4a99-414f-449d-89c2-fa06e5751c60\") " Oct 31 04:27:15.336871 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:15.336745 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/716d4a99-414f-449d-89c2-fa06e5751c60-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/716d4a99-414f-449d-89c2-fa06e5751c60-workload-socket\") pod \"716d4a99-414f-449d-89c2-fa06e5751c60\" (UID: \"716d4a99-414f-449d-89c2-fa06e5751c60\") " Oct 31 04:27:15.336871 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:15.336777 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31c92206-bebe-492e-be35-a008408e1e86-istio-token\" (UniqueName: \"kubernetes.io/projected/31c92206-bebe-492e-be35-a008408e1e86-istio-token\") pod \"31c92206-bebe-492e-be35-a008408e1e86\" (UID: \"31c92206-bebe-492e-be35-a008408e1e86\") " Oct 31 04:27:15.336871 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:15.336865 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31c92206-bebe-492e-be35-a008408e1e86-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31c92206-bebe-492e-be35-a008408e1e86-credential-socket\") pod \"31c92206-bebe-492e-be35-a008408e1e86\" (UID: \"31c92206-bebe-492e-be35-a008408e1e86\") " Oct 31 04:27:15.337388 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:15.336893 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31c92206-bebe-492e-be35-a008408e1e86-workload-socket" pod "31c92206-bebe-492e-be35-a008408e1e86" (UID: "31c92206-bebe-492e-be35-a008408e1e86"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:27:15.337388 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:15.336916 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31c92206-bebe-492e-be35-a008408e1e86-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31c92206-bebe-492e-be35-a008408e1e86-workload-certs\") pod \"31c92206-bebe-492e-be35-a008408e1e86\" (UID: \"31c92206-bebe-492e-be35-a008408e1e86\") " Oct 31 04:27:15.337388 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:15.336987 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31c92206-bebe-492e-be35-a008408e1e86-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31c92206-bebe-492e-be35-a008408e1e86-istio-data\") pod \"31c92206-bebe-492e-be35-a008408e1e86\" (UID: \"31c92206-bebe-492e-be35-a008408e1e86\") " Oct 31 04:27:15.337388 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:15.337026 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/31c92206-bebe-492e-be35-a008408e1e86-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31c92206-bebe-492e-be35-a008408e1e86-istio-podinfo\") pod \"31c92206-bebe-492e-be35-a008408e1e86\" (UID: \"31c92206-bebe-492e-be35-a008408e1e86\") " Oct 31 04:27:15.337388 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:15.337091 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/716d4a99-414f-449d-89c2-fa06e5751c60-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/716d4a99-414f-449d-89c2-fa06e5751c60-istio-envoy\") pod \"716d4a99-414f-449d-89c2-fa06e5751c60\" (UID: \"716d4a99-414f-449d-89c2-fa06e5751c60\") " Oct 31 04:27:15.337388 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:15.337144 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/716d4a99-414f-449d-89c2-fa06e5751c60-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/716d4a99-414f-449d-89c2-fa06e5751c60-istio-podinfo\") pod \"716d4a99-414f-449d-89c2-fa06e5751c60\" (UID: \"716d4a99-414f-449d-89c2-fa06e5751c60\") " Oct 31 04:27:15.337388 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:15.337278 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/716d4a99-414f-449d-89c2-fa06e5751c60-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/716d4a99-414f-449d-89c2-fa06e5751c60-istio-data\") pod \"716d4a99-414f-449d-89c2-fa06e5751c60\" (UID: \"716d4a99-414f-449d-89c2-fa06e5751c60\") " Oct 31 04:27:15.337699 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:15.337428 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31c92206-bebe-492e-be35-a008408e1e86-kube-api-access-v4dt9\" (UniqueName: \"kubernetes.io/projected/31c92206-bebe-492e-be35-a008408e1e86-kube-api-access-v4dt9\") pod \"31c92206-bebe-492e-be35-a008408e1e86\" (UID: \"31c92206-bebe-492e-be35-a008408e1e86\") " Oct 31 04:27:15.337699 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:15.337471 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/716d4a99-414f-449d-89c2-fa06e5751c60-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/716d4a99-414f-449d-89c2-fa06e5751c60-istiod-ca-cert\") pod \"716d4a99-414f-449d-89c2-fa06e5751c60\" (UID: \"716d4a99-414f-449d-89c2-fa06e5751c60\") " Oct 31 04:27:15.337699 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:15.337469 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/716d4a99-414f-449d-89c2-fa06e5751c60-workload-socket" pod "716d4a99-414f-449d-89c2-fa06e5751c60" (UID: "716d4a99-414f-449d-89c2-fa06e5751c60"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:27:15.337699 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:15.337500 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/716d4a99-414f-449d-89c2-fa06e5751c60-kube-api-access-dhz97\" (UniqueName: \"kubernetes.io/projected/716d4a99-414f-449d-89c2-fa06e5751c60-kube-api-access-dhz97\") pod \"716d4a99-414f-449d-89c2-fa06e5751c60\" (UID: \"716d4a99-414f-449d-89c2-fa06e5751c60\") " Oct 31 04:27:15.337699 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:15.337540 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31c92206-bebe-492e-be35-a008408e1e86-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31c92206-bebe-492e-be35-a008408e1e86-istio-envoy\") pod \"31c92206-bebe-492e-be35-a008408e1e86\" (UID: \"31c92206-bebe-492e-be35-a008408e1e86\") " Oct 31 04:27:15.338504 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:15.337972 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/716d4a99-414f-449d-89c2-fa06e5751c60-workload-certs" pod "716d4a99-414f-449d-89c2-fa06e5751c60" (UID: "716d4a99-414f-449d-89c2-fa06e5751c60"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:27:15.339397 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:15.339114 4376 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/716d4a99-414f-449d-89c2-fa06e5751c60-credential-socket\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:27:15.339397 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:15.339138 4376 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31c92206-bebe-492e-be35-a008408e1e86-workload-socket\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:27:15.339397 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:15.339152 4376 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/716d4a99-414f-449d-89c2-fa06e5751c60-workload-certs\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:27:15.339397 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:15.339167 4376 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/716d4a99-414f-449d-89c2-fa06e5751c60-workload-socket\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:27:15.340010 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:15.339744 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31c92206-bebe-492e-be35-a008408e1e86-istiod-ca-cert" pod "31c92206-bebe-492e-be35-a008408e1e86" (UID: "31c92206-bebe-492e-be35-a008408e1e86"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:27:15.340010 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:15.339969 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/716d4a99-414f-449d-89c2-fa06e5751c60-istio-token" pod "716d4a99-414f-449d-89c2-fa06e5751c60" (UID: "716d4a99-414f-449d-89c2-fa06e5751c60"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:27:15.340495 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:15.340469 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31c92206-bebe-492e-be35-a008408e1e86-istio-envoy" pod "31c92206-bebe-492e-be35-a008408e1e86" (UID: "31c92206-bebe-492e-be35-a008408e1e86"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:27:15.340802 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:15.340774 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/716d4a99-414f-449d-89c2-fa06e5751c60-istiod-ca-cert" pod "716d4a99-414f-449d-89c2-fa06e5751c60" (UID: "716d4a99-414f-449d-89c2-fa06e5751c60"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:27:15.343043 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:15.343017 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/716d4a99-414f-449d-89c2-fa06e5751c60-kube-api-access-dhz97" pod "716d4a99-414f-449d-89c2-fa06e5751c60" (UID: "716d4a99-414f-449d-89c2-fa06e5751c60"). InnerVolumeSpecName "kube-api-access-dhz97". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:27:15.343237 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:15.343218 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/716d4a99-414f-449d-89c2-fa06e5751c60-istio-data" pod "716d4a99-414f-449d-89c2-fa06e5751c60" (UID: "716d4a99-414f-449d-89c2-fa06e5751c60"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:27:15.343468 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:15.343447 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31c92206-bebe-492e-be35-a008408e1e86-workload-certs" pod "31c92206-bebe-492e-be35-a008408e1e86" (UID: "31c92206-bebe-492e-be35-a008408e1e86"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:27:15.343638 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:15.343620 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31c92206-bebe-492e-be35-a008408e1e86-credential-socket" pod "31c92206-bebe-492e-be35-a008408e1e86" (UID: "31c92206-bebe-492e-be35-a008408e1e86"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:27:15.343817 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:15.343794 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31c92206-bebe-492e-be35-a008408e1e86-istio-data" pod "31c92206-bebe-492e-be35-a008408e1e86" (UID: "31c92206-bebe-492e-be35-a008408e1e86"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:27:15.343920 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:15.343847 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31c92206-bebe-492e-be35-a008408e1e86-kube-api-access-v4dt9" pod "31c92206-bebe-492e-be35-a008408e1e86" (UID: "31c92206-bebe-492e-be35-a008408e1e86"). InnerVolumeSpecName "kube-api-access-v4dt9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:27:15.344728 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:15.344700 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31c92206-bebe-492e-be35-a008408e1e86-istio-token" pod "31c92206-bebe-492e-be35-a008408e1e86" (UID: "31c92206-bebe-492e-be35-a008408e1e86"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:27:15.346314 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:15.345947 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/31c92206-bebe-492e-be35-a008408e1e86-istio-podinfo" pod "31c92206-bebe-492e-be35-a008408e1e86" (UID: "31c92206-bebe-492e-be35-a008408e1e86"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 04:27:15.346497 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:15.346438 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/716d4a99-414f-449d-89c2-fa06e5751c60-istio-envoy" pod "716d4a99-414f-449d-89c2-fa06e5751c60" (UID: "716d4a99-414f-449d-89c2-fa06e5751c60"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:27:15.353538 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:15.353509 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/716d4a99-414f-449d-89c2-fa06e5751c60-istio-podinfo" pod "716d4a99-414f-449d-89c2-fa06e5751c60" (UID: "716d4a99-414f-449d-89c2-fa06e5751c60"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 04:27:15.440699 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:15.440608 4376 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31c92206-bebe-492e-be35-a008408e1e86-istiod-ca-cert\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:27:15.440699 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:15.440638 4376 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31c92206-bebe-492e-be35-a008408e1e86-istio-token\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:27:15.440699 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:15.440660 4376 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31c92206-bebe-492e-be35-a008408e1e86-credential-socket\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:27:15.440699 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:15.440671 4376 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31c92206-bebe-492e-be35-a008408e1e86-workload-certs\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:27:15.440699 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:15.440685 4376 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31c92206-bebe-492e-be35-a008408e1e86-istio-data\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:27:15.440699 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:15.440696 4376 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31c92206-bebe-492e-be35-a008408e1e86-istio-podinfo\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:27:15.441025 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:15.440707 4376 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/716d4a99-414f-449d-89c2-fa06e5751c60-istio-envoy\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:27:15.441025 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:15.440719 4376 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/716d4a99-414f-449d-89c2-fa06e5751c60-istio-podinfo\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:27:15.441025 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:15.440729 4376 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/716d4a99-414f-449d-89c2-fa06e5751c60-istio-data\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:27:15.441025 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:15.440741 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4dt9\" (UniqueName: \"kubernetes.io/projected/31c92206-bebe-492e-be35-a008408e1e86-kube-api-access-v4dt9\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:27:15.441025 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:15.440755 4376 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/716d4a99-414f-449d-89c2-fa06e5751c60-istiod-ca-cert\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:27:15.441025 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:15.440768 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhz97\" (UniqueName: \"kubernetes.io/projected/716d4a99-414f-449d-89c2-fa06e5751c60-kube-api-access-dhz97\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:27:15.441025 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:15.440781 4376 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31c92206-bebe-492e-be35-a008408e1e86-istio-envoy\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:27:15.441025 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:15.440804 4376 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/716d4a99-414f-449d-89c2-fa06e5751c60-istio-token\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:27:15.578068 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:15.578016 4376 generic.go:356] "Generic (PLEG): container finished" podID="716d4a99-414f-449d-89c2-fa06e5751c60" containerID="45d080900be9400444f7c02487989b189b36ce89754d4a8a50599929415695a7" exitCode=0 Oct 31 04:27:15.578267 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:15.578191 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-k2jb2" event={"ID":"716d4a99-414f-449d-89c2-fa06e5751c60","Type":"ContainerDied","Data":"45d080900be9400444f7c02487989b189b36ce89754d4a8a50599929415695a7"} Oct 31 04:27:15.578267 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:15.578249 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-k2jb2" event={"ID":"716d4a99-414f-449d-89c2-fa06e5751c60","Type":"ContainerDied","Data":"4890131363f9d90b67b89971ca6e9361848d4401391ff4f6b87b4af9b195dc5e"} Oct 31 04:27:15.578499 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:15.578280 4376 scope.go:122] "RemoveContainer" containerID="45d080900be9400444f7c02487989b189b36ce89754d4a8a50599929415695a7" Oct 31 04:27:15.578499 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:15.578481 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-k2jb2" Oct 31 04:27:15.583893 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:15.583859 4376 generic.go:356] "Generic (PLEG): container finished" podID="31c92206-bebe-492e-be35-a008408e1e86" containerID="379565cdf4008e1cecc097ff692ae676392277493bbda4f557b86a7536232897" exitCode=0 Oct 31 04:27:15.583993 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:15.583963 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-n6t9x" event={"ID":"31c92206-bebe-492e-be35-a008408e1e86","Type":"ContainerDied","Data":"379565cdf4008e1cecc097ff692ae676392277493bbda4f557b86a7536232897"} Oct 31 04:27:15.584037 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:15.584001 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-n6t9x" event={"ID":"31c92206-bebe-492e-be35-a008408e1e86","Type":"ContainerDied","Data":"2281b255574b50e80fa3bfef1d986e2019efb566bd8a3e7cb76c4191bbb8c3c5"} Oct 31 04:27:15.584085 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:15.584074 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-n6t9x" Oct 31 04:27:15.593736 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:15.593714 4376 scope.go:122] "RemoveContainer" containerID="45d080900be9400444f7c02487989b189b36ce89754d4a8a50599929415695a7" Oct 31 04:27:15.594053 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:27:15.594020 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45d080900be9400444f7c02487989b189b36ce89754d4a8a50599929415695a7\": container with ID starting with 45d080900be9400444f7c02487989b189b36ce89754d4a8a50599929415695a7 not found: ID does not exist" containerID="45d080900be9400444f7c02487989b189b36ce89754d4a8a50599929415695a7" Oct 31 04:27:15.594116 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:15.594067 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45d080900be9400444f7c02487989b189b36ce89754d4a8a50599929415695a7"} err="failed to get container status \"45d080900be9400444f7c02487989b189b36ce89754d4a8a50599929415695a7\": rpc error: code = NotFound desc = could not find container \"45d080900be9400444f7c02487989b189b36ce89754d4a8a50599929415695a7\": container with ID starting with 45d080900be9400444f7c02487989b189b36ce89754d4a8a50599929415695a7 not found: ID does not exist" Oct 31 04:27:15.594116 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:15.594092 4376 scope.go:122] "RemoveContainer" containerID="379565cdf4008e1cecc097ff692ae676392277493bbda4f557b86a7536232897" Oct 31 04:27:15.603276 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:15.603255 4376 scope.go:122] "RemoveContainer" containerID="379565cdf4008e1cecc097ff692ae676392277493bbda4f557b86a7536232897" Oct 31 04:27:15.603563 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:27:15.603543 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"379565cdf4008e1cecc097ff692ae676392277493bbda4f557b86a7536232897\": container with ID starting with 379565cdf4008e1cecc097ff692ae676392277493bbda4f557b86a7536232897 not found: ID does not exist" containerID="379565cdf4008e1cecc097ff692ae676392277493bbda4f557b86a7536232897" Oct 31 04:27:15.603633 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:15.603570 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"379565cdf4008e1cecc097ff692ae676392277493bbda4f557b86a7536232897"} err="failed to get container status \"379565cdf4008e1cecc097ff692ae676392277493bbda4f557b86a7536232897\": rpc error: code = NotFound desc = could not find container \"379565cdf4008e1cecc097ff692ae676392277493bbda4f557b86a7536232897\": container with ID starting with 379565cdf4008e1cecc097ff692ae676392277493bbda4f557b86a7536232897 not found: ID does not exist" Oct 31 04:27:15.633959 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:15.633920 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-k2jb2"] Oct 31 04:27:15.637482 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:15.637461 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-k2jb2"] Oct 31 04:27:15.651152 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:15.651125 4376 scope.go:122] "RemoveContainer" containerID="7fd115bca1878e75a1e201abd476d9f7cf3b445b90b7cecb9d8c841751a56d26" Oct 31 04:27:15.672672 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:15.672644 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-n6t9x"] Oct 31 04:27:15.678420 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:15.678394 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-n6t9x"] Oct 31 04:27:16.443378 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:16.443346 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31c92206-bebe-492e-be35-a008408e1e86" path="/var/lib/kubelet/pods/31c92206-bebe-492e-be35-a008408e1e86/volumes" Oct 31 04:27:16.443855 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:16.443841 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="716d4a99-414f-449d-89c2-fa06e5751c60" path="/var/lib/kubelet/pods/716d4a99-414f-449d-89c2-fa06e5751c60/volumes" Oct 31 04:27:19.463526 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:19.463498 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c6859a-59d59469c9-4r9lt" Oct 31 04:27:19.575849 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:19.575816 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-9ee727-5fbc8757cf-2gvbr"] Oct 31 04:27:19.576459 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:19.576440 4376 cpu_manager.go:414] "containerMap: removing container" podUID="716d4a99-414f-449d-89c2-fa06e5751c60" containerName="istio-proxy" Oct 31 04:27:19.576459 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:19.576457 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="716d4a99-414f-449d-89c2-fa06e5751c60" containerName="istio-proxy" Oct 31 04:27:19.576598 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:19.576471 4376 cpu_manager.go:414] "containerMap: removing container" podUID="151dc5b7-7589-4707-8586-8f70ab9b8d66" containerName="payload-processing" Oct 31 04:27:19.576598 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:19.576476 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="151dc5b7-7589-4707-8586-8f70ab9b8d66" containerName="payload-processing" Oct 31 04:27:19.576598 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:19.576486 4376 cpu_manager.go:414] "containerMap: removing container" podUID="31c92206-bebe-492e-be35-a008408e1e86" containerName="istio-proxy" Oct 31 04:27:19.576598 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:19.576492 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31c92206-bebe-492e-be35-a008408e1e86" containerName="istio-proxy" Oct 31 04:27:19.576734 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:19.576623 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="716d4a99-414f-449d-89c2-fa06e5751c60" containerName="istio-proxy" Oct 31 04:27:19.576734 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:19.576632 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="151dc5b7-7589-4707-8586-8f70ab9b8d66" containerName="payload-processing" Oct 31 04:27:19.576734 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:19.576638 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="31c92206-bebe-492e-be35-a008408e1e86" containerName="istio-proxy" Oct 31 04:27:19.582144 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:19.582126 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-9ee727-5fbc8757cf-2gvbr" Oct 31 04:27:19.597777 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:19.597756 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-9ee727-serving-cert\"" Oct 31 04:27:19.600439 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:19.600421 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-9ee727-5fbc8757cf-2gvbr"] Oct 31 04:27:19.733849 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:19.733768 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b6a7f8f5-13b4-46cc-93c7-3bdd9e107f7f-metrics-certs\") pod \"maas-api-e2e-praxis-a-9ee727-5fbc8757cf-2gvbr\" (UID: \"b6a7f8f5-13b4-46cc-93c7-3bdd9e107f7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-9ee727-5fbc8757cf-2gvbr" Oct 31 04:27:19.733849 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:19.733808 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6a7f8f5-13b4-46cc-93c7-3bdd9e107f7f-maas-api-tls\") pod \"maas-api-e2e-praxis-a-9ee727-5fbc8757cf-2gvbr\" (UID: \"b6a7f8f5-13b4-46cc-93c7-3bdd9e107f7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-9ee727-5fbc8757cf-2gvbr" Oct 31 04:27:19.734089 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:19.733959 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmzsd\" (UniqueName: \"kubernetes.io/projected/b6a7f8f5-13b4-46cc-93c7-3bdd9e107f7f-kube-api-access-rmzsd\") pod \"maas-api-e2e-praxis-a-9ee727-5fbc8757cf-2gvbr\" (UID: \"b6a7f8f5-13b4-46cc-93c7-3bdd9e107f7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-9ee727-5fbc8757cf-2gvbr" Oct 31 04:27:19.734089 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:19.734000 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6a7f8f5-13b4-46cc-93c7-3bdd9e107f7f-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-9ee727-5fbc8757cf-2gvbr\" (UID: \"b6a7f8f5-13b4-46cc-93c7-3bdd9e107f7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-9ee727-5fbc8757cf-2gvbr" Oct 31 04:27:19.850353 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:19.835360 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmzsd\" (UniqueName: \"kubernetes.io/projected/b6a7f8f5-13b4-46cc-93c7-3bdd9e107f7f-kube-api-access-rmzsd\") pod \"maas-api-e2e-praxis-a-9ee727-5fbc8757cf-2gvbr\" (UID: \"b6a7f8f5-13b4-46cc-93c7-3bdd9e107f7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-9ee727-5fbc8757cf-2gvbr" Oct 31 04:27:19.857485 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:19.857445 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6a7f8f5-13b4-46cc-93c7-3bdd9e107f7f-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-9ee727-5fbc8757cf-2gvbr\" (UID: \"b6a7f8f5-13b4-46cc-93c7-3bdd9e107f7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-9ee727-5fbc8757cf-2gvbr" Oct 31 04:27:19.858806 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:19.858771 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b6a7f8f5-13b4-46cc-93c7-3bdd9e107f7f-metrics-certs\") pod \"maas-api-e2e-praxis-a-9ee727-5fbc8757cf-2gvbr\" (UID: \"b6a7f8f5-13b4-46cc-93c7-3bdd9e107f7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-9ee727-5fbc8757cf-2gvbr" Oct 31 04:27:19.858955 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:19.858818 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6a7f8f5-13b4-46cc-93c7-3bdd9e107f7f-maas-api-tls\") pod \"maas-api-e2e-praxis-a-9ee727-5fbc8757cf-2gvbr\" (UID: \"b6a7f8f5-13b4-46cc-93c7-3bdd9e107f7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-9ee727-5fbc8757cf-2gvbr" Oct 31 04:27:19.859382 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:19.858460 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6a7f8f5-13b4-46cc-93c7-3bdd9e107f7f-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-9ee727-5fbc8757cf-2gvbr\" (UID: \"b6a7f8f5-13b4-46cc-93c7-3bdd9e107f7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-9ee727-5fbc8757cf-2gvbr" Oct 31 04:27:19.896987 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:19.896951 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmzsd\" (UniqueName: \"kubernetes.io/projected/b6a7f8f5-13b4-46cc-93c7-3bdd9e107f7f-kube-api-access-rmzsd\") pod \"maas-api-e2e-praxis-a-9ee727-5fbc8757cf-2gvbr\" (UID: \"b6a7f8f5-13b4-46cc-93c7-3bdd9e107f7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-9ee727-5fbc8757cf-2gvbr" Oct 31 04:27:19.913838 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:19.913796 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b6a7f8f5-13b4-46cc-93c7-3bdd9e107f7f-metrics-certs\") pod \"maas-api-e2e-praxis-a-9ee727-5fbc8757cf-2gvbr\" (UID: \"b6a7f8f5-13b4-46cc-93c7-3bdd9e107f7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-9ee727-5fbc8757cf-2gvbr" Oct 31 04:27:19.918179 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:19.918149 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6a7f8f5-13b4-46cc-93c7-3bdd9e107f7f-maas-api-tls\") pod \"maas-api-e2e-praxis-a-9ee727-5fbc8757cf-2gvbr\" (UID: \"b6a7f8f5-13b4-46cc-93c7-3bdd9e107f7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-9ee727-5fbc8757cf-2gvbr" Oct 31 04:27:20.196163 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:20.196132 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-9ee727-5fbc8757cf-2gvbr" Oct 31 04:27:20.496043 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:27:20.496011 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6a7f8f5_13b4_46cc_93c7_3bdd9e107f7f.slice/crio-2ce63e68032fc918851bf890550edde6f5314725a60c2941eede495cb353d75e WatchSource:0}: Error finding container 2ce63e68032fc918851bf890550edde6f5314725a60c2941eede495cb353d75e: Status 404 returned error can't find the container with id 2ce63e68032fc918851bf890550edde6f5314725a60c2941eede495cb353d75e Oct 31 04:27:20.497498 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:20.497473 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-9ee727-5fbc8757cf-2gvbr"] Oct 31 04:27:20.616039 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:20.616006 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-9ee727-5fbc8757cf-2gvbr" event={"ID":"b6a7f8f5-13b4-46cc-93c7-3bdd9e107f7f","Type":"ContainerStarted","Data":"2ce63e68032fc918851bf890550edde6f5314725a60c2941eede495cb353d75e"} Oct 31 04:27:21.525900 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:21.525866 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-c6859a-5cbcf46d46-c4t77"] Oct 31 04:27:21.529909 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:21.529887 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c6859a-5cbcf46d46-c4t77" Oct 31 04:27:21.548664 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:21.548637 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-c6859a-5cbcf46d46-c4t77"] Oct 31 04:27:21.557520 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:21.557494 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-c6859a-serving-cert\"" Oct 31 04:27:21.604316 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:21.604285 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef7c032f-cdf7-43e3-9c9e-ef46f21ac3e4-metrics-certs\") pod \"maas-api-e2e-praxis-a-c6859a-5cbcf46d46-c4t77\" (UID: \"ef7c032f-cdf7-43e3-9c9e-ef46f21ac3e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c6859a-5cbcf46d46-c4t77" Oct 31 04:27:21.604511 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:21.604436 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nz5qq\" (UniqueName: \"kubernetes.io/projected/ef7c032f-cdf7-43e3-9c9e-ef46f21ac3e4-kube-api-access-nz5qq\") pod \"maas-api-e2e-praxis-a-c6859a-5cbcf46d46-c4t77\" (UID: \"ef7c032f-cdf7-43e3-9c9e-ef46f21ac3e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c6859a-5cbcf46d46-c4t77" Oct 31 04:27:21.604511 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:21.604497 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef7c032f-cdf7-43e3-9c9e-ef46f21ac3e4-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-c6859a-5cbcf46d46-c4t77\" (UID: \"ef7c032f-cdf7-43e3-9c9e-ef46f21ac3e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c6859a-5cbcf46d46-c4t77" Oct 31 04:27:21.604644 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:21.604621 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef7c032f-cdf7-43e3-9c9e-ef46f21ac3e4-maas-api-tls\") pod \"maas-api-e2e-praxis-a-c6859a-5cbcf46d46-c4t77\" (UID: \"ef7c032f-cdf7-43e3-9c9e-ef46f21ac3e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c6859a-5cbcf46d46-c4t77" Oct 31 04:27:21.624267 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:21.624236 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-9ee727-5fbc8757cf-2gvbr" event={"ID":"b6a7f8f5-13b4-46cc-93c7-3bdd9e107f7f","Type":"ContainerStarted","Data":"5a8c1d6b418f57a17cbfc0a96ff07d4e00e1eac67a5e4900c1a90201a4e59ae9"} Oct 31 04:27:21.624435 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:21.624371 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-9ee727-5fbc8757cf-2gvbr" Oct 31 04:27:21.669094 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:21.669035 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-9ee727-5fbc8757cf-2gvbr" podStartSLOduration=2.157339285 podStartE2EDuration="2.669020885s" totalImagesPullingTime="511.6816ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:27:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:27:21.667002678 +0000 UTC m=+1389.381337295" watchObservedRunningTime="2026-10-31 04:27:21.669020885 +0000 UTC m=+1389.383355501" Oct 31 04:27:21.706050 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:21.705992 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef7c032f-cdf7-43e3-9c9e-ef46f21ac3e4-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-c6859a-5cbcf46d46-c4t77\" (UID: \"ef7c032f-cdf7-43e3-9c9e-ef46f21ac3e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c6859a-5cbcf46d46-c4t77" Oct 31 04:27:21.706239 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:21.706133 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef7c032f-cdf7-43e3-9c9e-ef46f21ac3e4-maas-api-tls\") pod \"maas-api-e2e-praxis-a-c6859a-5cbcf46d46-c4t77\" (UID: \"ef7c032f-cdf7-43e3-9c9e-ef46f21ac3e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c6859a-5cbcf46d46-c4t77" Oct 31 04:27:21.706360 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:21.706324 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef7c032f-cdf7-43e3-9c9e-ef46f21ac3e4-metrics-certs\") pod \"maas-api-e2e-praxis-a-c6859a-5cbcf46d46-c4t77\" (UID: \"ef7c032f-cdf7-43e3-9c9e-ef46f21ac3e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c6859a-5cbcf46d46-c4t77" Oct 31 04:27:21.706687 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:21.706662 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nz5qq\" (UniqueName: \"kubernetes.io/projected/ef7c032f-cdf7-43e3-9c9e-ef46f21ac3e4-kube-api-access-nz5qq\") pod \"maas-api-e2e-praxis-a-c6859a-5cbcf46d46-c4t77\" (UID: \"ef7c032f-cdf7-43e3-9c9e-ef46f21ac3e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c6859a-5cbcf46d46-c4t77" Oct 31 04:27:21.706799 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:21.706711 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef7c032f-cdf7-43e3-9c9e-ef46f21ac3e4-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-c6859a-5cbcf46d46-c4t77\" (UID: \"ef7c032f-cdf7-43e3-9c9e-ef46f21ac3e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c6859a-5cbcf46d46-c4t77" Oct 31 04:27:21.708592 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:21.708572 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef7c032f-cdf7-43e3-9c9e-ef46f21ac3e4-maas-api-tls\") pod \"maas-api-e2e-praxis-a-c6859a-5cbcf46d46-c4t77\" (UID: \"ef7c032f-cdf7-43e3-9c9e-ef46f21ac3e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c6859a-5cbcf46d46-c4t77" Oct 31 04:27:21.708872 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:21.708848 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef7c032f-cdf7-43e3-9c9e-ef46f21ac3e4-metrics-certs\") pod \"maas-api-e2e-praxis-a-c6859a-5cbcf46d46-c4t77\" (UID: \"ef7c032f-cdf7-43e3-9c9e-ef46f21ac3e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c6859a-5cbcf46d46-c4t77" Oct 31 04:27:21.730268 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:21.730240 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nz5qq\" (UniqueName: \"kubernetes.io/projected/ef7c032f-cdf7-43e3-9c9e-ef46f21ac3e4-kube-api-access-nz5qq\") pod \"maas-api-e2e-praxis-a-c6859a-5cbcf46d46-c4t77\" (UID: \"ef7c032f-cdf7-43e3-9c9e-ef46f21ac3e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c6859a-5cbcf46d46-c4t77" Oct 31 04:27:21.859519 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:21.855675 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c6859a-5cbcf46d46-c4t77" Oct 31 04:27:22.194170 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:27:22.194137 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef7c032f_cdf7_43e3_9c9e_ef46f21ac3e4.slice/crio-df23409131c05267a8b8d1326bca1afaceb56ae56bcaec17d029061630e55316 WatchSource:0}: Error finding container df23409131c05267a8b8d1326bca1afaceb56ae56bcaec17d029061630e55316: Status 404 returned error can't find the container with id df23409131c05267a8b8d1326bca1afaceb56ae56bcaec17d029061630e55316 Oct 31 04:27:22.195834 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:22.195811 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-c6859a-5cbcf46d46-c4t77"] Oct 31 04:27:22.633743 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:22.633709 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c6859a-5cbcf46d46-c4t77" event={"ID":"ef7c032f-cdf7-43e3-9c9e-ef46f21ac3e4","Type":"ContainerStarted","Data":"2ad44c477aad082341fc8196ef46f6ce13c72d97894fa89491608b7a39c19044"} Oct 31 04:27:22.634211 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:22.633789 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c6859a-5cbcf46d46-c4t77" event={"ID":"ef7c032f-cdf7-43e3-9c9e-ef46f21ac3e4","Type":"ContainerStarted","Data":"df23409131c05267a8b8d1326bca1afaceb56ae56bcaec17d029061630e55316"} Oct 31 04:27:22.634211 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:22.634166 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c6859a-5cbcf46d46-c4t77" Oct 31 04:27:22.677891 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:22.677841 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c6859a-5cbcf46d46-c4t77" podStartSLOduration=1.343936074 podStartE2EDuration="1.677826562s" totalImagesPullingTime="333.890488ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:27:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:27:22.676274953 +0000 UTC m=+1390.390609595" watchObservedRunningTime="2026-10-31 04:27:22.677826562 +0000 UTC m=+1390.392161179" Oct 31 04:27:27.639185 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:27.639103 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-9ee727-5fbc8757cf-2gvbr" Oct 31 04:27:28.648479 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:28.648448 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c6859a-5cbcf46d46-c4t77" Oct 31 04:27:30.838509 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:30.838473 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 04:27:31.854753 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:31.854719 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-d1d441-664bf44798-dr568"] Oct 31 04:27:31.859170 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:31.859150 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d1d441-664bf44798-dr568" Oct 31 04:27:31.888886 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:31.888859 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-d1d441-serving-cert\"" Oct 31 04:27:31.893822 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:31.893795 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-d1d441-664bf44798-dr568"] Oct 31 04:27:31.901804 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:31.901778 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3970ed6-ce46-488f-a08c-46a01d41181d-metrics-certs\") pod \"maas-api-e2e-praxis-a-d1d441-664bf44798-dr568\" (UID: \"c3970ed6-ce46-488f-a08c-46a01d41181d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d1d441-664bf44798-dr568" Oct 31 04:27:31.901939 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:31.901853 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3970ed6-ce46-488f-a08c-46a01d41181d-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-d1d441-664bf44798-dr568\" (UID: \"c3970ed6-ce46-488f-a08c-46a01d41181d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d1d441-664bf44798-dr568" Oct 31 04:27:31.902073 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:31.902021 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fx8f\" (UniqueName: \"kubernetes.io/projected/c3970ed6-ce46-488f-a08c-46a01d41181d-kube-api-access-4fx8f\") pod \"maas-api-e2e-praxis-a-d1d441-664bf44798-dr568\" (UID: \"c3970ed6-ce46-488f-a08c-46a01d41181d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d1d441-664bf44798-dr568" Oct 31 04:27:31.902160 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:31.902146 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3970ed6-ce46-488f-a08c-46a01d41181d-maas-api-tls\") pod \"maas-api-e2e-praxis-a-d1d441-664bf44798-dr568\" (UID: \"c3970ed6-ce46-488f-a08c-46a01d41181d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d1d441-664bf44798-dr568" Oct 31 04:27:32.003026 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:32.002991 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3970ed6-ce46-488f-a08c-46a01d41181d-metrics-certs\") pod \"maas-api-e2e-praxis-a-d1d441-664bf44798-dr568\" (UID: \"c3970ed6-ce46-488f-a08c-46a01d41181d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d1d441-664bf44798-dr568" Oct 31 04:27:32.003198 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:32.003087 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3970ed6-ce46-488f-a08c-46a01d41181d-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-d1d441-664bf44798-dr568\" (UID: \"c3970ed6-ce46-488f-a08c-46a01d41181d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d1d441-664bf44798-dr568" Oct 31 04:27:32.003256 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:32.003198 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fx8f\" (UniqueName: \"kubernetes.io/projected/c3970ed6-ce46-488f-a08c-46a01d41181d-kube-api-access-4fx8f\") pod \"maas-api-e2e-praxis-a-d1d441-664bf44798-dr568\" (UID: \"c3970ed6-ce46-488f-a08c-46a01d41181d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d1d441-664bf44798-dr568" Oct 31 04:27:32.003370 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:32.003322 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3970ed6-ce46-488f-a08c-46a01d41181d-maas-api-tls\") pod \"maas-api-e2e-praxis-a-d1d441-664bf44798-dr568\" (UID: \"c3970ed6-ce46-488f-a08c-46a01d41181d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d1d441-664bf44798-dr568" Oct 31 04:27:32.003511 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:27:32.003487 4376 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-a-d1d441-serving-cert: secret "maas-api-e2e-praxis-a-d1d441-serving-cert" not found Oct 31 04:27:32.003609 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:27:32.003595 4376 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3970ed6-ce46-488f-a08c-46a01d41181d-maas-api-tls podName:c3970ed6-ce46-488f-a08c-46a01d41181d nodeName:}" failed. No retries permitted until 2026-10-31 04:27:32.503576351 +0000 UTC m=+1400.217910962 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c3970ed6-ce46-488f-a08c-46a01d41181d-maas-api-tls") pod "maas-api-e2e-praxis-a-d1d441-664bf44798-dr568" (UID: "c3970ed6-ce46-488f-a08c-46a01d41181d") : secret "maas-api-e2e-praxis-a-d1d441-serving-cert" not found Oct 31 04:27:32.003880 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:32.003861 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3970ed6-ce46-488f-a08c-46a01d41181d-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-d1d441-664bf44798-dr568\" (UID: \"c3970ed6-ce46-488f-a08c-46a01d41181d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d1d441-664bf44798-dr568" Oct 31 04:27:32.005573 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:32.005551 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3970ed6-ce46-488f-a08c-46a01d41181d-metrics-certs\") pod \"maas-api-e2e-praxis-a-d1d441-664bf44798-dr568\" (UID: \"c3970ed6-ce46-488f-a08c-46a01d41181d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d1d441-664bf44798-dr568" Oct 31 04:27:32.027321 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:32.027287 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fx8f\" (UniqueName: \"kubernetes.io/projected/c3970ed6-ce46-488f-a08c-46a01d41181d-kube-api-access-4fx8f\") pod \"maas-api-e2e-praxis-a-d1d441-664bf44798-dr568\" (UID: \"c3970ed6-ce46-488f-a08c-46a01d41181d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d1d441-664bf44798-dr568" Oct 31 04:27:32.516410 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:32.516378 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3970ed6-ce46-488f-a08c-46a01d41181d-maas-api-tls\") pod \"maas-api-e2e-praxis-a-d1d441-664bf44798-dr568\" (UID: \"c3970ed6-ce46-488f-a08c-46a01d41181d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d1d441-664bf44798-dr568" Oct 31 04:27:32.518845 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:32.518823 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3970ed6-ce46-488f-a08c-46a01d41181d-maas-api-tls\") pod \"maas-api-e2e-praxis-a-d1d441-664bf44798-dr568\" (UID: \"c3970ed6-ce46-488f-a08c-46a01d41181d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d1d441-664bf44798-dr568" Oct 31 04:27:32.772868 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:32.772763 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d1d441-664bf44798-dr568" Oct 31 04:27:33.178582 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:33.176952 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-d1d441-664bf44798-dr568"] Oct 31 04:27:33.201879 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:27:33.201843 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3970ed6_ce46_488f_a08c_46a01d41181d.slice/crio-76743cebc8df9704e7dc6f06fccddfe50cedb6d75563c83500c398a29666752b WatchSource:0}: Error finding container 76743cebc8df9704e7dc6f06fccddfe50cedb6d75563c83500c398a29666752b: Status 404 returned error can't find the container with id 76743cebc8df9704e7dc6f06fccddfe50cedb6d75563c83500c398a29666752b Oct 31 04:27:33.712903 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:33.712637 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d1d441-664bf44798-dr568" event={"ID":"c3970ed6-ce46-488f-a08c-46a01d41181d","Type":"ContainerStarted","Data":"416d6964bcd237c6217dcefefccf5f78fe9dd7f2f52cb9e27c4cb2f9505e8d8d"} Oct 31 04:27:33.712903 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:33.712702 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d1d441-664bf44798-dr568" event={"ID":"c3970ed6-ce46-488f-a08c-46a01d41181d","Type":"ContainerStarted","Data":"76743cebc8df9704e7dc6f06fccddfe50cedb6d75563c83500c398a29666752b"} Oct 31 04:27:33.712903 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:33.712869 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d1d441-664bf44798-dr568" Oct 31 04:27:33.758119 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:33.758051 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d1d441-664bf44798-dr568" podStartSLOduration=2.506903708 podStartE2EDuration="2.75803466s" totalImagesPullingTime="251.130952ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:27:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:27:33.756104206 +0000 UTC m=+1401.470438833" watchObservedRunningTime="2026-10-31 04:27:33.75803466 +0000 UTC m=+1401.472369276" Oct 31 04:27:39.723962 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:39.723930 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d1d441-664bf44798-dr568" Oct 31 04:27:41.002594 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:27:41.002561 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 04:27:42.159291 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:42.159254 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-f91cd1-openshift-default-556cb85755-v94km"] Oct 31 04:27:42.159838 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:42.159550 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-f91cd1-openshift-default-556cb85755-v94km" podUID="fd4632c9-069d-466e-b00a-c75aa4976d0d" containerName="istio-proxy" containerID="cri-o://ba339f16daedb9c2675cf1aac882661a7c9e2bdb4351accdfd30732a6e6f240d" gracePeriod=30 Oct 31 04:27:47.432412 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:47.432350 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f91cd1-openshift-default-556cb85755-v94km" Oct 31 04:27:47.522249 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:47.522212 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd4632c9-069d-466e-b00a-c75aa4976d0d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd4632c9-069d-466e-b00a-c75aa4976d0d-workload-socket\") pod \"fd4632c9-069d-466e-b00a-c75aa4976d0d\" (UID: \"fd4632c9-069d-466e-b00a-c75aa4976d0d\") " Oct 31 04:27:47.522443 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:47.522336 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd4632c9-069d-466e-b00a-c75aa4976d0d-kube-api-access-n9lmb\" (UniqueName: \"kubernetes.io/projected/fd4632c9-069d-466e-b00a-c75aa4976d0d-kube-api-access-n9lmb\") pod \"fd4632c9-069d-466e-b00a-c75aa4976d0d\" (UID: \"fd4632c9-069d-466e-b00a-c75aa4976d0d\") " Oct 31 04:27:47.522443 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:47.522379 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fd4632c9-069d-466e-b00a-c75aa4976d0d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd4632c9-069d-466e-b00a-c75aa4976d0d-istio-podinfo\") pod \"fd4632c9-069d-466e-b00a-c75aa4976d0d\" (UID: \"fd4632c9-069d-466e-b00a-c75aa4976d0d\") " Oct 31 04:27:47.522443 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:47.522412 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd4632c9-069d-466e-b00a-c75aa4976d0d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd4632c9-069d-466e-b00a-c75aa4976d0d-istio-data\") pod \"fd4632c9-069d-466e-b00a-c75aa4976d0d\" (UID: \"fd4632c9-069d-466e-b00a-c75aa4976d0d\") " Oct 31 04:27:47.522443 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:47.522442 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd4632c9-069d-466e-b00a-c75aa4976d0d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd4632c9-069d-466e-b00a-c75aa4976d0d-istio-envoy\") pod \"fd4632c9-069d-466e-b00a-c75aa4976d0d\" (UID: \"fd4632c9-069d-466e-b00a-c75aa4976d0d\") " Oct 31 04:27:47.522677 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:47.522493 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd4632c9-069d-466e-b00a-c75aa4976d0d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd4632c9-069d-466e-b00a-c75aa4976d0d-workload-certs\") pod \"fd4632c9-069d-466e-b00a-c75aa4976d0d\" (UID: \"fd4632c9-069d-466e-b00a-c75aa4976d0d\") " Oct 31 04:27:47.522677 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:47.522525 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd4632c9-069d-466e-b00a-c75aa4976d0d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd4632c9-069d-466e-b00a-c75aa4976d0d-credential-socket\") pod \"fd4632c9-069d-466e-b00a-c75aa4976d0d\" (UID: \"fd4632c9-069d-466e-b00a-c75aa4976d0d\") " Oct 31 04:27:47.522677 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:47.522535 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd4632c9-069d-466e-b00a-c75aa4976d0d-workload-socket" pod "fd4632c9-069d-466e-b00a-c75aa4976d0d" (UID: "fd4632c9-069d-466e-b00a-c75aa4976d0d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:27:47.522677 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:47.522585 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd4632c9-069d-466e-b00a-c75aa4976d0d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd4632c9-069d-466e-b00a-c75aa4976d0d-istiod-ca-cert\") pod \"fd4632c9-069d-466e-b00a-c75aa4976d0d\" (UID: \"fd4632c9-069d-466e-b00a-c75aa4976d0d\") " Oct 31 04:27:47.522677 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:47.522613 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd4632c9-069d-466e-b00a-c75aa4976d0d-istio-token\" (UniqueName: \"kubernetes.io/projected/fd4632c9-069d-466e-b00a-c75aa4976d0d-istio-token\") pod \"fd4632c9-069d-466e-b00a-c75aa4976d0d\" (UID: \"fd4632c9-069d-466e-b00a-c75aa4976d0d\") " Oct 31 04:27:47.522924 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:47.522871 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd4632c9-069d-466e-b00a-c75aa4976d0d-credential-socket" pod "fd4632c9-069d-466e-b00a-c75aa4976d0d" (UID: "fd4632c9-069d-466e-b00a-c75aa4976d0d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:27:47.522988 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:47.522961 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd4632c9-069d-466e-b00a-c75aa4976d0d-workload-certs" pod "fd4632c9-069d-466e-b00a-c75aa4976d0d" (UID: "fd4632c9-069d-466e-b00a-c75aa4976d0d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:27:47.522988 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:47.522978 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd4632c9-069d-466e-b00a-c75aa4976d0d-istio-data" pod "fd4632c9-069d-466e-b00a-c75aa4976d0d" (UID: "fd4632c9-069d-466e-b00a-c75aa4976d0d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:27:47.523087 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:47.523039 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd4632c9-069d-466e-b00a-c75aa4976d0d-istiod-ca-cert" pod "fd4632c9-069d-466e-b00a-c75aa4976d0d" (UID: "fd4632c9-069d-466e-b00a-c75aa4976d0d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:27:47.523409 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:47.523389 4376 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd4632c9-069d-466e-b00a-c75aa4976d0d-istio-data\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:27:47.523522 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:47.523413 4376 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd4632c9-069d-466e-b00a-c75aa4976d0d-workload-certs\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:27:47.523522 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:47.523426 4376 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd4632c9-069d-466e-b00a-c75aa4976d0d-credential-socket\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:27:47.523522 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:47.523435 4376 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd4632c9-069d-466e-b00a-c75aa4976d0d-istiod-ca-cert\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:27:47.523522 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:47.523443 4376 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd4632c9-069d-466e-b00a-c75aa4976d0d-workload-socket\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:27:47.525142 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:47.525115 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd4632c9-069d-466e-b00a-c75aa4976d0d-istio-envoy" pod "fd4632c9-069d-466e-b00a-c75aa4976d0d" (UID: "fd4632c9-069d-466e-b00a-c75aa4976d0d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:27:47.525277 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:47.525253 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fd4632c9-069d-466e-b00a-c75aa4976d0d-istio-podinfo" pod "fd4632c9-069d-466e-b00a-c75aa4976d0d" (UID: "fd4632c9-069d-466e-b00a-c75aa4976d0d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 04:27:47.525399 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:47.525282 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd4632c9-069d-466e-b00a-c75aa4976d0d-kube-api-access-n9lmb" pod "fd4632c9-069d-466e-b00a-c75aa4976d0d" (UID: "fd4632c9-069d-466e-b00a-c75aa4976d0d"). InnerVolumeSpecName "kube-api-access-n9lmb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:27:47.525742 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:47.525718 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd4632c9-069d-466e-b00a-c75aa4976d0d-istio-token" pod "fd4632c9-069d-466e-b00a-c75aa4976d0d" (UID: "fd4632c9-069d-466e-b00a-c75aa4976d0d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:27:47.625003 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:47.624965 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9lmb\" (UniqueName: \"kubernetes.io/projected/fd4632c9-069d-466e-b00a-c75aa4976d0d-kube-api-access-n9lmb\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:27:47.625003 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:47.624998 4376 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd4632c9-069d-466e-b00a-c75aa4976d0d-istio-podinfo\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:27:47.625003 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:47.625007 4376 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd4632c9-069d-466e-b00a-c75aa4976d0d-istio-envoy\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:27:47.625231 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:47.625016 4376 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd4632c9-069d-466e-b00a-c75aa4976d0d-istio-token\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:27:47.805563 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:47.805468 4376 generic.go:356] "Generic (PLEG): container finished" podID="fd4632c9-069d-466e-b00a-c75aa4976d0d" containerID="ba339f16daedb9c2675cf1aac882661a7c9e2bdb4351accdfd30732a6e6f240d" exitCode=0 Oct 31 04:27:47.805563 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:47.805520 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f91cd1-openshift-default-556cb85755-v94km" Oct 31 04:27:47.805563 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:47.805527 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f91cd1-openshift-default-556cb85755-v94km" event={"ID":"fd4632c9-069d-466e-b00a-c75aa4976d0d","Type":"ContainerDied","Data":"ba339f16daedb9c2675cf1aac882661a7c9e2bdb4351accdfd30732a6e6f240d"} Oct 31 04:27:47.805821 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:47.805575 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f91cd1-openshift-default-556cb85755-v94km" event={"ID":"fd4632c9-069d-466e-b00a-c75aa4976d0d","Type":"ContainerDied","Data":"a767d9c686b06a4570d14a42ec6922a14979f7cd3b1eacd2f7fbcf32220c4791"} Oct 31 04:27:47.805821 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:47.805604 4376 scope.go:122] "RemoveContainer" containerID="ba339f16daedb9c2675cf1aac882661a7c9e2bdb4351accdfd30732a6e6f240d" Oct 31 04:27:47.825301 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:47.825276 4376 scope.go:122] "RemoveContainer" containerID="3dd4da46f8a97e21d8f1fa204272a8ef9df80bb2368eb2ad619cd0b2063cfbb8" Oct 31 04:27:47.835450 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:47.835427 4376 scope.go:122] "RemoveContainer" containerID="ba339f16daedb9c2675cf1aac882661a7c9e2bdb4351accdfd30732a6e6f240d" Oct 31 04:27:47.835720 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:27:47.835700 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba339f16daedb9c2675cf1aac882661a7c9e2bdb4351accdfd30732a6e6f240d\": container with ID starting with ba339f16daedb9c2675cf1aac882661a7c9e2bdb4351accdfd30732a6e6f240d not found: ID does not exist" containerID="ba339f16daedb9c2675cf1aac882661a7c9e2bdb4351accdfd30732a6e6f240d" Oct 31 04:27:47.835765 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:47.835729 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba339f16daedb9c2675cf1aac882661a7c9e2bdb4351accdfd30732a6e6f240d"} err="failed to get container status \"ba339f16daedb9c2675cf1aac882661a7c9e2bdb4351accdfd30732a6e6f240d\": rpc error: code = NotFound desc = could not find container \"ba339f16daedb9c2675cf1aac882661a7c9e2bdb4351accdfd30732a6e6f240d\": container with ID starting with ba339f16daedb9c2675cf1aac882661a7c9e2bdb4351accdfd30732a6e6f240d not found: ID does not exist" Oct 31 04:27:47.835765 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:47.835746 4376 scope.go:122] "RemoveContainer" containerID="3dd4da46f8a97e21d8f1fa204272a8ef9df80bb2368eb2ad619cd0b2063cfbb8" Oct 31 04:27:47.836031 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:27:47.836013 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3dd4da46f8a97e21d8f1fa204272a8ef9df80bb2368eb2ad619cd0b2063cfbb8\": container with ID starting with 3dd4da46f8a97e21d8f1fa204272a8ef9df80bb2368eb2ad619cd0b2063cfbb8 not found: ID does not exist" containerID="3dd4da46f8a97e21d8f1fa204272a8ef9df80bb2368eb2ad619cd0b2063cfbb8" Oct 31 04:27:47.836080 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:47.836037 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3dd4da46f8a97e21d8f1fa204272a8ef9df80bb2368eb2ad619cd0b2063cfbb8"} err="failed to get container status \"3dd4da46f8a97e21d8f1fa204272a8ef9df80bb2368eb2ad619cd0b2063cfbb8\": rpc error: code = NotFound desc = could not find container \"3dd4da46f8a97e21d8f1fa204272a8ef9df80bb2368eb2ad619cd0b2063cfbb8\": container with ID starting with 3dd4da46f8a97e21d8f1fa204272a8ef9df80bb2368eb2ad619cd0b2063cfbb8 not found: ID does not exist" Oct 31 04:27:47.863013 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:47.862983 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-f91cd1-openshift-default-556cb85755-v94km"] Oct 31 04:27:47.875507 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:47.875476 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-f91cd1-openshift-default-556cb85755-v94km"] Oct 31 04:27:48.447304 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:48.447269 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd4632c9-069d-466e-b00a-c75aa4976d0d" path="/var/lib/kubelet/pods/fd4632c9-069d-466e-b00a-c75aa4976d0d/volumes" Oct 31 04:27:50.308212 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:50.308180 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-edc450-openshift-default-84c757994f-q4vwz"] Oct 31 04:27:50.309001 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:50.308979 4376 cpu_manager.go:414] "containerMap: removing container" podUID="fd4632c9-069d-466e-b00a-c75aa4976d0d" containerName="istio-proxy" Oct 31 04:27:50.309119 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:50.309002 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd4632c9-069d-466e-b00a-c75aa4976d0d" containerName="istio-proxy" Oct 31 04:27:50.309178 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:50.309163 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd4632c9-069d-466e-b00a-c75aa4976d0d" containerName="istio-proxy" Oct 31 04:27:50.309231 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:50.309183 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd4632c9-069d-466e-b00a-c75aa4976d0d" containerName="istio-proxy" Oct 31 04:27:50.314458 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:50.314437 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-edc450-openshift-default-84c757994f-q4vwz" Oct 31 04:27:50.338378 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:50.338351 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-edc450-openshift-default-84c757994f-q4vwz"] Oct 31 04:27:50.347122 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:50.347098 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-edc450-openshift-default-dockercfg-jl8g8\"" Oct 31 04:27:50.397970 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:50.397937 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0cc41ad-7f5b-4a5c-a989-bfafee0857e8-istiod-ca-cert\") pod \"e2e-praxis-b-edc450-openshift-default-84c757994f-q4vwz\" (UID: \"c0cc41ad-7f5b-4a5c-a989-bfafee0857e8\") " pod="openshift-ingress/e2e-praxis-b-edc450-openshift-default-84c757994f-q4vwz" Oct 31 04:27:50.398113 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:50.397986 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0cc41ad-7f5b-4a5c-a989-bfafee0857e8-istio-data\") pod \"e2e-praxis-b-edc450-openshift-default-84c757994f-q4vwz\" (UID: \"c0cc41ad-7f5b-4a5c-a989-bfafee0857e8\") " pod="openshift-ingress/e2e-praxis-b-edc450-openshift-default-84c757994f-q4vwz" Oct 31 04:27:50.398113 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:50.398076 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wm5l8\" (UniqueName: \"kubernetes.io/projected/c0cc41ad-7f5b-4a5c-a989-bfafee0857e8-kube-api-access-wm5l8\") pod \"e2e-praxis-b-edc450-openshift-default-84c757994f-q4vwz\" (UID: \"c0cc41ad-7f5b-4a5c-a989-bfafee0857e8\") " pod="openshift-ingress/e2e-praxis-b-edc450-openshift-default-84c757994f-q4vwz" Oct 31 04:27:50.398212 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:50.398196 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0cc41ad-7f5b-4a5c-a989-bfafee0857e8-istio-envoy\") pod \"e2e-praxis-b-edc450-openshift-default-84c757994f-q4vwz\" (UID: \"c0cc41ad-7f5b-4a5c-a989-bfafee0857e8\") " pod="openshift-ingress/e2e-praxis-b-edc450-openshift-default-84c757994f-q4vwz" Oct 31 04:27:50.398250 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:50.398236 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0cc41ad-7f5b-4a5c-a989-bfafee0857e8-workload-socket\") pod \"e2e-praxis-b-edc450-openshift-default-84c757994f-q4vwz\" (UID: \"c0cc41ad-7f5b-4a5c-a989-bfafee0857e8\") " pod="openshift-ingress/e2e-praxis-b-edc450-openshift-default-84c757994f-q4vwz" Oct 31 04:27:50.398337 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:50.398307 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0cc41ad-7f5b-4a5c-a989-bfafee0857e8-credential-socket\") pod \"e2e-praxis-b-edc450-openshift-default-84c757994f-q4vwz\" (UID: \"c0cc41ad-7f5b-4a5c-a989-bfafee0857e8\") " pod="openshift-ingress/e2e-praxis-b-edc450-openshift-default-84c757994f-q4vwz" Oct 31 04:27:50.398441 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:50.398427 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0cc41ad-7f5b-4a5c-a989-bfafee0857e8-istio-podinfo\") pod \"e2e-praxis-b-edc450-openshift-default-84c757994f-q4vwz\" (UID: \"c0cc41ad-7f5b-4a5c-a989-bfafee0857e8\") " pod="openshift-ingress/e2e-praxis-b-edc450-openshift-default-84c757994f-q4vwz" Oct 31 04:27:50.398478 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:50.398463 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0cc41ad-7f5b-4a5c-a989-bfafee0857e8-istio-token\") pod \"e2e-praxis-b-edc450-openshift-default-84c757994f-q4vwz\" (UID: \"c0cc41ad-7f5b-4a5c-a989-bfafee0857e8\") " pod="openshift-ingress/e2e-praxis-b-edc450-openshift-default-84c757994f-q4vwz" Oct 31 04:27:50.398512 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:50.398488 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0cc41ad-7f5b-4a5c-a989-bfafee0857e8-workload-certs\") pod \"e2e-praxis-b-edc450-openshift-default-84c757994f-q4vwz\" (UID: \"c0cc41ad-7f5b-4a5c-a989-bfafee0857e8\") " pod="openshift-ingress/e2e-praxis-b-edc450-openshift-default-84c757994f-q4vwz" Oct 31 04:27:50.565400 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:50.542791 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0cc41ad-7f5b-4a5c-a989-bfafee0857e8-istio-podinfo\") pod \"e2e-praxis-b-edc450-openshift-default-84c757994f-q4vwz\" (UID: \"c0cc41ad-7f5b-4a5c-a989-bfafee0857e8\") " pod="openshift-ingress/e2e-praxis-b-edc450-openshift-default-84c757994f-q4vwz" Oct 31 04:27:50.565729 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:50.565698 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0cc41ad-7f5b-4a5c-a989-bfafee0857e8-istio-token\") pod \"e2e-praxis-b-edc450-openshift-default-84c757994f-q4vwz\" (UID: \"c0cc41ad-7f5b-4a5c-a989-bfafee0857e8\") " pod="openshift-ingress/e2e-praxis-b-edc450-openshift-default-84c757994f-q4vwz" Oct 31 04:27:50.565870 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:50.565856 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0cc41ad-7f5b-4a5c-a989-bfafee0857e8-workload-certs\") pod \"e2e-praxis-b-edc450-openshift-default-84c757994f-q4vwz\" (UID: \"c0cc41ad-7f5b-4a5c-a989-bfafee0857e8\") " pod="openshift-ingress/e2e-praxis-b-edc450-openshift-default-84c757994f-q4vwz" Oct 31 04:27:50.566040 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:50.566026 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0cc41ad-7f5b-4a5c-a989-bfafee0857e8-istiod-ca-cert\") pod \"e2e-praxis-b-edc450-openshift-default-84c757994f-q4vwz\" (UID: \"c0cc41ad-7f5b-4a5c-a989-bfafee0857e8\") " pod="openshift-ingress/e2e-praxis-b-edc450-openshift-default-84c757994f-q4vwz" Oct 31 04:27:50.566177 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:50.566163 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0cc41ad-7f5b-4a5c-a989-bfafee0857e8-istio-data\") pod \"e2e-praxis-b-edc450-openshift-default-84c757994f-q4vwz\" (UID: \"c0cc41ad-7f5b-4a5c-a989-bfafee0857e8\") " pod="openshift-ingress/e2e-praxis-b-edc450-openshift-default-84c757994f-q4vwz" Oct 31 04:27:50.566305 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:50.566289 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wm5l8\" (UniqueName: \"kubernetes.io/projected/c0cc41ad-7f5b-4a5c-a989-bfafee0857e8-kube-api-access-wm5l8\") pod \"e2e-praxis-b-edc450-openshift-default-84c757994f-q4vwz\" (UID: \"c0cc41ad-7f5b-4a5c-a989-bfafee0857e8\") " pod="openshift-ingress/e2e-praxis-b-edc450-openshift-default-84c757994f-q4vwz" Oct 31 04:27:50.591630 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:50.591591 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0cc41ad-7f5b-4a5c-a989-bfafee0857e8-istio-envoy\") pod \"e2e-praxis-b-edc450-openshift-default-84c757994f-q4vwz\" (UID: \"c0cc41ad-7f5b-4a5c-a989-bfafee0857e8\") " pod="openshift-ingress/e2e-praxis-b-edc450-openshift-default-84c757994f-q4vwz" Oct 31 04:27:50.591903 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:50.591885 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0cc41ad-7f5b-4a5c-a989-bfafee0857e8-workload-socket\") pod \"e2e-praxis-b-edc450-openshift-default-84c757994f-q4vwz\" (UID: \"c0cc41ad-7f5b-4a5c-a989-bfafee0857e8\") " pod="openshift-ingress/e2e-praxis-b-edc450-openshift-default-84c757994f-q4vwz" Oct 31 04:27:50.592028 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:50.592013 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0cc41ad-7f5b-4a5c-a989-bfafee0857e8-credential-socket\") pod \"e2e-praxis-b-edc450-openshift-default-84c757994f-q4vwz\" (UID: \"c0cc41ad-7f5b-4a5c-a989-bfafee0857e8\") " pod="openshift-ingress/e2e-praxis-b-edc450-openshift-default-84c757994f-q4vwz" Oct 31 04:27:50.615350 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:50.592653 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0cc41ad-7f5b-4a5c-a989-bfafee0857e8-credential-socket\") pod \"e2e-praxis-b-edc450-openshift-default-84c757994f-q4vwz\" (UID: \"c0cc41ad-7f5b-4a5c-a989-bfafee0857e8\") " pod="openshift-ingress/e2e-praxis-b-edc450-openshift-default-84c757994f-q4vwz" Oct 31 04:27:50.616413 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:50.566947 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0cc41ad-7f5b-4a5c-a989-bfafee0857e8-workload-certs\") pod \"e2e-praxis-b-edc450-openshift-default-84c757994f-q4vwz\" (UID: \"c0cc41ad-7f5b-4a5c-a989-bfafee0857e8\") " pod="openshift-ingress/e2e-praxis-b-edc450-openshift-default-84c757994f-q4vwz" Oct 31 04:27:50.617009 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:50.606396 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0cc41ad-7f5b-4a5c-a989-bfafee0857e8-istio-envoy\") pod \"e2e-praxis-b-edc450-openshift-default-84c757994f-q4vwz\" (UID: \"c0cc41ad-7f5b-4a5c-a989-bfafee0857e8\") " pod="openshift-ingress/e2e-praxis-b-edc450-openshift-default-84c757994f-q4vwz" Oct 31 04:27:50.617128 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:50.573689 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0cc41ad-7f5b-4a5c-a989-bfafee0857e8-istio-data\") pod \"e2e-praxis-b-edc450-openshift-default-84c757994f-q4vwz\" (UID: \"c0cc41ad-7f5b-4a5c-a989-bfafee0857e8\") " pod="openshift-ingress/e2e-praxis-b-edc450-openshift-default-84c757994f-q4vwz" Oct 31 04:27:50.617217 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:50.606654 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0cc41ad-7f5b-4a5c-a989-bfafee0857e8-workload-socket\") pod \"e2e-praxis-b-edc450-openshift-default-84c757994f-q4vwz\" (UID: \"c0cc41ad-7f5b-4a5c-a989-bfafee0857e8\") " pod="openshift-ingress/e2e-praxis-b-edc450-openshift-default-84c757994f-q4vwz" Oct 31 04:27:50.617311 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:50.582722 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0cc41ad-7f5b-4a5c-a989-bfafee0857e8-istio-podinfo\") pod \"e2e-praxis-b-edc450-openshift-default-84c757994f-q4vwz\" (UID: \"c0cc41ad-7f5b-4a5c-a989-bfafee0857e8\") " pod="openshift-ingress/e2e-praxis-b-edc450-openshift-default-84c757994f-q4vwz" Oct 31 04:27:50.617423 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:50.616309 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0cc41ad-7f5b-4a5c-a989-bfafee0857e8-istiod-ca-cert\") pod \"e2e-praxis-b-edc450-openshift-default-84c757994f-q4vwz\" (UID: \"c0cc41ad-7f5b-4a5c-a989-bfafee0857e8\") " pod="openshift-ingress/e2e-praxis-b-edc450-openshift-default-84c757994f-q4vwz" Oct 31 04:27:50.666209 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:50.663083 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wm5l8\" (UniqueName: \"kubernetes.io/projected/c0cc41ad-7f5b-4a5c-a989-bfafee0857e8-kube-api-access-wm5l8\") pod \"e2e-praxis-b-edc450-openshift-default-84c757994f-q4vwz\" (UID: \"c0cc41ad-7f5b-4a5c-a989-bfafee0857e8\") " pod="openshift-ingress/e2e-praxis-b-edc450-openshift-default-84c757994f-q4vwz" Oct 31 04:27:50.666209 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:50.664187 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0cc41ad-7f5b-4a5c-a989-bfafee0857e8-istio-token\") pod \"e2e-praxis-b-edc450-openshift-default-84c757994f-q4vwz\" (UID: \"c0cc41ad-7f5b-4a5c-a989-bfafee0857e8\") " pod="openshift-ingress/e2e-praxis-b-edc450-openshift-default-84c757994f-q4vwz" Oct 31 04:27:50.937858 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:50.937816 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-edc450-openshift-default-84c757994f-q4vwz" Oct 31 04:27:51.263069 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:27:51.263037 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0cc41ad_7f5b_4a5c_a989_bfafee0857e8.slice/crio-5078d7ccfc62292358c6823e11662eeb291f4c0ebbec36a4ba2450b53b23f720 WatchSource:0}: Error finding container 5078d7ccfc62292358c6823e11662eeb291f4c0ebbec36a4ba2450b53b23f720: Status 404 returned error can't find the container with id 5078d7ccfc62292358c6823e11662eeb291f4c0ebbec36a4ba2450b53b23f720 Oct 31 04:27:51.264450 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:51.264425 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-edc450-openshift-default-84c757994f-q4vwz"] Oct 31 04:27:51.265829 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:51.265798 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:27:51.265911 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:51.265862 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:27:51.265911 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:51.265900 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:27:51.850254 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:51.850209 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-edc450-openshift-default-84c757994f-q4vwz" event={"ID":"c0cc41ad-7f5b-4a5c-a989-bfafee0857e8","Type":"ContainerStarted","Data":"28555215b5812fed9bd2c710a9a4ddd1010c7957faa478947a5a28fb3f010ad8"} Oct 31 04:27:51.850729 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:51.850342 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-edc450-openshift-default-84c757994f-q4vwz" event={"ID":"c0cc41ad-7f5b-4a5c-a989-bfafee0857e8","Type":"ContainerStarted","Data":"5078d7ccfc62292358c6823e11662eeb291f4c0ebbec36a4ba2450b53b23f720"} Oct 31 04:27:51.926046 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:51.925983 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-edc450-openshift-default-84c757994f-q4vwz" podStartSLOduration=1.9259676300000002 podStartE2EDuration="1.92596763s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:27:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:27:51.923626477 +0000 UTC m=+1419.637961094" watchObservedRunningTime="2026-10-31 04:27:51.92596763 +0000 UTC m=+1419.640302247" Oct 31 04:27:51.938830 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:51.938794 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-edc450-openshift-default-84c757994f-q4vwz" Oct 31 04:27:52.945631 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:52.945598 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-edc450-openshift-default-84c757994f-q4vwz" Oct 31 04:27:53.863368 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:53.863323 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-edc450-openshift-default-84c757994f-q4vwz" Oct 31 04:27:53.864246 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:53.864227 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-edc450-openshift-default-84c757994f-q4vwz" Oct 31 04:27:56.712100 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:56.712062 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-edc450-7b5666bb59-jnjrl"] Oct 31 04:27:56.712895 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:56.712874 4376 cpu_manager.go:414] "containerMap: removing container" podUID="fd4632c9-069d-466e-b00a-c75aa4976d0d" containerName="istio-proxy" Oct 31 04:27:56.712895 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:56.712895 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd4632c9-069d-466e-b00a-c75aa4976d0d" containerName="istio-proxy" Oct 31 04:27:56.717658 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:56.717636 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-edc450-7b5666bb59-jnjrl" Oct 31 04:27:56.733860 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:56.733834 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-edc450-7b5666bb59-jnjrl"] Oct 31 04:27:56.765506 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:56.765449 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-edc450\"" Oct 31 04:27:56.781132 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:56.781110 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-edc450-dockercfg-6kwjc\"" Oct 31 04:27:56.867352 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:56.846590 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6d87z\" (UniqueName: \"kubernetes.io/projected/03cd0818-43b9-4122-bb8b-d89063169326-kube-api-access-6d87z\") pod \"payload-pre-processing-e2e-praxis-b-edc450-7b5666bb59-jnjrl\" (UID: \"03cd0818-43b9-4122-bb8b-d89063169326\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-edc450-7b5666bb59-jnjrl" Oct 31 04:27:56.868206 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:56.868174 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03cd0818-43b9-4122-bb8b-d89063169326-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-edc450-7b5666bb59-jnjrl\" (UID: \"03cd0818-43b9-4122-bb8b-d89063169326\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-edc450-7b5666bb59-jnjrl" Oct 31 04:27:56.972863 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:56.972767 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6d87z\" (UniqueName: \"kubernetes.io/projected/03cd0818-43b9-4122-bb8b-d89063169326-kube-api-access-6d87z\") pod \"payload-pre-processing-e2e-praxis-b-edc450-7b5666bb59-jnjrl\" (UID: \"03cd0818-43b9-4122-bb8b-d89063169326\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-edc450-7b5666bb59-jnjrl" Oct 31 04:27:56.973204 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:56.973178 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03cd0818-43b9-4122-bb8b-d89063169326-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-edc450-7b5666bb59-jnjrl\" (UID: \"03cd0818-43b9-4122-bb8b-d89063169326\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-edc450-7b5666bb59-jnjrl" Oct 31 04:27:56.974512 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:56.974485 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03cd0818-43b9-4122-bb8b-d89063169326-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-edc450-7b5666bb59-jnjrl\" (UID: \"03cd0818-43b9-4122-bb8b-d89063169326\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-edc450-7b5666bb59-jnjrl" Oct 31 04:27:57.011243 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:57.011210 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6d87z\" (UniqueName: \"kubernetes.io/projected/03cd0818-43b9-4122-bb8b-d89063169326-kube-api-access-6d87z\") pod \"payload-pre-processing-e2e-praxis-b-edc450-7b5666bb59-jnjrl\" (UID: \"03cd0818-43b9-4122-bb8b-d89063169326\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-edc450-7b5666bb59-jnjrl" Oct 31 04:27:57.036000 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:57.034568 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-edc450-7b5666bb59-jnjrl" Oct 31 04:27:57.416361 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:57.416241 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-edc450-7b5666bb59-jnjrl"] Oct 31 04:27:57.914048 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:57.914012 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-edc450-7b5666bb59-jnjrl" event={"ID":"03cd0818-43b9-4122-bb8b-d89063169326","Type":"ContainerStarted","Data":"6b6191984193535b65425777c6b06d836aa1bbcb782e8a483c6fdd9cb0c115f8"} Oct 31 04:27:57.914461 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:57.914064 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-edc450-7b5666bb59-jnjrl" event={"ID":"03cd0818-43b9-4122-bb8b-d89063169326","Type":"ContainerStarted","Data":"b2b65f090320178f6c7dc7e01c0623dd504480ca82801ee81492f133e13aa698"} Oct 31 04:27:57.914461 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:57.914096 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-edc450-7b5666bb59-jnjrl" Oct 31 04:27:57.955981 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:57.955925 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-edc450-7b5666bb59-jnjrl" podStartSLOduration=1.95591106 podStartE2EDuration="1.95591106s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:27:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:27:57.954668566 +0000 UTC m=+1425.669003185" watchObservedRunningTime="2026-10-31 04:27:57.95591106 +0000 UTC m=+1425.670245677" Oct 31 04:27:58.234400 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:27:58.228063 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 04:28:01.279139 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:01.279107 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b8bb1e-84bc5d79f5-pw6x4"] Oct 31 04:28:01.279556 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:01.279362 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b8bb1e-84bc5d79f5-pw6x4" podUID="31582135-4095-446e-b948-86fe2c21278a" containerName="maas-api" containerID="cri-o://adbf07c1f6d0a310859fb6f4ef4521330ad3c879e1f8236f138a57755a88d929" gracePeriod=30 Oct 31 04:28:01.477863 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:01.477832 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b8bb1e-74c8bd9999-427ss"] Oct 31 04:28:01.479341 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:01.478783 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b8bb1e-74c8bd9999-427ss" podUID="d27ea922-cf10-4656-8166-3bb39619ae9a" containerName="payload-pre-processing" containerID="cri-o://ceed94c185cdc7020584ae174af089f4244de7f145f06d7a65c8a8d7a431f4b6" gracePeriod=30 Oct 31 04:28:01.630112 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:01.630047 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b8bb1e-84bc5d79f5-pw6x4" Oct 31 04:28:01.705090 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:01.704999 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/31582135-4095-446e-b948-86fe2c21278a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/31582135-4095-446e-b948-86fe2c21278a-maas-api-tls\") pod \"31582135-4095-446e-b948-86fe2c21278a\" (UID: \"31582135-4095-446e-b948-86fe2c21278a\") " Oct 31 04:28:01.705260 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:01.705133 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/31582135-4095-446e-b948-86fe2c21278a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/31582135-4095-446e-b948-86fe2c21278a-metrics-certs\") pod \"31582135-4095-446e-b948-86fe2c21278a\" (UID: \"31582135-4095-446e-b948-86fe2c21278a\") " Oct 31 04:28:01.705260 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:01.705180 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31582135-4095-446e-b948-86fe2c21278a-kube-api-access-xxn5h\" (UniqueName: \"kubernetes.io/projected/31582135-4095-446e-b948-86fe2c21278a-kube-api-access-xxn5h\") pod \"31582135-4095-446e-b948-86fe2c21278a\" (UID: \"31582135-4095-446e-b948-86fe2c21278a\") " Oct 31 04:28:01.705419 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:01.705304 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31582135-4095-446e-b948-86fe2c21278a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/31582135-4095-446e-b948-86fe2c21278a-openshift-service-ca\") pod \"31582135-4095-446e-b948-86fe2c21278a\" (UID: \"31582135-4095-446e-b948-86fe2c21278a\") " Oct 31 04:28:01.707237 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:01.706880 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31582135-4095-446e-b948-86fe2c21278a-openshift-service-ca" pod "31582135-4095-446e-b948-86fe2c21278a" (UID: "31582135-4095-446e-b948-86fe2c21278a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:28:01.713008 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:01.712969 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31582135-4095-446e-b948-86fe2c21278a-kube-api-access-xxn5h" pod "31582135-4095-446e-b948-86fe2c21278a" (UID: "31582135-4095-446e-b948-86fe2c21278a"). InnerVolumeSpecName "kube-api-access-xxn5h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:28:01.713008 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:01.712983 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31582135-4095-446e-b948-86fe2c21278a-maas-api-tls" pod "31582135-4095-446e-b948-86fe2c21278a" (UID: "31582135-4095-446e-b948-86fe2c21278a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:28:01.713215 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:01.713118 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31582135-4095-446e-b948-86fe2c21278a-metrics-certs" pod "31582135-4095-446e-b948-86fe2c21278a" (UID: "31582135-4095-446e-b948-86fe2c21278a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:28:01.804148 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:01.803854 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b8bb1e-74c8bd9999-427ss" Oct 31 04:28:01.834186 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:01.833409 4376 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/31582135-4095-446e-b948-86fe2c21278a-maas-api-tls\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:28:01.834186 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:01.833444 4376 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/31582135-4095-446e-b948-86fe2c21278a-metrics-certs\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:28:01.834186 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:01.833458 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxn5h\" (UniqueName: \"kubernetes.io/projected/31582135-4095-446e-b948-86fe2c21278a-kube-api-access-xxn5h\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:28:01.834186 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:01.833473 4376 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/31582135-4095-446e-b948-86fe2c21278a-openshift-service-ca\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:28:01.949362 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:01.948858 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d27ea922-cf10-4656-8166-3bb39619ae9a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d27ea922-cf10-4656-8166-3bb39619ae9a-plugins-config-volume\") pod \"d27ea922-cf10-4656-8166-3bb39619ae9a\" (UID: \"d27ea922-cf10-4656-8166-3bb39619ae9a\") " Oct 31 04:28:01.949362 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:01.949083 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d27ea922-cf10-4656-8166-3bb39619ae9a-kube-api-access-vk8tq\" (UniqueName: \"kubernetes.io/projected/d27ea922-cf10-4656-8166-3bb39619ae9a-kube-api-access-vk8tq\") pod \"d27ea922-cf10-4656-8166-3bb39619ae9a\" (UID: \"d27ea922-cf10-4656-8166-3bb39619ae9a\") " Oct 31 04:28:01.970350 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:01.963162 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d27ea922-cf10-4656-8166-3bb39619ae9a-plugins-config-volume" pod "d27ea922-cf10-4656-8166-3bb39619ae9a" (UID: "d27ea922-cf10-4656-8166-3bb39619ae9a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:28:02.043867 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:02.043174 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d27ea922-cf10-4656-8166-3bb39619ae9a-kube-api-access-vk8tq" pod "d27ea922-cf10-4656-8166-3bb39619ae9a" (UID: "d27ea922-cf10-4656-8166-3bb39619ae9a"). InnerVolumeSpecName "kube-api-access-vk8tq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:28:02.059390 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:02.054929 4376 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d27ea922-cf10-4656-8166-3bb39619ae9a-plugins-config-volume\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:28:02.059390 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:02.054967 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vk8tq\" (UniqueName: \"kubernetes.io/projected/d27ea922-cf10-4656-8166-3bb39619ae9a-kube-api-access-vk8tq\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:28:02.110707 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:02.110668 4376 generic.go:356] "Generic (PLEG): container finished" podID="d27ea922-cf10-4656-8166-3bb39619ae9a" containerID="ceed94c185cdc7020584ae174af089f4244de7f145f06d7a65c8a8d7a431f4b6" exitCode=0 Oct 31 04:28:02.110934 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:02.110906 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b8bb1e-74c8bd9999-427ss" event={"ID":"d27ea922-cf10-4656-8166-3bb39619ae9a","Type":"ContainerDied","Data":"ceed94c185cdc7020584ae174af089f4244de7f145f06d7a65c8a8d7a431f4b6"} Oct 31 04:28:02.111041 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:02.110973 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b8bb1e-74c8bd9999-427ss" event={"ID":"d27ea922-cf10-4656-8166-3bb39619ae9a","Type":"ContainerDied","Data":"6efa9bd8b941eb4d396bbc669e3576dce69c2203d8662605b3100572071fee92"} Oct 31 04:28:02.111041 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:02.111004 4376 scope.go:122] "RemoveContainer" containerID="ceed94c185cdc7020584ae174af089f4244de7f145f06d7a65c8a8d7a431f4b6" Oct 31 04:28:02.111191 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:02.111175 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b8bb1e-74c8bd9999-427ss" Oct 31 04:28:02.114220 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:02.114190 4376 generic.go:356] "Generic (PLEG): container finished" podID="31582135-4095-446e-b948-86fe2c21278a" containerID="adbf07c1f6d0a310859fb6f4ef4521330ad3c879e1f8236f138a57755a88d929" exitCode=0 Oct 31 04:28:02.114364 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:02.114267 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b8bb1e-84bc5d79f5-pw6x4" event={"ID":"31582135-4095-446e-b948-86fe2c21278a","Type":"ContainerDied","Data":"adbf07c1f6d0a310859fb6f4ef4521330ad3c879e1f8236f138a57755a88d929"} Oct 31 04:28:02.114449 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:02.114359 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b8bb1e-84bc5d79f5-pw6x4" event={"ID":"31582135-4095-446e-b948-86fe2c21278a","Type":"ContainerDied","Data":"38b761c3e17db80ec7ee5cad261a9ea973c11800354f8816b5fad723db032cf0"} Oct 31 04:28:02.114449 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:02.114415 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b8bb1e-84bc5d79f5-pw6x4" Oct 31 04:28:02.125398 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:02.125377 4376 scope.go:122] "RemoveContainer" containerID="ceed94c185cdc7020584ae174af089f4244de7f145f06d7a65c8a8d7a431f4b6" Oct 31 04:28:02.125844 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:28:02.125816 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ceed94c185cdc7020584ae174af089f4244de7f145f06d7a65c8a8d7a431f4b6\": container with ID starting with ceed94c185cdc7020584ae174af089f4244de7f145f06d7a65c8a8d7a431f4b6 not found: ID does not exist" containerID="ceed94c185cdc7020584ae174af089f4244de7f145f06d7a65c8a8d7a431f4b6" Oct 31 04:28:02.125941 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:02.125858 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ceed94c185cdc7020584ae174af089f4244de7f145f06d7a65c8a8d7a431f4b6"} err="failed to get container status \"ceed94c185cdc7020584ae174af089f4244de7f145f06d7a65c8a8d7a431f4b6\": rpc error: code = NotFound desc = could not find container \"ceed94c185cdc7020584ae174af089f4244de7f145f06d7a65c8a8d7a431f4b6\": container with ID starting with ceed94c185cdc7020584ae174af089f4244de7f145f06d7a65c8a8d7a431f4b6 not found: ID does not exist" Oct 31 04:28:02.125941 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:02.125885 4376 scope.go:122] "RemoveContainer" containerID="adbf07c1f6d0a310859fb6f4ef4521330ad3c879e1f8236f138a57755a88d929" Oct 31 04:28:02.136000 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:02.135978 4376 scope.go:122] "RemoveContainer" containerID="adbf07c1f6d0a310859fb6f4ef4521330ad3c879e1f8236f138a57755a88d929" Oct 31 04:28:02.136314 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:28:02.136297 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"adbf07c1f6d0a310859fb6f4ef4521330ad3c879e1f8236f138a57755a88d929\": container with ID starting with adbf07c1f6d0a310859fb6f4ef4521330ad3c879e1f8236f138a57755a88d929 not found: ID does not exist" containerID="adbf07c1f6d0a310859fb6f4ef4521330ad3c879e1f8236f138a57755a88d929" Oct 31 04:28:02.136379 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:02.136322 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"adbf07c1f6d0a310859fb6f4ef4521330ad3c879e1f8236f138a57755a88d929"} err="failed to get container status \"adbf07c1f6d0a310859fb6f4ef4521330ad3c879e1f8236f138a57755a88d929\": rpc error: code = NotFound desc = could not find container \"adbf07c1f6d0a310859fb6f4ef4521330ad3c879e1f8236f138a57755a88d929\": container with ID starting with adbf07c1f6d0a310859fb6f4ef4521330ad3c879e1f8236f138a57755a88d929 not found: ID does not exist" Oct 31 04:28:02.179172 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:02.179133 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b8bb1e-74c8bd9999-427ss"] Oct 31 04:28:02.191600 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:02.191565 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b8bb1e-74c8bd9999-427ss"] Oct 31 04:28:02.217835 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:02.217755 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b8bb1e-84bc5d79f5-pw6x4"] Oct 31 04:28:02.220694 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:02.220672 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b8bb1e-84bc5d79f5-pw6x4"] Oct 31 04:28:02.464466 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:02.464423 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31582135-4095-446e-b948-86fe2c21278a" path="/var/lib/kubelet/pods/31582135-4095-446e-b948-86fe2c21278a/volumes" Oct 31 04:28:02.465018 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:02.464994 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d27ea922-cf10-4656-8166-3bb39619ae9a" path="/var/lib/kubelet/pods/d27ea922-cf10-4656-8166-3bb39619ae9a/volumes" Oct 31 04:28:04.435319 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:04.435287 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-81c810-openshift-default-5fbd5b64c7-kbvbk"] Oct 31 04:28:04.435940 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:04.435923 4376 cpu_manager.go:414] "containerMap: removing container" podUID="d27ea922-cf10-4656-8166-3bb39619ae9a" containerName="payload-pre-processing" Oct 31 04:28:04.436004 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:04.435941 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d27ea922-cf10-4656-8166-3bb39619ae9a" containerName="payload-pre-processing" Oct 31 04:28:04.436004 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:04.435959 4376 cpu_manager.go:414] "containerMap: removing container" podUID="31582135-4095-446e-b948-86fe2c21278a" containerName="maas-api" Oct 31 04:28:04.436004 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:04.435965 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31582135-4095-446e-b948-86fe2c21278a" containerName="maas-api" Oct 31 04:28:04.436166 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:04.436092 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="d27ea922-cf10-4656-8166-3bb39619ae9a" containerName="payload-pre-processing" Oct 31 04:28:04.436166 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:04.436107 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="31582135-4095-446e-b948-86fe2c21278a" containerName="maas-api" Oct 31 04:28:04.463352 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:04.460483 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-81c810-openshift-default-5fbd5b64c7-kbvbk" Oct 31 04:28:04.483214 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:04.482516 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-81c810-openshift-default-dockercfg-bf77l\"" Oct 31 04:28:04.488052 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:04.487940 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-81c810-openshift-default-5fbd5b64c7-kbvbk"] Oct 31 04:28:04.606214 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:04.606181 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c34e357-743e-442a-957d-788f1618aaf3-istio-token\") pod \"e2e-praxis-b-81c810-openshift-default-5fbd5b64c7-kbvbk\" (UID: \"3c34e357-743e-442a-957d-788f1618aaf3\") " pod="openshift-ingress/e2e-praxis-b-81c810-openshift-default-5fbd5b64c7-kbvbk" Oct 31 04:28:04.606421 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:04.606256 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c34e357-743e-442a-957d-788f1618aaf3-istio-podinfo\") pod \"e2e-praxis-b-81c810-openshift-default-5fbd5b64c7-kbvbk\" (UID: \"3c34e357-743e-442a-957d-788f1618aaf3\") " pod="openshift-ingress/e2e-praxis-b-81c810-openshift-default-5fbd5b64c7-kbvbk" Oct 31 04:28:04.606421 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:04.606370 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c34e357-743e-442a-957d-788f1618aaf3-workload-certs\") pod \"e2e-praxis-b-81c810-openshift-default-5fbd5b64c7-kbvbk\" (UID: \"3c34e357-743e-442a-957d-788f1618aaf3\") " pod="openshift-ingress/e2e-praxis-b-81c810-openshift-default-5fbd5b64c7-kbvbk" Oct 31 04:28:04.606421 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:04.606411 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c34e357-743e-442a-957d-788f1618aaf3-istio-data\") pod \"e2e-praxis-b-81c810-openshift-default-5fbd5b64c7-kbvbk\" (UID: \"3c34e357-743e-442a-957d-788f1618aaf3\") " pod="openshift-ingress/e2e-praxis-b-81c810-openshift-default-5fbd5b64c7-kbvbk" Oct 31 04:28:04.606787 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:04.606495 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c34e357-743e-442a-957d-788f1618aaf3-istiod-ca-cert\") pod \"e2e-praxis-b-81c810-openshift-default-5fbd5b64c7-kbvbk\" (UID: \"3c34e357-743e-442a-957d-788f1618aaf3\") " pod="openshift-ingress/e2e-praxis-b-81c810-openshift-default-5fbd5b64c7-kbvbk" Oct 31 04:28:04.606787 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:04.606530 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7mbq\" (UniqueName: \"kubernetes.io/projected/3c34e357-743e-442a-957d-788f1618aaf3-kube-api-access-k7mbq\") pod \"e2e-praxis-b-81c810-openshift-default-5fbd5b64c7-kbvbk\" (UID: \"3c34e357-743e-442a-957d-788f1618aaf3\") " pod="openshift-ingress/e2e-praxis-b-81c810-openshift-default-5fbd5b64c7-kbvbk" Oct 31 04:28:04.606787 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:04.606598 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c34e357-743e-442a-957d-788f1618aaf3-credential-socket\") pod \"e2e-praxis-b-81c810-openshift-default-5fbd5b64c7-kbvbk\" (UID: \"3c34e357-743e-442a-957d-788f1618aaf3\") " pod="openshift-ingress/e2e-praxis-b-81c810-openshift-default-5fbd5b64c7-kbvbk" Oct 31 04:28:04.606787 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:04.606683 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c34e357-743e-442a-957d-788f1618aaf3-workload-socket\") pod \"e2e-praxis-b-81c810-openshift-default-5fbd5b64c7-kbvbk\" (UID: \"3c34e357-743e-442a-957d-788f1618aaf3\") " pod="openshift-ingress/e2e-praxis-b-81c810-openshift-default-5fbd5b64c7-kbvbk" Oct 31 04:28:04.606787 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:04.606727 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c34e357-743e-442a-957d-788f1618aaf3-istio-envoy\") pod \"e2e-praxis-b-81c810-openshift-default-5fbd5b64c7-kbvbk\" (UID: \"3c34e357-743e-442a-957d-788f1618aaf3\") " pod="openshift-ingress/e2e-praxis-b-81c810-openshift-default-5fbd5b64c7-kbvbk" Oct 31 04:28:04.709398 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:04.709198 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c34e357-743e-442a-957d-788f1618aaf3-workload-socket\") pod \"e2e-praxis-b-81c810-openshift-default-5fbd5b64c7-kbvbk\" (UID: \"3c34e357-743e-442a-957d-788f1618aaf3\") " pod="openshift-ingress/e2e-praxis-b-81c810-openshift-default-5fbd5b64c7-kbvbk" Oct 31 04:28:04.709398 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:04.709291 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c34e357-743e-442a-957d-788f1618aaf3-istio-envoy\") pod \"e2e-praxis-b-81c810-openshift-default-5fbd5b64c7-kbvbk\" (UID: \"3c34e357-743e-442a-957d-788f1618aaf3\") " pod="openshift-ingress/e2e-praxis-b-81c810-openshift-default-5fbd5b64c7-kbvbk" Oct 31 04:28:04.709398 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:04.709397 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c34e357-743e-442a-957d-788f1618aaf3-istio-token\") pod \"e2e-praxis-b-81c810-openshift-default-5fbd5b64c7-kbvbk\" (UID: \"3c34e357-743e-442a-957d-788f1618aaf3\") " pod="openshift-ingress/e2e-praxis-b-81c810-openshift-default-5fbd5b64c7-kbvbk" Oct 31 04:28:04.709698 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:04.709488 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c34e357-743e-442a-957d-788f1618aaf3-istio-podinfo\") pod \"e2e-praxis-b-81c810-openshift-default-5fbd5b64c7-kbvbk\" (UID: \"3c34e357-743e-442a-957d-788f1618aaf3\") " pod="openshift-ingress/e2e-praxis-b-81c810-openshift-default-5fbd5b64c7-kbvbk" Oct 31 04:28:04.709698 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:04.709548 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c34e357-743e-442a-957d-788f1618aaf3-workload-certs\") pod \"e2e-praxis-b-81c810-openshift-default-5fbd5b64c7-kbvbk\" (UID: \"3c34e357-743e-442a-957d-788f1618aaf3\") " pod="openshift-ingress/e2e-praxis-b-81c810-openshift-default-5fbd5b64c7-kbvbk" Oct 31 04:28:04.709698 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:04.709587 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c34e357-743e-442a-957d-788f1618aaf3-istio-data\") pod \"e2e-praxis-b-81c810-openshift-default-5fbd5b64c7-kbvbk\" (UID: \"3c34e357-743e-442a-957d-788f1618aaf3\") " pod="openshift-ingress/e2e-praxis-b-81c810-openshift-default-5fbd5b64c7-kbvbk" Oct 31 04:28:04.709698 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:04.709650 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c34e357-743e-442a-957d-788f1618aaf3-istiod-ca-cert\") pod \"e2e-praxis-b-81c810-openshift-default-5fbd5b64c7-kbvbk\" (UID: \"3c34e357-743e-442a-957d-788f1618aaf3\") " pod="openshift-ingress/e2e-praxis-b-81c810-openshift-default-5fbd5b64c7-kbvbk" Oct 31 04:28:04.709919 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:04.709698 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7mbq\" (UniqueName: \"kubernetes.io/projected/3c34e357-743e-442a-957d-788f1618aaf3-kube-api-access-k7mbq\") pod \"e2e-praxis-b-81c810-openshift-default-5fbd5b64c7-kbvbk\" (UID: \"3c34e357-743e-442a-957d-788f1618aaf3\") " pod="openshift-ingress/e2e-praxis-b-81c810-openshift-default-5fbd5b64c7-kbvbk" Oct 31 04:28:04.709919 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:04.709765 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c34e357-743e-442a-957d-788f1618aaf3-credential-socket\") pod \"e2e-praxis-b-81c810-openshift-default-5fbd5b64c7-kbvbk\" (UID: \"3c34e357-743e-442a-957d-788f1618aaf3\") " pod="openshift-ingress/e2e-praxis-b-81c810-openshift-default-5fbd5b64c7-kbvbk" Oct 31 04:28:04.734386 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:04.710297 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c34e357-743e-442a-957d-788f1618aaf3-credential-socket\") pod \"e2e-praxis-b-81c810-openshift-default-5fbd5b64c7-kbvbk\" (UID: \"3c34e357-743e-442a-957d-788f1618aaf3\") " pod="openshift-ingress/e2e-praxis-b-81c810-openshift-default-5fbd5b64c7-kbvbk" Oct 31 04:28:04.734386 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:04.710575 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c34e357-743e-442a-957d-788f1618aaf3-workload-socket\") pod \"e2e-praxis-b-81c810-openshift-default-5fbd5b64c7-kbvbk\" (UID: \"3c34e357-743e-442a-957d-788f1618aaf3\") " pod="openshift-ingress/e2e-praxis-b-81c810-openshift-default-5fbd5b64c7-kbvbk" Oct 31 04:28:04.751402 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:04.749776 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c34e357-743e-442a-957d-788f1618aaf3-workload-certs\") pod \"e2e-praxis-b-81c810-openshift-default-5fbd5b64c7-kbvbk\" (UID: \"3c34e357-743e-442a-957d-788f1618aaf3\") " pod="openshift-ingress/e2e-praxis-b-81c810-openshift-default-5fbd5b64c7-kbvbk" Oct 31 04:28:04.752085 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:04.752053 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c34e357-743e-442a-957d-788f1618aaf3-istiod-ca-cert\") pod \"e2e-praxis-b-81c810-openshift-default-5fbd5b64c7-kbvbk\" (UID: \"3c34e357-743e-442a-957d-788f1618aaf3\") " pod="openshift-ingress/e2e-praxis-b-81c810-openshift-default-5fbd5b64c7-kbvbk" Oct 31 04:28:04.806353 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:04.752524 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c34e357-743e-442a-957d-788f1618aaf3-istio-data\") pod \"e2e-praxis-b-81c810-openshift-default-5fbd5b64c7-kbvbk\" (UID: \"3c34e357-743e-442a-957d-788f1618aaf3\") " pod="openshift-ingress/e2e-praxis-b-81c810-openshift-default-5fbd5b64c7-kbvbk" Oct 31 04:28:04.807275 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:04.759649 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c34e357-743e-442a-957d-788f1618aaf3-istio-envoy\") pod \"e2e-praxis-b-81c810-openshift-default-5fbd5b64c7-kbvbk\" (UID: \"3c34e357-743e-442a-957d-788f1618aaf3\") " pod="openshift-ingress/e2e-praxis-b-81c810-openshift-default-5fbd5b64c7-kbvbk" Oct 31 04:28:04.807494 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:04.807200 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c34e357-743e-442a-957d-788f1618aaf3-istio-podinfo\") pod \"e2e-praxis-b-81c810-openshift-default-5fbd5b64c7-kbvbk\" (UID: \"3c34e357-743e-442a-957d-788f1618aaf3\") " pod="openshift-ingress/e2e-praxis-b-81c810-openshift-default-5fbd5b64c7-kbvbk" Oct 31 04:28:04.835174 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:04.835140 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7mbq\" (UniqueName: \"kubernetes.io/projected/3c34e357-743e-442a-957d-788f1618aaf3-kube-api-access-k7mbq\") pod \"e2e-praxis-b-81c810-openshift-default-5fbd5b64c7-kbvbk\" (UID: \"3c34e357-743e-442a-957d-788f1618aaf3\") " pod="openshift-ingress/e2e-praxis-b-81c810-openshift-default-5fbd5b64c7-kbvbk" Oct 31 04:28:04.836292 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:04.836265 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c34e357-743e-442a-957d-788f1618aaf3-istio-token\") pod \"e2e-praxis-b-81c810-openshift-default-5fbd5b64c7-kbvbk\" (UID: \"3c34e357-743e-442a-957d-788f1618aaf3\") " pod="openshift-ingress/e2e-praxis-b-81c810-openshift-default-5fbd5b64c7-kbvbk" Oct 31 04:28:05.099109 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:05.099025 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-81c810-openshift-default-5fbd5b64c7-kbvbk" Oct 31 04:28:05.451737 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:05.451664 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-81c810-openshift-default-5fbd5b64c7-kbvbk"] Oct 31 04:28:05.452511 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:28:05.452486 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c34e357_743e_442a_957d_788f1618aaf3.slice/crio-e7eb828a5b246059d3c1eff7c4d034e07c711d818749a272acc813193df06fe3 WatchSource:0}: Error finding container e7eb828a5b246059d3c1eff7c4d034e07c711d818749a272acc813193df06fe3: Status 404 returned error can't find the container with id e7eb828a5b246059d3c1eff7c4d034e07c711d818749a272acc813193df06fe3 Oct 31 04:28:05.455482 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:05.455452 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:28:05.455560 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:05.455517 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:28:05.455560 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:05.455545 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:28:06.146385 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:06.146350 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-81c810-openshift-default-5fbd5b64c7-kbvbk" event={"ID":"3c34e357-743e-442a-957d-788f1618aaf3","Type":"ContainerStarted","Data":"a0c0db5b3fe8488903d4b2f1b246295390bce9fbf6fe570b7d7cd55ebe738ca1"} Oct 31 04:28:06.146560 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:06.146402 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-81c810-openshift-default-5fbd5b64c7-kbvbk" event={"ID":"3c34e357-743e-442a-957d-788f1618aaf3","Type":"ContainerStarted","Data":"e7eb828a5b246059d3c1eff7c4d034e07c711d818749a272acc813193df06fe3"} Oct 31 04:28:06.216892 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:06.216833 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-81c810-openshift-default-5fbd5b64c7-kbvbk" podStartSLOduration=2.21682055 podStartE2EDuration="2.21682055s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:28:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:28:06.203777555 +0000 UTC m=+1433.918112174" watchObservedRunningTime="2026-10-31 04:28:06.21682055 +0000 UTC m=+1433.931155167" Oct 31 04:28:07.099831 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:07.099792 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-81c810-openshift-default-5fbd5b64c7-kbvbk" Oct 31 04:28:07.105914 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:07.105882 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-81c810-openshift-default-5fbd5b64c7-kbvbk" Oct 31 04:28:07.158926 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:07.158897 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-81c810-openshift-default-5fbd5b64c7-kbvbk" Oct 31 04:28:07.159787 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:07.159764 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-81c810-openshift-default-5fbd5b64c7-kbvbk" Oct 31 04:28:08.978554 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:08.978523 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-edc450-7b5666bb59-jnjrl" Oct 31 04:28:10.181168 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:10.181135 4376 generic.go:356] "Generic (PLEG): container finished" podID="c9240e5d-789e-4ebf-8cd3-e7807036e8a7" containerID="256c492ec07aeb1473d776dac5053a9be1813e76d7b35ba430f04da83e72bb7b" exitCode=0 Oct 31 04:28:10.181606 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:10.181209 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-d1d441-openshift-default-c79997b97-84mqf" event={"ID":"c9240e5d-789e-4ebf-8cd3-e7807036e8a7","Type":"ContainerDied","Data":"256c492ec07aeb1473d776dac5053a9be1813e76d7b35ba430f04da83e72bb7b"} Oct 31 04:28:10.181680 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:10.181664 4376 scope.go:122] "RemoveContainer" containerID="256c492ec07aeb1473d776dac5053a9be1813e76d7b35ba430f04da83e72bb7b" Oct 31 04:28:10.183272 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:10.183243 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:28:10.183347 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:10.183315 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:28:10.183436 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:10.183368 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:28:10.510653 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:10.510574 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-d1d441-openshift-default-c79997b97-84mqf" Oct 31 04:28:10.725363 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:10.725228 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-81c810-5749ff85ff-5fr52"] Oct 31 04:28:10.742378 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:10.739547 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-81c810-5749ff85ff-5fr52"] Oct 31 04:28:10.742378 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:10.739678 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-81c810-5749ff85ff-5fr52" Oct 31 04:28:10.775042 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:10.768840 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-81c810-dockercfg-jfssc\"" Oct 31 04:28:10.775042 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:10.768956 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-81c810\"" Oct 31 04:28:10.905130 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:10.905091 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2e98cfe-b0c2-420e-8b92-540de74cd07f-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-81c810-5749ff85ff-5fr52\" (UID: \"c2e98cfe-b0c2-420e-8b92-540de74cd07f\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-81c810-5749ff85ff-5fr52" Oct 31 04:28:10.905684 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:10.905295 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pl7z7\" (UniqueName: \"kubernetes.io/projected/c2e98cfe-b0c2-420e-8b92-540de74cd07f-kube-api-access-pl7z7\") pod \"payload-processing-e2e-praxis-b-81c810-5749ff85ff-5fr52\" (UID: \"c2e98cfe-b0c2-420e-8b92-540de74cd07f\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-81c810-5749ff85ff-5fr52" Oct 31 04:28:11.001492 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:28:11.001452 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 04:28:11.006371 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:11.006348 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pl7z7\" (UniqueName: \"kubernetes.io/projected/c2e98cfe-b0c2-420e-8b92-540de74cd07f-kube-api-access-pl7z7\") pod \"payload-processing-e2e-praxis-b-81c810-5749ff85ff-5fr52\" (UID: \"c2e98cfe-b0c2-420e-8b92-540de74cd07f\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-81c810-5749ff85ff-5fr52" Oct 31 04:28:11.006511 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:11.006497 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2e98cfe-b0c2-420e-8b92-540de74cd07f-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-81c810-5749ff85ff-5fr52\" (UID: \"c2e98cfe-b0c2-420e-8b92-540de74cd07f\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-81c810-5749ff85ff-5fr52" Oct 31 04:28:11.007156 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:11.007128 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2e98cfe-b0c2-420e-8b92-540de74cd07f-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-81c810-5749ff85ff-5fr52\" (UID: \"c2e98cfe-b0c2-420e-8b92-540de74cd07f\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-81c810-5749ff85ff-5fr52" Oct 31 04:28:11.030709 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:11.030641 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pl7z7\" (UniqueName: \"kubernetes.io/projected/c2e98cfe-b0c2-420e-8b92-540de74cd07f-kube-api-access-pl7z7\") pod \"payload-processing-e2e-praxis-b-81c810-5749ff85ff-5fr52\" (UID: \"c2e98cfe-b0c2-420e-8b92-540de74cd07f\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-81c810-5749ff85ff-5fr52" Oct 31 04:28:11.165737 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:11.165704 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-81c810-5749ff85ff-5fr52" Oct 31 04:28:11.191068 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:11.191037 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-d1d441-openshift-default-c79997b97-84mqf" event={"ID":"c9240e5d-789e-4ebf-8cd3-e7807036e8a7","Type":"ContainerStarted","Data":"921c2507634db695656d081bd28dd097aa6799a82fd730ce384a56c618ea72e8"} Oct 31 04:28:11.296090 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:11.296013 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-d1d441-openshift-default-c79997b97-84mqf" Oct 31 04:28:11.296736 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:11.296709 4376 patch_prober.go:29] interesting pod/e2e-praxis-a-d1d441-openshift-default-c79997b97-84mqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body= Oct 31 04:28:11.296840 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:11.296757 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-d1d441-openshift-default-c79997b97-84mqf" podUID="c9240e5d-789e-4ebf-8cd3-e7807036e8a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" Oct 31 04:28:11.472582 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:28:11.472538 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2e98cfe_b0c2_420e_8b92_540de74cd07f.slice/crio-3e812823b3379a76df8cac91a73162bdbfe7ffdcdf8f2dba0990c2008276c978 WatchSource:0}: Error finding container 3e812823b3379a76df8cac91a73162bdbfe7ffdcdf8f2dba0990c2008276c978: Status 404 returned error can't find the container with id 3e812823b3379a76df8cac91a73162bdbfe7ffdcdf8f2dba0990c2008276c978 Oct 31 04:28:11.473277 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:11.473252 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-81c810-5749ff85ff-5fr52"] Oct 31 04:28:12.207516 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:12.207477 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-81c810-5749ff85ff-5fr52" event={"ID":"c2e98cfe-b0c2-420e-8b92-540de74cd07f","Type":"ContainerStarted","Data":"ba3dcb8b2618c605e83785f85594c60ad78cb2266701d357c65940ce4110001d"} Oct 31 04:28:12.207922 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:12.207542 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-81c810-5749ff85ff-5fr52" event={"ID":"c2e98cfe-b0c2-420e-8b92-540de74cd07f","Type":"ContainerStarted","Data":"3e812823b3379a76df8cac91a73162bdbfe7ffdcdf8f2dba0990c2008276c978"} Oct 31 04:28:12.207922 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:12.207576 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-81c810-5749ff85ff-5fr52" Oct 31 04:28:12.262353 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:12.254001 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-81c810-5749ff85ff-5fr52" podStartSLOduration=2.25397952 podStartE2EDuration="2.25397952s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:28:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:28:12.251729716 +0000 UTC m=+1439.966064353" watchObservedRunningTime="2026-10-31 04:28:12.25397952 +0000 UTC m=+1439.968314141" Oct 31 04:28:12.295317 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:12.295284 4376 patch_prober.go:29] interesting pod/e2e-praxis-a-d1d441-openshift-default-c79997b97-84mqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body= Oct 31 04:28:12.295540 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:12.295356 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-d1d441-openshift-default-c79997b97-84mqf" podUID="c9240e5d-789e-4ebf-8cd3-e7807036e8a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" Oct 31 04:28:13.296803 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:13.296756 4376 patch_prober.go:29] interesting pod/e2e-praxis-a-d1d441-openshift-default-c79997b97-84mqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body= Oct 31 04:28:13.298275 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:13.298237 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-d1d441-openshift-default-c79997b97-84mqf" podUID="c9240e5d-789e-4ebf-8cd3-e7807036e8a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" Oct 31 04:28:13.321137 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:13.321086 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-b8bb1e-openshift-default-557bbf897c-phbbm"] Oct 31 04:28:13.321994 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:13.321926 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-b8bb1e-openshift-default-557bbf897c-phbbm" podUID="61a59ba0-c79d-4942-8b8a-1493cb9e9182" containerName="istio-proxy" containerID="cri-o://61198db9bad5b173ddd7905ca76f72f06a462385c41b993e199df5c09cd97cb0" gracePeriod=30 Oct 31 04:28:14.295392 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:14.295355 4376 patch_prober.go:29] interesting pod/e2e-praxis-a-d1d441-openshift-default-c79997b97-84mqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body= Oct 31 04:28:14.295594 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:14.295410 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-d1d441-openshift-default-c79997b97-84mqf" podUID="c9240e5d-789e-4ebf-8cd3-e7807036e8a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" Oct 31 04:28:15.295643 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:15.295609 4376 patch_prober.go:29] interesting pod/e2e-praxis-a-d1d441-openshift-default-c79997b97-84mqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body= Oct 31 04:28:15.296048 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:15.295658 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-d1d441-openshift-default-c79997b97-84mqf" podUID="c9240e5d-789e-4ebf-8cd3-e7807036e8a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" Oct 31 04:28:16.296030 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:16.295990 4376 patch_prober.go:29] interesting pod/e2e-praxis-a-d1d441-openshift-default-c79997b97-84mqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body= Oct 31 04:28:16.296474 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:16.296048 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-d1d441-openshift-default-c79997b97-84mqf" podUID="c9240e5d-789e-4ebf-8cd3-e7807036e8a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" Oct 31 04:28:17.331867 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:17.331832 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-d1d441-openshift-default-c79997b97-84mqf" Oct 31 04:28:17.333206 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:17.333180 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-d1d441-openshift-default-c79997b97-84mqf" Oct 31 04:28:17.334677 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:17.334646 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-d1d441-openshift-default-c79997b97-84mqf" Oct 31 04:28:18.594848 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:18.594821 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-b8bb1e-openshift-default-557bbf897c-phbbm" Oct 31 04:28:18.713656 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:18.713625 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61a59ba0-c79d-4942-8b8a-1493cb9e9182-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61a59ba0-c79d-4942-8b8a-1493cb9e9182-credential-socket\") pod \"61a59ba0-c79d-4942-8b8a-1493cb9e9182\" (UID: \"61a59ba0-c79d-4942-8b8a-1493cb9e9182\") " Oct 31 04:28:18.713656 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:18.713661 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61a59ba0-c79d-4942-8b8a-1493cb9e9182-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61a59ba0-c79d-4942-8b8a-1493cb9e9182-istiod-ca-cert\") pod \"61a59ba0-c79d-4942-8b8a-1493cb9e9182\" (UID: \"61a59ba0-c79d-4942-8b8a-1493cb9e9182\") " Oct 31 04:28:18.713897 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:18.713684 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61a59ba0-c79d-4942-8b8a-1493cb9e9182-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61a59ba0-c79d-4942-8b8a-1493cb9e9182-workload-socket\") pod \"61a59ba0-c79d-4942-8b8a-1493cb9e9182\" (UID: \"61a59ba0-c79d-4942-8b8a-1493cb9e9182\") " Oct 31 04:28:18.713897 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:18.713774 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61a59ba0-c79d-4942-8b8a-1493cb9e9182-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61a59ba0-c79d-4942-8b8a-1493cb9e9182-istio-envoy\") pod \"61a59ba0-c79d-4942-8b8a-1493cb9e9182\" (UID: \"61a59ba0-c79d-4942-8b8a-1493cb9e9182\") " Oct 31 04:28:18.713897 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:18.713853 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61a59ba0-c79d-4942-8b8a-1493cb9e9182-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61a59ba0-c79d-4942-8b8a-1493cb9e9182-istio-data\") pod \"61a59ba0-c79d-4942-8b8a-1493cb9e9182\" (UID: \"61a59ba0-c79d-4942-8b8a-1493cb9e9182\") " Oct 31 04:28:18.714053 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:18.713911 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61a59ba0-c79d-4942-8b8a-1493cb9e9182-istio-token\" (UniqueName: \"kubernetes.io/projected/61a59ba0-c79d-4942-8b8a-1493cb9e9182-istio-token\") pod \"61a59ba0-c79d-4942-8b8a-1493cb9e9182\" (UID: \"61a59ba0-c79d-4942-8b8a-1493cb9e9182\") " Oct 31 04:28:18.714053 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:18.713936 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61a59ba0-c79d-4942-8b8a-1493cb9e9182-credential-socket" pod "61a59ba0-c79d-4942-8b8a-1493cb9e9182" (UID: "61a59ba0-c79d-4942-8b8a-1493cb9e9182"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:28:18.714053 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:18.714010 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61a59ba0-c79d-4942-8b8a-1493cb9e9182-kube-api-access-48gpj\" (UniqueName: \"kubernetes.io/projected/61a59ba0-c79d-4942-8b8a-1493cb9e9182-kube-api-access-48gpj\") pod \"61a59ba0-c79d-4942-8b8a-1493cb9e9182\" (UID: \"61a59ba0-c79d-4942-8b8a-1493cb9e9182\") " Oct 31 04:28:18.714221 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:18.714067 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61a59ba0-c79d-4942-8b8a-1493cb9e9182-workload-socket" pod "61a59ba0-c79d-4942-8b8a-1493cb9e9182" (UID: "61a59ba0-c79d-4942-8b8a-1493cb9e9182"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:28:18.714221 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:18.714084 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61a59ba0-c79d-4942-8b8a-1493cb9e9182-istiod-ca-cert" pod "61a59ba0-c79d-4942-8b8a-1493cb9e9182" (UID: "61a59ba0-c79d-4942-8b8a-1493cb9e9182"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:28:18.714221 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:18.714151 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/61a59ba0-c79d-4942-8b8a-1493cb9e9182-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61a59ba0-c79d-4942-8b8a-1493cb9e9182-istio-podinfo\") pod \"61a59ba0-c79d-4942-8b8a-1493cb9e9182\" (UID: \"61a59ba0-c79d-4942-8b8a-1493cb9e9182\") " Oct 31 04:28:18.714221 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:18.714178 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61a59ba0-c79d-4942-8b8a-1493cb9e9182-istio-data" pod "61a59ba0-c79d-4942-8b8a-1493cb9e9182" (UID: "61a59ba0-c79d-4942-8b8a-1493cb9e9182"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:28:18.714438 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:18.714312 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61a59ba0-c79d-4942-8b8a-1493cb9e9182-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61a59ba0-c79d-4942-8b8a-1493cb9e9182-workload-certs\") pod \"61a59ba0-c79d-4942-8b8a-1493cb9e9182\" (UID: \"61a59ba0-c79d-4942-8b8a-1493cb9e9182\") " Oct 31 04:28:18.714544 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:18.714526 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61a59ba0-c79d-4942-8b8a-1493cb9e9182-workload-certs" pod "61a59ba0-c79d-4942-8b8a-1493cb9e9182" (UID: "61a59ba0-c79d-4942-8b8a-1493cb9e9182"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:28:18.715041 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:18.715017 4376 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61a59ba0-c79d-4942-8b8a-1493cb9e9182-workload-certs\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:28:18.715107 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:18.715048 4376 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61a59ba0-c79d-4942-8b8a-1493cb9e9182-credential-socket\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:28:18.715107 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:18.715062 4376 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61a59ba0-c79d-4942-8b8a-1493cb9e9182-istiod-ca-cert\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:28:18.715107 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:18.715076 4376 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61a59ba0-c79d-4942-8b8a-1493cb9e9182-workload-socket\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:28:18.715107 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:18.715088 4376 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61a59ba0-c79d-4942-8b8a-1493cb9e9182-istio-data\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:28:18.716036 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:18.716007 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61a59ba0-c79d-4942-8b8a-1493cb9e9182-istio-envoy" pod "61a59ba0-c79d-4942-8b8a-1493cb9e9182" (UID: "61a59ba0-c79d-4942-8b8a-1493cb9e9182"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:28:18.716126 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:18.716102 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61a59ba0-c79d-4942-8b8a-1493cb9e9182-istio-token" pod "61a59ba0-c79d-4942-8b8a-1493cb9e9182" (UID: "61a59ba0-c79d-4942-8b8a-1493cb9e9182"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:28:18.716536 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:18.716516 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/61a59ba0-c79d-4942-8b8a-1493cb9e9182-istio-podinfo" pod "61a59ba0-c79d-4942-8b8a-1493cb9e9182" (UID: "61a59ba0-c79d-4942-8b8a-1493cb9e9182"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 04:28:18.716594 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:18.716535 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61a59ba0-c79d-4942-8b8a-1493cb9e9182-kube-api-access-48gpj" pod "61a59ba0-c79d-4942-8b8a-1493cb9e9182" (UID: "61a59ba0-c79d-4942-8b8a-1493cb9e9182"). InnerVolumeSpecName "kube-api-access-48gpj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:28:18.816221 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:18.816187 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48gpj\" (UniqueName: \"kubernetes.io/projected/61a59ba0-c79d-4942-8b8a-1493cb9e9182-kube-api-access-48gpj\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:28:18.816221 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:18.816220 4376 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61a59ba0-c79d-4942-8b8a-1493cb9e9182-istio-podinfo\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:28:18.816221 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:18.816231 4376 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61a59ba0-c79d-4942-8b8a-1493cb9e9182-istio-envoy\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:28:18.816490 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:18.816239 4376 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61a59ba0-c79d-4942-8b8a-1493cb9e9182-istio-token\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:28:19.060732 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:19.060648 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-edc450-5c6c7f49c4-q2z97"] Oct 31 04:28:19.061247 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:19.061232 4376 cpu_manager.go:414] "containerMap: removing container" podUID="61a59ba0-c79d-4942-8b8a-1493cb9e9182" containerName="istio-proxy" Oct 31 04:28:19.061314 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:19.061248 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61a59ba0-c79d-4942-8b8a-1493cb9e9182" containerName="istio-proxy" Oct 31 04:28:19.061392 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:19.061381 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="61a59ba0-c79d-4942-8b8a-1493cb9e9182" containerName="istio-proxy" Oct 31 04:28:19.061446 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:19.061396 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="61a59ba0-c79d-4942-8b8a-1493cb9e9182" containerName="istio-proxy" Oct 31 04:28:19.065199 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:19.065179 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-edc450-5c6c7f49c4-q2z97" Oct 31 04:28:19.080826 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:19.080792 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-edc450-serving-cert\"" Oct 31 04:28:19.084342 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:19.084307 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-edc450-5c6c7f49c4-q2z97"] Oct 31 04:28:19.221442 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:19.221401 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f3c1098-557f-4aae-90e3-b0e0869f5e30-metrics-certs\") pod \"maas-api-e2e-praxis-b-edc450-5c6c7f49c4-q2z97\" (UID: \"2f3c1098-557f-4aae-90e3-b0e0869f5e30\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-edc450-5c6c7f49c4-q2z97" Oct 31 04:28:19.221612 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:19.221519 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22fnc\" (UniqueName: \"kubernetes.io/projected/2f3c1098-557f-4aae-90e3-b0e0869f5e30-kube-api-access-22fnc\") pod \"maas-api-e2e-praxis-b-edc450-5c6c7f49c4-q2z97\" (UID: \"2f3c1098-557f-4aae-90e3-b0e0869f5e30\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-edc450-5c6c7f49c4-q2z97" Oct 31 04:28:19.221655 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:19.221613 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f3c1098-557f-4aae-90e3-b0e0869f5e30-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-edc450-5c6c7f49c4-q2z97\" (UID: \"2f3c1098-557f-4aae-90e3-b0e0869f5e30\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-edc450-5c6c7f49c4-q2z97" Oct 31 04:28:19.221769 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:19.221757 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f3c1098-557f-4aae-90e3-b0e0869f5e30-maas-api-tls\") pod \"maas-api-e2e-praxis-b-edc450-5c6c7f49c4-q2z97\" (UID: \"2f3c1098-557f-4aae-90e3-b0e0869f5e30\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-edc450-5c6c7f49c4-q2z97" Oct 31 04:28:19.328162 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:19.328080 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22fnc\" (UniqueName: \"kubernetes.io/projected/2f3c1098-557f-4aae-90e3-b0e0869f5e30-kube-api-access-22fnc\") pod \"maas-api-e2e-praxis-b-edc450-5c6c7f49c4-q2z97\" (UID: \"2f3c1098-557f-4aae-90e3-b0e0869f5e30\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-edc450-5c6c7f49c4-q2z97" Oct 31 04:28:19.328315 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:19.328165 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f3c1098-557f-4aae-90e3-b0e0869f5e30-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-edc450-5c6c7f49c4-q2z97\" (UID: \"2f3c1098-557f-4aae-90e3-b0e0869f5e30\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-edc450-5c6c7f49c4-q2z97" Oct 31 04:28:19.328424 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:19.328321 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f3c1098-557f-4aae-90e3-b0e0869f5e30-maas-api-tls\") pod \"maas-api-e2e-praxis-b-edc450-5c6c7f49c4-q2z97\" (UID: \"2f3c1098-557f-4aae-90e3-b0e0869f5e30\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-edc450-5c6c7f49c4-q2z97" Oct 31 04:28:19.328485 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:19.328423 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f3c1098-557f-4aae-90e3-b0e0869f5e30-metrics-certs\") pod \"maas-api-e2e-praxis-b-edc450-5c6c7f49c4-q2z97\" (UID: \"2f3c1098-557f-4aae-90e3-b0e0869f5e30\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-edc450-5c6c7f49c4-q2z97" Oct 31 04:28:19.342351 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:19.339803 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f3c1098-557f-4aae-90e3-b0e0869f5e30-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-edc450-5c6c7f49c4-q2z97\" (UID: \"2f3c1098-557f-4aae-90e3-b0e0869f5e30\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-edc450-5c6c7f49c4-q2z97" Oct 31 04:28:19.342351 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:19.341767 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f3c1098-557f-4aae-90e3-b0e0869f5e30-metrics-certs\") pod \"maas-api-e2e-praxis-b-edc450-5c6c7f49c4-q2z97\" (UID: \"2f3c1098-557f-4aae-90e3-b0e0869f5e30\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-edc450-5c6c7f49c4-q2z97" Oct 31 04:28:19.363368 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:19.361756 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22fnc\" (UniqueName: \"kubernetes.io/projected/2f3c1098-557f-4aae-90e3-b0e0869f5e30-kube-api-access-22fnc\") pod \"maas-api-e2e-praxis-b-edc450-5c6c7f49c4-q2z97\" (UID: \"2f3c1098-557f-4aae-90e3-b0e0869f5e30\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-edc450-5c6c7f49c4-q2z97" Oct 31 04:28:19.377349 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:19.369023 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f3c1098-557f-4aae-90e3-b0e0869f5e30-maas-api-tls\") pod \"maas-api-e2e-praxis-b-edc450-5c6c7f49c4-q2z97\" (UID: \"2f3c1098-557f-4aae-90e3-b0e0869f5e30\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-edc450-5c6c7f49c4-q2z97" Oct 31 04:28:19.377349 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:19.369189 4376 generic.go:356] "Generic (PLEG): container finished" podID="61a59ba0-c79d-4942-8b8a-1493cb9e9182" containerID="61198db9bad5b173ddd7905ca76f72f06a462385c41b993e199df5c09cd97cb0" exitCode=0 Oct 31 04:28:19.377349 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:19.370256 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-b8bb1e-openshift-default-557bbf897c-phbbm" Oct 31 04:28:19.399352 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:19.395882 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b8bb1e-openshift-default-557bbf897c-phbbm" event={"ID":"61a59ba0-c79d-4942-8b8a-1493cb9e9182","Type":"ContainerDied","Data":"61198db9bad5b173ddd7905ca76f72f06a462385c41b993e199df5c09cd97cb0"} Oct 31 04:28:19.399352 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:19.395996 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b8bb1e-openshift-default-557bbf897c-phbbm" event={"ID":"61a59ba0-c79d-4942-8b8a-1493cb9e9182","Type":"ContainerDied","Data":"0ab1a3671171bc76797fbb3ad195b86924ca53801677cd8e85ff90e5bdaf977d"} Oct 31 04:28:19.399352 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:19.396036 4376 scope.go:122] "RemoveContainer" containerID="61198db9bad5b173ddd7905ca76f72f06a462385c41b993e199df5c09cd97cb0" Oct 31 04:28:19.399352 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:19.396557 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-edc450-5c6c7f49c4-q2z97" Oct 31 04:28:19.467018 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:19.466898 4376 scope.go:122] "RemoveContainer" containerID="4fd82931ebe743135b6a825d8f86ecdb8db0e895ee103a632b0318de61547947" Oct 31 04:28:19.475165 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:19.475100 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-b8bb1e-openshift-default-557bbf897c-phbbm"] Oct 31 04:28:19.476126 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:19.476063 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-b8bb1e-openshift-default-557bbf897c-phbbm"] Oct 31 04:28:19.503678 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:19.503649 4376 scope.go:122] "RemoveContainer" containerID="61198db9bad5b173ddd7905ca76f72f06a462385c41b993e199df5c09cd97cb0" Oct 31 04:28:19.504072 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:28:19.504045 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61198db9bad5b173ddd7905ca76f72f06a462385c41b993e199df5c09cd97cb0\": container with ID starting with 61198db9bad5b173ddd7905ca76f72f06a462385c41b993e199df5c09cd97cb0 not found: ID does not exist" containerID="61198db9bad5b173ddd7905ca76f72f06a462385c41b993e199df5c09cd97cb0" Oct 31 04:28:19.504173 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:19.504082 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61198db9bad5b173ddd7905ca76f72f06a462385c41b993e199df5c09cd97cb0"} err="failed to get container status \"61198db9bad5b173ddd7905ca76f72f06a462385c41b993e199df5c09cd97cb0\": rpc error: code = NotFound desc = could not find container \"61198db9bad5b173ddd7905ca76f72f06a462385c41b993e199df5c09cd97cb0\": container with ID starting with 61198db9bad5b173ddd7905ca76f72f06a462385c41b993e199df5c09cd97cb0 not found: ID does not exist" Oct 31 04:28:19.504173 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:19.504110 4376 scope.go:122] "RemoveContainer" containerID="4fd82931ebe743135b6a825d8f86ecdb8db0e895ee103a632b0318de61547947" Oct 31 04:28:19.504459 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:28:19.504396 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4fd82931ebe743135b6a825d8f86ecdb8db0e895ee103a632b0318de61547947\": container with ID starting with 4fd82931ebe743135b6a825d8f86ecdb8db0e895ee103a632b0318de61547947 not found: ID does not exist" containerID="4fd82931ebe743135b6a825d8f86ecdb8db0e895ee103a632b0318de61547947" Oct 31 04:28:19.504459 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:19.504434 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fd82931ebe743135b6a825d8f86ecdb8db0e895ee103a632b0318de61547947"} err="failed to get container status \"4fd82931ebe743135b6a825d8f86ecdb8db0e895ee103a632b0318de61547947\": rpc error: code = NotFound desc = could not find container \"4fd82931ebe743135b6a825d8f86ecdb8db0e895ee103a632b0318de61547947\": container with ID starting with 4fd82931ebe743135b6a825d8f86ecdb8db0e895ee103a632b0318de61547947 not found: ID does not exist" Oct 31 04:28:19.792275 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:28:19.792239 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f3c1098_557f_4aae_90e3_b0e0869f5e30.slice/crio-b4abd19f022408560be143564cf66abf3fd5c8fc1e4f57fc09fcd7f177723f13 WatchSource:0}: Error finding container b4abd19f022408560be143564cf66abf3fd5c8fc1e4f57fc09fcd7f177723f13: Status 404 returned error can't find the container with id b4abd19f022408560be143564cf66abf3fd5c8fc1e4f57fc09fcd7f177723f13 Oct 31 04:28:19.793937 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:19.793909 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-edc450-5c6c7f49c4-q2z97"] Oct 31 04:28:20.377162 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:20.377130 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-edc450-5c6c7f49c4-q2z97" event={"ID":"2f3c1098-557f-4aae-90e3-b0e0869f5e30","Type":"ContainerStarted","Data":"d26c8f267da2a718fd61ece4cc85fd341f58fae7e29e868f5ef332ea6143422f"} Oct 31 04:28:20.377377 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:20.377181 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-edc450-5c6c7f49c4-q2z97" event={"ID":"2f3c1098-557f-4aae-90e3-b0e0869f5e30","Type":"ContainerStarted","Data":"b4abd19f022408560be143564cf66abf3fd5c8fc1e4f57fc09fcd7f177723f13"} Oct 31 04:28:20.377377 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:20.377205 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-edc450-5c6c7f49c4-q2z97" Oct 31 04:28:20.422871 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:20.422767 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-edc450-5c6c7f49c4-q2z97" podStartSLOduration=1.051776135 podStartE2EDuration="1.422752895s" totalImagesPullingTime="370.97676ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:28:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:28:20.422067708 +0000 UTC m=+1448.136402327" watchObservedRunningTime="2026-10-31 04:28:20.422752895 +0000 UTC m=+1448.137087512" Oct 31 04:28:20.457286 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:20.457238 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61a59ba0-c79d-4942-8b8a-1493cb9e9182" path="/var/lib/kubelet/pods/61a59ba0-c79d-4942-8b8a-1493cb9e9182/volumes" Oct 31 04:28:21.392428 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:21.392362 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-b3ca79-776b5db7b5-8f4k2"] Oct 31 04:28:21.393225 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:21.393206 4376 cpu_manager.go:414] "containerMap: removing container" podUID="61a59ba0-c79d-4942-8b8a-1493cb9e9182" containerName="istio-proxy" Oct 31 04:28:21.393291 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:21.393227 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61a59ba0-c79d-4942-8b8a-1493cb9e9182" containerName="istio-proxy" Oct 31 04:28:21.397732 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:21.397711 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-b3ca79-776b5db7b5-8f4k2" Oct 31 04:28:21.404171 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:21.404140 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45ff2ef8-aff5-4df7-8b70-6cea9e0b9332-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-b3ca79-776b5db7b5-8f4k2\" (UID: \"45ff2ef8-aff5-4df7-8b70-6cea9e0b9332\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-b3ca79-776b5db7b5-8f4k2" Oct 31 04:28:21.404427 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:21.404230 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ngms\" (UniqueName: \"kubernetes.io/projected/45ff2ef8-aff5-4df7-8b70-6cea9e0b9332-kube-api-access-4ngms\") pod \"payload-processing-e2e-praxis-a-b3ca79-776b5db7b5-8f4k2\" (UID: \"45ff2ef8-aff5-4df7-8b70-6cea9e0b9332\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-b3ca79-776b5db7b5-8f4k2" Oct 31 04:28:21.414409 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:21.414379 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-b3ca79-776b5db7b5-8f4k2"] Oct 31 04:28:21.422972 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:21.422931 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-b3ca79\"" Oct 31 04:28:21.423110 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:21.422985 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-b3ca79-dockercfg-kcbj2\"" Oct 31 04:28:21.455955 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:21.455919 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-b3ca79-6f8bf456dc-t4295"] Oct 31 04:28:21.460544 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:21.460518 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b3ca79-6f8bf456dc-t4295" Oct 31 04:28:21.479144 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:21.479116 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-b3ca79-6f8bf456dc-t4295"] Oct 31 04:28:21.506295 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:21.505689 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9spv\" (UniqueName: \"kubernetes.io/projected/25409dbd-265a-4852-9b66-a7689871e139-kube-api-access-z9spv\") pod \"payload-pre-processing-e2e-praxis-a-b3ca79-6f8bf456dc-t4295\" (UID: \"25409dbd-265a-4852-9b66-a7689871e139\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b3ca79-6f8bf456dc-t4295" Oct 31 04:28:21.506295 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:21.505948 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45ff2ef8-aff5-4df7-8b70-6cea9e0b9332-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-b3ca79-776b5db7b5-8f4k2\" (UID: \"45ff2ef8-aff5-4df7-8b70-6cea9e0b9332\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-b3ca79-776b5db7b5-8f4k2" Oct 31 04:28:21.506295 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:21.505983 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25409dbd-265a-4852-9b66-a7689871e139-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-b3ca79-6f8bf456dc-t4295\" (UID: \"25409dbd-265a-4852-9b66-a7689871e139\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b3ca79-6f8bf456dc-t4295" Oct 31 04:28:21.506295 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:21.506067 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ngms\" (UniqueName: \"kubernetes.io/projected/45ff2ef8-aff5-4df7-8b70-6cea9e0b9332-kube-api-access-4ngms\") pod \"payload-processing-e2e-praxis-a-b3ca79-776b5db7b5-8f4k2\" (UID: \"45ff2ef8-aff5-4df7-8b70-6cea9e0b9332\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-b3ca79-776b5db7b5-8f4k2" Oct 31 04:28:21.508194 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:21.508134 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45ff2ef8-aff5-4df7-8b70-6cea9e0b9332-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-b3ca79-776b5db7b5-8f4k2\" (UID: \"45ff2ef8-aff5-4df7-8b70-6cea9e0b9332\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-b3ca79-776b5db7b5-8f4k2" Oct 31 04:28:21.613363 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:21.609972 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ngms\" (UniqueName: \"kubernetes.io/projected/45ff2ef8-aff5-4df7-8b70-6cea9e0b9332-kube-api-access-4ngms\") pod \"payload-processing-e2e-praxis-a-b3ca79-776b5db7b5-8f4k2\" (UID: \"45ff2ef8-aff5-4df7-8b70-6cea9e0b9332\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-b3ca79-776b5db7b5-8f4k2" Oct 31 04:28:21.613363 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:21.611768 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9spv\" (UniqueName: \"kubernetes.io/projected/25409dbd-265a-4852-9b66-a7689871e139-kube-api-access-z9spv\") pod \"payload-pre-processing-e2e-praxis-a-b3ca79-6f8bf456dc-t4295\" (UID: \"25409dbd-265a-4852-9b66-a7689871e139\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b3ca79-6f8bf456dc-t4295" Oct 31 04:28:21.613363 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:21.612092 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25409dbd-265a-4852-9b66-a7689871e139-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-b3ca79-6f8bf456dc-t4295\" (UID: \"25409dbd-265a-4852-9b66-a7689871e139\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b3ca79-6f8bf456dc-t4295" Oct 31 04:28:21.613363 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:21.613277 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25409dbd-265a-4852-9b66-a7689871e139-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-b3ca79-6f8bf456dc-t4295\" (UID: \"25409dbd-265a-4852-9b66-a7689871e139\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b3ca79-6f8bf456dc-t4295" Oct 31 04:28:21.715378 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:21.714428 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-b3ca79-776b5db7b5-8f4k2" Oct 31 04:28:21.764825 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:21.759056 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9spv\" (UniqueName: \"kubernetes.io/projected/25409dbd-265a-4852-9b66-a7689871e139-kube-api-access-z9spv\") pod \"payload-pre-processing-e2e-praxis-a-b3ca79-6f8bf456dc-t4295\" (UID: \"25409dbd-265a-4852-9b66-a7689871e139\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b3ca79-6f8bf456dc-t4295" Oct 31 04:28:21.776565 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:21.776514 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b3ca79-6f8bf456dc-t4295" Oct 31 04:28:22.144662 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:28:22.144631 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45ff2ef8_aff5_4df7_8b70_6cea9e0b9332.slice/crio-91456758cab9d968eb5a38ef3d9810e35ae7484c6b02e5ac2644f08c1ad2218e WatchSource:0}: Error finding container 91456758cab9d968eb5a38ef3d9810e35ae7484c6b02e5ac2644f08c1ad2218e: Status 404 returned error can't find the container with id 91456758cab9d968eb5a38ef3d9810e35ae7484c6b02e5ac2644f08c1ad2218e Oct 31 04:28:22.146103 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:22.146069 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-b3ca79-776b5db7b5-8f4k2"] Oct 31 04:28:22.170972 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:22.170945 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-b3ca79-6f8bf456dc-t4295"] Oct 31 04:28:22.178565 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:28:22.178536 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25409dbd_265a_4852_9b66_a7689871e139.slice/crio-49edf5e78905662852cb533faafb2a3fd5e05136b41005e4729e82c352eb0f08 WatchSource:0}: Error finding container 49edf5e78905662852cb533faafb2a3fd5e05136b41005e4729e82c352eb0f08: Status 404 returned error can't find the container with id 49edf5e78905662852cb533faafb2a3fd5e05136b41005e4729e82c352eb0f08 Oct 31 04:28:22.392915 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:22.392820 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-b3ca79-776b5db7b5-8f4k2" event={"ID":"45ff2ef8-aff5-4df7-8b70-6cea9e0b9332","Type":"ContainerStarted","Data":"c7197a13aefa66039a025bd6c77e313b8880e05a79cf716885bb47769a373ad9"} Oct 31 04:28:22.392915 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:22.392874 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-b3ca79-776b5db7b5-8f4k2" event={"ID":"45ff2ef8-aff5-4df7-8b70-6cea9e0b9332","Type":"ContainerStarted","Data":"91456758cab9d968eb5a38ef3d9810e35ae7484c6b02e5ac2644f08c1ad2218e"} Oct 31 04:28:22.393411 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:22.393017 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-b3ca79-776b5db7b5-8f4k2" Oct 31 04:28:22.394654 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:22.394625 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b3ca79-6f8bf456dc-t4295" event={"ID":"25409dbd-265a-4852-9b66-a7689871e139","Type":"ContainerStarted","Data":"d23098ddb93b43c5232d7156e15d9632b9f579252f1ea63a47072952b0bcf11d"} Oct 31 04:28:22.394794 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:22.394667 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b3ca79-6f8bf456dc-t4295" event={"ID":"25409dbd-265a-4852-9b66-a7689871e139","Type":"ContainerStarted","Data":"49edf5e78905662852cb533faafb2a3fd5e05136b41005e4729e82c352eb0f08"} Oct 31 04:28:22.394794 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:22.394710 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b3ca79-6f8bf456dc-t4295" Oct 31 04:28:22.436861 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:22.436813 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-b3ca79-776b5db7b5-8f4k2" podStartSLOduration=1.436792994 podStartE2EDuration="1.436792994s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:28:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:28:22.433147138 +0000 UTC m=+1450.147481755" watchObservedRunningTime="2026-10-31 04:28:22.436792994 +0000 UTC m=+1450.151127611" Oct 31 04:28:22.476702 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:22.476646 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b3ca79-6f8bf456dc-t4295" podStartSLOduration=1.476631336 podStartE2EDuration="1.476631336s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:28:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:28:22.475049113 +0000 UTC m=+1450.189383757" watchObservedRunningTime="2026-10-31 04:28:22.476631336 +0000 UTC m=+1450.190965952" Oct 31 04:28:23.217607 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:23.217574 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-81c810-5749ff85ff-5fr52" Oct 31 04:28:26.390064 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:26.390031 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-edc450-5c6c7f49c4-q2z97" Oct 31 04:28:33.410844 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:33.410813 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-b3ca79-776b5db7b5-8f4k2" Oct 31 04:28:33.411302 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:33.411182 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b3ca79-6f8bf456dc-t4295" Oct 31 04:28:41.001577 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:28:41.001540 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 04:28:44.876446 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:44.876403 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 04:28:46.711398 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:46.708438 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-b3ca79-78445fb8df-792kj"] Oct 31 04:28:46.714418 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:46.714386 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b3ca79-78445fb8df-792kj" Oct 31 04:28:46.720800 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:46.720766 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-b3ca79-78445fb8df-792kj"] Oct 31 04:28:46.730118 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:46.730090 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-b3ca79-serving-cert\"" Oct 31 04:28:46.826810 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:46.826769 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lr7mn\" (UniqueName: \"kubernetes.io/projected/623914fe-4610-4eeb-ba51-15e3e55faa31-kube-api-access-lr7mn\") pod \"maas-api-e2e-praxis-a-b3ca79-78445fb8df-792kj\" (UID: \"623914fe-4610-4eeb-ba51-15e3e55faa31\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b3ca79-78445fb8df-792kj" Oct 31 04:28:46.827008 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:46.826988 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/623914fe-4610-4eeb-ba51-15e3e55faa31-maas-api-tls\") pod \"maas-api-e2e-praxis-a-b3ca79-78445fb8df-792kj\" (UID: \"623914fe-4610-4eeb-ba51-15e3e55faa31\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b3ca79-78445fb8df-792kj" Oct 31 04:28:46.827106 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:46.827033 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/623914fe-4610-4eeb-ba51-15e3e55faa31-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-b3ca79-78445fb8df-792kj\" (UID: \"623914fe-4610-4eeb-ba51-15e3e55faa31\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b3ca79-78445fb8df-792kj" Oct 31 04:28:46.827145 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:46.827126 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/623914fe-4610-4eeb-ba51-15e3e55faa31-metrics-certs\") pod \"maas-api-e2e-praxis-a-b3ca79-78445fb8df-792kj\" (UID: \"623914fe-4610-4eeb-ba51-15e3e55faa31\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b3ca79-78445fb8df-792kj" Oct 31 04:28:46.930302 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:46.930261 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/623914fe-4610-4eeb-ba51-15e3e55faa31-metrics-certs\") pod \"maas-api-e2e-praxis-a-b3ca79-78445fb8df-792kj\" (UID: \"623914fe-4610-4eeb-ba51-15e3e55faa31\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b3ca79-78445fb8df-792kj" Oct 31 04:28:46.930643 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:46.930619 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lr7mn\" (UniqueName: \"kubernetes.io/projected/623914fe-4610-4eeb-ba51-15e3e55faa31-kube-api-access-lr7mn\") pod \"maas-api-e2e-praxis-a-b3ca79-78445fb8df-792kj\" (UID: \"623914fe-4610-4eeb-ba51-15e3e55faa31\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b3ca79-78445fb8df-792kj" Oct 31 04:28:46.946676 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:46.946640 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/623914fe-4610-4eeb-ba51-15e3e55faa31-maas-api-tls\") pod \"maas-api-e2e-praxis-a-b3ca79-78445fb8df-792kj\" (UID: \"623914fe-4610-4eeb-ba51-15e3e55faa31\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b3ca79-78445fb8df-792kj" Oct 31 04:28:46.946939 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:46.946917 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/623914fe-4610-4eeb-ba51-15e3e55faa31-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-b3ca79-78445fb8df-792kj\" (UID: \"623914fe-4610-4eeb-ba51-15e3e55faa31\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b3ca79-78445fb8df-792kj" Oct 31 04:28:46.970064 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:46.948082 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/623914fe-4610-4eeb-ba51-15e3e55faa31-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-b3ca79-78445fb8df-792kj\" (UID: \"623914fe-4610-4eeb-ba51-15e3e55faa31\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b3ca79-78445fb8df-792kj" Oct 31 04:28:46.985703 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:46.973229 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/623914fe-4610-4eeb-ba51-15e3e55faa31-maas-api-tls\") pod \"maas-api-e2e-praxis-a-b3ca79-78445fb8df-792kj\" (UID: \"623914fe-4610-4eeb-ba51-15e3e55faa31\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b3ca79-78445fb8df-792kj" Oct 31 04:28:46.989645 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:46.989584 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lr7mn\" (UniqueName: \"kubernetes.io/projected/623914fe-4610-4eeb-ba51-15e3e55faa31-kube-api-access-lr7mn\") pod \"maas-api-e2e-praxis-a-b3ca79-78445fb8df-792kj\" (UID: \"623914fe-4610-4eeb-ba51-15e3e55faa31\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b3ca79-78445fb8df-792kj" Oct 31 04:28:47.023410 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:47.017608 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/623914fe-4610-4eeb-ba51-15e3e55faa31-metrics-certs\") pod \"maas-api-e2e-praxis-a-b3ca79-78445fb8df-792kj\" (UID: \"623914fe-4610-4eeb-ba51-15e3e55faa31\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b3ca79-78445fb8df-792kj" Oct 31 04:28:47.053351 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:47.048601 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b3ca79-78445fb8df-792kj" Oct 31 04:28:47.517209 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:28:47.517178 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod623914fe_4610_4eeb_ba51_15e3e55faa31.slice/crio-0a2f3d609369c2e4a7c09249482d14139e5d49603095b02e0060a2c887ffa42b WatchSource:0}: Error finding container 0a2f3d609369c2e4a7c09249482d14139e5d49603095b02e0060a2c887ffa42b: Status 404 returned error can't find the container with id 0a2f3d609369c2e4a7c09249482d14139e5d49603095b02e0060a2c887ffa42b Oct 31 04:28:47.519733 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:47.519708 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-b3ca79-78445fb8df-792kj"] Oct 31 04:28:47.692151 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:47.692072 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b3ca79-78445fb8df-792kj" event={"ID":"623914fe-4610-4eeb-ba51-15e3e55faa31","Type":"ContainerStarted","Data":"0a2f3d609369c2e4a7c09249482d14139e5d49603095b02e0060a2c887ffa42b"} Oct 31 04:28:48.701789 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:48.701759 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b3ca79-78445fb8df-792kj" event={"ID":"623914fe-4610-4eeb-ba51-15e3e55faa31","Type":"ContainerStarted","Data":"a297c0bb9eb4f3466d2e583a1e09e4f2bdb0d57e8143d06a9f7374bc33c048c1"} Oct 31 04:28:48.702367 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:48.702349 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b3ca79-78445fb8df-792kj" Oct 31 04:28:48.777352 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:48.776664 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b3ca79-78445fb8df-792kj" podStartSLOduration=2.373528746 podStartE2EDuration="2.776642695s" totalImagesPullingTime="403.113949ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:28:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:28:48.757634894 +0000 UTC m=+1476.471969538" watchObservedRunningTime="2026-10-31 04:28:48.776642695 +0000 UTC m=+1476.490977315" Oct 31 04:28:54.715031 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:54.714998 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b3ca79-78445fb8df-792kj" Oct 31 04:28:54.905559 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:54.905470 4376 patch_prober.go:29] interesting pod/e2e-praxis-b-edc450-openshift-default-84c757994f-q4vwz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 04:28:54.905559 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:54.905549 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-edc450-openshift-default-84c757994f-q4vwz" podUID="c0cc41ad-7f5b-4a5c-a989-bfafee0857e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 04:28:56.784614 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:56.784573 4376 generic.go:356] "Generic (PLEG): container finished" podID="c0cc41ad-7f5b-4a5c-a989-bfafee0857e8" containerID="28555215b5812fed9bd2c710a9a4ddd1010c7957faa478947a5a28fb3f010ad8" exitCode=0 Oct 31 04:28:56.785107 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:56.784644 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-edc450-openshift-default-84c757994f-q4vwz" event={"ID":"c0cc41ad-7f5b-4a5c-a989-bfafee0857e8","Type":"ContainerDied","Data":"28555215b5812fed9bd2c710a9a4ddd1010c7957faa478947a5a28fb3f010ad8"} Oct 31 04:28:56.785197 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:56.785180 4376 scope.go:122] "RemoveContainer" containerID="28555215b5812fed9bd2c710a9a4ddd1010c7957faa478947a5a28fb3f010ad8" Oct 31 04:28:56.786684 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:56.786664 4376 generic.go:356] "Generic (PLEG): container finished" podID="3c34e357-743e-442a-957d-788f1618aaf3" containerID="a0c0db5b3fe8488903d4b2f1b246295390bce9fbf6fe570b7d7cd55ebe738ca1" exitCode=0 Oct 31 04:28:56.786773 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:56.786708 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-81c810-openshift-default-5fbd5b64c7-kbvbk" event={"ID":"3c34e357-743e-442a-957d-788f1618aaf3","Type":"ContainerDied","Data":"a0c0db5b3fe8488903d4b2f1b246295390bce9fbf6fe570b7d7cd55ebe738ca1"} Oct 31 04:28:56.786980 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:56.786954 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:28:56.787035 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:56.787022 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:28:56.787071 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:56.787038 4376 scope.go:122] "RemoveContainer" containerID="a0c0db5b3fe8488903d4b2f1b246295390bce9fbf6fe570b7d7cd55ebe738ca1" Oct 31 04:28:56.787071 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:56.787055 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:28:56.788499 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:56.788460 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:28:56.788572 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:56.788540 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:28:56.788608 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:56.788572 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:28:57.797919 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:57.797880 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-edc450-openshift-default-84c757994f-q4vwz" event={"ID":"c0cc41ad-7f5b-4a5c-a989-bfafee0857e8","Type":"ContainerStarted","Data":"20210477399c246dd0b73fbb8ffe62f810561347e339808647f65ad299caa95d"} Oct 31 04:28:57.800676 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:57.800651 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-81c810-openshift-default-5fbd5b64c7-kbvbk" event={"ID":"3c34e357-743e-442a-957d-788f1618aaf3","Type":"ContainerStarted","Data":"a7bb272b3c6d152f9c092163497389233315768c200b9bf8eb4f1e4699cdbd59"} Oct 31 04:28:57.940761 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:57.940729 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-edc450-openshift-default-84c757994f-q4vwz" Oct 31 04:28:57.941567 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:57.941542 4376 patch_prober.go:29] interesting pod/e2e-praxis-b-edc450-openshift-default-84c757994f-q4vwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body= Oct 31 04:28:57.941756 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:57.941731 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-edc450-openshift-default-84c757994f-q4vwz" podUID="c0cc41ad-7f5b-4a5c-a989-bfafee0857e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" Oct 31 04:28:58.099452 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:58.099360 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-81c810-openshift-default-5fbd5b64c7-kbvbk" Oct 31 04:28:58.099615 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:58.099546 4376 patch_prober.go:29] interesting pod/e2e-praxis-b-81c810-openshift-default-5fbd5b64c7-kbvbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Oct 31 04:28:58.099615 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:58.099589 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-81c810-openshift-default-5fbd5b64c7-kbvbk" podUID="3c34e357-743e-442a-957d-788f1618aaf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Oct 31 04:28:58.947749 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:58.947710 4376 patch_prober.go:29] interesting pod/e2e-praxis-b-edc450-openshift-default-84c757994f-q4vwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body= Oct 31 04:28:58.955190 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:58.955151 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-edc450-openshift-default-84c757994f-q4vwz" podUID="c0cc41ad-7f5b-4a5c-a989-bfafee0857e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" Oct 31 04:28:59.099845 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:59.099789 4376 patch_prober.go:29] interesting pod/e2e-praxis-b-81c810-openshift-default-5fbd5b64c7-kbvbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Oct 31 04:28:59.100018 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:59.099864 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-81c810-openshift-default-5fbd5b64c7-kbvbk" podUID="3c34e357-743e-442a-957d-788f1618aaf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Oct 31 04:28:59.948491 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:59.948457 4376 patch_prober.go:29] interesting pod/e2e-praxis-b-edc450-openshift-default-84c757994f-q4vwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body= Oct 31 04:28:59.964575 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:28:59.964421 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-edc450-openshift-default-84c757994f-q4vwz" podUID="c0cc41ad-7f5b-4a5c-a989-bfafee0857e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" Oct 31 04:29:00.106352 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:00.106060 4376 patch_prober.go:29] interesting pod/e2e-praxis-b-81c810-openshift-default-5fbd5b64c7-kbvbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Oct 31 04:29:00.106352 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:00.106122 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-81c810-openshift-default-5fbd5b64c7-kbvbk" podUID="3c34e357-743e-442a-957d-788f1618aaf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Oct 31 04:29:00.941692 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:00.941654 4376 patch_prober.go:29] interesting pod/e2e-praxis-b-edc450-openshift-default-84c757994f-q4vwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body= Oct 31 04:29:00.941958 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:00.941930 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-edc450-openshift-default-84c757994f-q4vwz" podUID="c0cc41ad-7f5b-4a5c-a989-bfafee0857e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" Oct 31 04:29:01.100284 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:01.100249 4376 patch_prober.go:29] interesting pod/e2e-praxis-b-81c810-openshift-default-5fbd5b64c7-kbvbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Oct 31 04:29:01.100778 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:01.100322 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-81c810-openshift-default-5fbd5b64c7-kbvbk" podUID="3c34e357-743e-442a-957d-788f1618aaf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Oct 31 04:29:01.632816 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:01.632780 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-9ee727-5fbc8757cf-2gvbr"] Oct 31 04:29:01.633197 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:01.633173 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-9ee727-5fbc8757cf-2gvbr" podUID="b6a7f8f5-13b4-46cc-93c7-3bdd9e107f7f" containerName="maas-api" containerID="cri-o://5a8c1d6b418f57a17cbfc0a96ff07d4e00e1eac67a5e4900c1a90201a4e59ae9" gracePeriod=30 Oct 31 04:29:01.831873 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:01.831835 4376 generic.go:356] "Generic (PLEG): container finished" podID="b6a7f8f5-13b4-46cc-93c7-3bdd9e107f7f" containerID="5a8c1d6b418f57a17cbfc0a96ff07d4e00e1eac67a5e4900c1a90201a4e59ae9" exitCode=0 Oct 31 04:29:01.832035 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:01.831944 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-9ee727-5fbc8757cf-2gvbr" event={"ID":"b6a7f8f5-13b4-46cc-93c7-3bdd9e107f7f","Type":"ContainerDied","Data":"5a8c1d6b418f57a17cbfc0a96ff07d4e00e1eac67a5e4900c1a90201a4e59ae9"} Oct 31 04:29:01.941081 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:01.940992 4376 patch_prober.go:29] interesting pod/e2e-praxis-b-edc450-openshift-default-84c757994f-q4vwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body= Oct 31 04:29:01.941362 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:01.941307 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-edc450-openshift-default-84c757994f-q4vwz" podUID="c0cc41ad-7f5b-4a5c-a989-bfafee0857e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" Oct 31 04:29:02.103834 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:02.103796 4376 patch_prober.go:29] interesting pod/e2e-praxis-b-81c810-openshift-default-5fbd5b64c7-kbvbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Oct 31 04:29:02.120360 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:02.116510 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-81c810-openshift-default-5fbd5b64c7-kbvbk" podUID="3c34e357-743e-442a-957d-788f1618aaf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Oct 31 04:29:02.504488 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:02.504460 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-9ee727-5fbc8757cf-2gvbr" Oct 31 04:29:02.594834 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:02.594796 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b6a7f8f5-13b4-46cc-93c7-3bdd9e107f7f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b6a7f8f5-13b4-46cc-93c7-3bdd9e107f7f-metrics-certs\") pod \"b6a7f8f5-13b4-46cc-93c7-3bdd9e107f7f\" (UID: \"b6a7f8f5-13b4-46cc-93c7-3bdd9e107f7f\") " Oct 31 04:29:02.595626 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:02.595601 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b6a7f8f5-13b4-46cc-93c7-3bdd9e107f7f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6a7f8f5-13b4-46cc-93c7-3bdd9e107f7f-maas-api-tls\") pod \"b6a7f8f5-13b4-46cc-93c7-3bdd9e107f7f\" (UID: \"b6a7f8f5-13b4-46cc-93c7-3bdd9e107f7f\") " Oct 31 04:29:02.596486 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:02.596457 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6a7f8f5-13b4-46cc-93c7-3bdd9e107f7f-kube-api-access-rmzsd\" (UniqueName: \"kubernetes.io/projected/b6a7f8f5-13b4-46cc-93c7-3bdd9e107f7f-kube-api-access-rmzsd\") pod \"b6a7f8f5-13b4-46cc-93c7-3bdd9e107f7f\" (UID: \"b6a7f8f5-13b4-46cc-93c7-3bdd9e107f7f\") " Oct 31 04:29:02.596596 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:02.596577 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6a7f8f5-13b4-46cc-93c7-3bdd9e107f7f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6a7f8f5-13b4-46cc-93c7-3bdd9e107f7f-openshift-service-ca\") pod \"b6a7f8f5-13b4-46cc-93c7-3bdd9e107f7f\" (UID: \"b6a7f8f5-13b4-46cc-93c7-3bdd9e107f7f\") " Oct 31 04:29:02.597284 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:02.597240 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6a7f8f5-13b4-46cc-93c7-3bdd9e107f7f-openshift-service-ca" pod "b6a7f8f5-13b4-46cc-93c7-3bdd9e107f7f" (UID: "b6a7f8f5-13b4-46cc-93c7-3bdd9e107f7f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:29:02.598442 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:02.598395 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6a7f8f5-13b4-46cc-93c7-3bdd9e107f7f-metrics-certs" pod "b6a7f8f5-13b4-46cc-93c7-3bdd9e107f7f" (UID: "b6a7f8f5-13b4-46cc-93c7-3bdd9e107f7f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:29:02.599460 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:02.599436 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6a7f8f5-13b4-46cc-93c7-3bdd9e107f7f-maas-api-tls" pod "b6a7f8f5-13b4-46cc-93c7-3bdd9e107f7f" (UID: "b6a7f8f5-13b4-46cc-93c7-3bdd9e107f7f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:29:02.599671 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:02.599648 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6a7f8f5-13b4-46cc-93c7-3bdd9e107f7f-kube-api-access-rmzsd" pod "b6a7f8f5-13b4-46cc-93c7-3bdd9e107f7f" (UID: "b6a7f8f5-13b4-46cc-93c7-3bdd9e107f7f"). InnerVolumeSpecName "kube-api-access-rmzsd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:29:02.702375 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:02.700838 4376 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b6a7f8f5-13b4-46cc-93c7-3bdd9e107f7f-metrics-certs\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:29:02.702375 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:02.700874 4376 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6a7f8f5-13b4-46cc-93c7-3bdd9e107f7f-maas-api-tls\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:29:02.702375 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:02.700889 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmzsd\" (UniqueName: \"kubernetes.io/projected/b6a7f8f5-13b4-46cc-93c7-3bdd9e107f7f-kube-api-access-rmzsd\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:29:02.702375 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:02.700905 4376 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6a7f8f5-13b4-46cc-93c7-3bdd9e107f7f-openshift-service-ca\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:29:02.870040 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:02.870001 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-9ee727-5fbc8757cf-2gvbr" event={"ID":"b6a7f8f5-13b4-46cc-93c7-3bdd9e107f7f","Type":"ContainerDied","Data":"2ce63e68032fc918851bf890550edde6f5314725a60c2941eede495cb353d75e"} Oct 31 04:29:02.870290 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:02.870272 4376 scope.go:122] "RemoveContainer" containerID="5a8c1d6b418f57a17cbfc0a96ff07d4e00e1eac67a5e4900c1a90201a4e59ae9" Oct 31 04:29:02.870867 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:02.870846 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-9ee727-5fbc8757cf-2gvbr" Oct 31 04:29:02.936766 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:02.936689 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-9ee727-5fbc8757cf-2gvbr"] Oct 31 04:29:02.969351 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:02.955434 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-9ee727-5fbc8757cf-2gvbr"] Oct 31 04:29:02.969351 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:02.969051 4376 patch_prober.go:29] interesting pod/e2e-praxis-b-edc450-openshift-default-84c757994f-q4vwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body= Oct 31 04:29:02.969351 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:02.969099 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-edc450-openshift-default-84c757994f-q4vwz" podUID="c0cc41ad-7f5b-4a5c-a989-bfafee0857e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" Oct 31 04:29:03.100443 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:03.100410 4376 patch_prober.go:29] interesting pod/e2e-praxis-b-81c810-openshift-default-5fbd5b64c7-kbvbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Oct 31 04:29:03.100601 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:03.100490 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-81c810-openshift-default-5fbd5b64c7-kbvbk" podUID="3c34e357-743e-442a-957d-788f1618aaf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Oct 31 04:29:03.971768 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:03.971732 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-edc450-openshift-default-84c757994f-q4vwz" Oct 31 04:29:03.977005 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:03.973207 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-edc450-openshift-default-84c757994f-q4vwz" Oct 31 04:29:03.977005 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:03.973920 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-edc450-openshift-default-84c757994f-q4vwz" Oct 31 04:29:04.104210 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:04.104178 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-81c810-openshift-default-5fbd5b64c7-kbvbk" Oct 31 04:29:04.104496 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:04.104475 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-81c810-openshift-default-5fbd5b64c7-kbvbk" Oct 31 04:29:04.105317 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:04.105297 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-81c810-openshift-default-5fbd5b64c7-kbvbk" Oct 31 04:29:04.445856 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:04.445822 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6a7f8f5-13b4-46cc-93c7-3bdd9e107f7f" path="/var/lib/kubelet/pods/b6a7f8f5-13b4-46cc-93c7-3bdd9e107f7f/volumes" Oct 31 04:29:04.896412 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:04.896377 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-c6859a-59d59469c9-4r9lt"] Oct 31 04:29:09.897551 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:09.897510 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c6859a-59d59469c9-4r9lt" podUID="aa104be0-8c9b-4345-bcaf-a0cc7d0bebb3" containerName="payload-pre-processing" containerID="cri-o://74b8f7d0151c90595af4fe1b255d0d750302623fbc5c8619e106883daeb50c19" gracePeriod=25 Oct 31 04:29:10.204422 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:10.204394 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c6859a-59d59469c9-4r9lt" Oct 31 04:29:10.377844 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:10.377814 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa104be0-8c9b-4345-bcaf-a0cc7d0bebb3-kube-api-access-lqshz\" (UniqueName: \"kubernetes.io/projected/aa104be0-8c9b-4345-bcaf-a0cc7d0bebb3-kube-api-access-lqshz\") pod \"aa104be0-8c9b-4345-bcaf-a0cc7d0bebb3\" (UID: \"aa104be0-8c9b-4345-bcaf-a0cc7d0bebb3\") " Oct 31 04:29:10.378036 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:10.377897 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa104be0-8c9b-4345-bcaf-a0cc7d0bebb3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa104be0-8c9b-4345-bcaf-a0cc7d0bebb3-plugins-config-volume\") pod \"aa104be0-8c9b-4345-bcaf-a0cc7d0bebb3\" (UID: \"aa104be0-8c9b-4345-bcaf-a0cc7d0bebb3\") " Oct 31 04:29:10.378349 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:10.378310 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa104be0-8c9b-4345-bcaf-a0cc7d0bebb3-plugins-config-volume" pod "aa104be0-8c9b-4345-bcaf-a0cc7d0bebb3" (UID: "aa104be0-8c9b-4345-bcaf-a0cc7d0bebb3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:29:10.378483 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:10.378470 4376 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa104be0-8c9b-4345-bcaf-a0cc7d0bebb3-plugins-config-volume\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:29:10.379825 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:10.379800 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa104be0-8c9b-4345-bcaf-a0cc7d0bebb3-kube-api-access-lqshz" pod "aa104be0-8c9b-4345-bcaf-a0cc7d0bebb3" (UID: "aa104be0-8c9b-4345-bcaf-a0cc7d0bebb3"). InnerVolumeSpecName "kube-api-access-lqshz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:29:10.479457 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:10.479424 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqshz\" (UniqueName: \"kubernetes.io/projected/aa104be0-8c9b-4345-bcaf-a0cc7d0bebb3-kube-api-access-lqshz\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:29:11.001676 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:11.001635 4376 generic.go:356] "Generic (PLEG): container finished" podID="aa104be0-8c9b-4345-bcaf-a0cc7d0bebb3" containerID="74b8f7d0151c90595af4fe1b255d0d750302623fbc5c8619e106883daeb50c19" exitCode=0 Oct 31 04:29:11.004005 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:11.002373 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c6859a-59d59469c9-4r9lt" Oct 31 04:29:11.004391 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:11.002409 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c6859a-59d59469c9-4r9lt" event={"ID":"aa104be0-8c9b-4345-bcaf-a0cc7d0bebb3","Type":"ContainerDied","Data":"74b8f7d0151c90595af4fe1b255d0d750302623fbc5c8619e106883daeb50c19"} Oct 31 04:29:11.004656 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:11.004630 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c6859a-59d59469c9-4r9lt" event={"ID":"aa104be0-8c9b-4345-bcaf-a0cc7d0bebb3","Type":"ContainerDied","Data":"5d43580ef3d00390123c5ccb963740f494ad271be32e14804715e93276922279"} Oct 31 04:29:11.005157 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:11.005086 4376 scope.go:122] "RemoveContainer" containerID="74b8f7d0151c90595af4fe1b255d0d750302623fbc5c8619e106883daeb50c19" Oct 31 04:29:11.072913 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:11.072871 4376 scope.go:122] "RemoveContainer" containerID="74b8f7d0151c90595af4fe1b255d0d750302623fbc5c8619e106883daeb50c19" Oct 31 04:29:11.073622 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:29:11.073590 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74b8f7d0151c90595af4fe1b255d0d750302623fbc5c8619e106883daeb50c19\": container with ID starting with 74b8f7d0151c90595af4fe1b255d0d750302623fbc5c8619e106883daeb50c19 not found: ID does not exist" containerID="74b8f7d0151c90595af4fe1b255d0d750302623fbc5c8619e106883daeb50c19" Oct 31 04:29:11.073771 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:11.073633 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74b8f7d0151c90595af4fe1b255d0d750302623fbc5c8619e106883daeb50c19"} err="failed to get container status \"74b8f7d0151c90595af4fe1b255d0d750302623fbc5c8619e106883daeb50c19\": rpc error: code = NotFound desc = could not find container \"74b8f7d0151c90595af4fe1b255d0d750302623fbc5c8619e106883daeb50c19\": container with ID starting with 74b8f7d0151c90595af4fe1b255d0d750302623fbc5c8619e106883daeb50c19 not found: ID does not exist" Oct 31 04:29:11.091581 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:11.091549 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-c6859a-59d59469c9-4r9lt"] Oct 31 04:29:11.095890 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:29:11.095865 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 04:29:11.102010 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:11.101987 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-c6859a-59d59469c9-4r9lt"] Oct 31 04:29:12.186352 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:12.181117 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 04:29:12.325975 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:12.325940 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-praxis-a-d1d441/praxis-route-d1d441-kserve-698cd98df-qv4m7"] Oct 31 04:29:12.327193 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:12.326883 4376 cpu_manager.go:414] "containerMap: removing container" podUID="b6a7f8f5-13b4-46cc-93c7-3bdd9e107f7f" containerName="maas-api" Oct 31 04:29:12.327193 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:12.326908 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6a7f8f5-13b4-46cc-93c7-3bdd9e107f7f" containerName="maas-api" Oct 31 04:29:12.327193 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:12.326974 4376 cpu_manager.go:414] "containerMap: removing container" podUID="aa104be0-8c9b-4345-bcaf-a0cc7d0bebb3" containerName="payload-pre-processing" Oct 31 04:29:12.327193 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:12.326982 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa104be0-8c9b-4345-bcaf-a0cc7d0bebb3" containerName="payload-pre-processing" Oct 31 04:29:12.327193 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:12.327141 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6a7f8f5-13b4-46cc-93c7-3bdd9e107f7f" containerName="maas-api" Oct 31 04:29:12.327193 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:12.327164 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa104be0-8c9b-4345-bcaf-a0cc7d0bebb3" containerName="payload-pre-processing" Oct 31 04:29:12.331472 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:12.331450 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-d1d441/praxis-route-d1d441-kserve-698cd98df-qv4m7" Oct 31 04:29:12.346869 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:12.346844 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-praxis-a-d1d441\"/\"kube-root-ca.crt\"" Oct 31 04:29:12.347806 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:12.347671 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-praxis-a-d1d441/praxis-route-d1d441-kserve-698cd98df-qv4m7"] Oct 31 04:29:12.347806 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:12.347733 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-praxis-a-d1d441\"/\"default-dockercfg-4pt7c\"" Oct 31 04:29:12.352604 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:12.352586 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-praxis-a-d1d441\"/\"praxis-route-d1d441-kserve-self-signed-certs\"" Oct 31 04:29:12.416297 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:12.416258 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d20ddaa5-bf86-4dfa-9d5a-91fd4894ed34-model-cache\") pod \"praxis-route-d1d441-kserve-698cd98df-qv4m7\" (UID: \"d20ddaa5-bf86-4dfa-9d5a-91fd4894ed34\") " pod="ai-tenant-e2e-praxis-a-d1d441/praxis-route-d1d441-kserve-698cd98df-qv4m7" Oct 31 04:29:12.416508 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:12.416411 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mwq8\" (UniqueName: \"kubernetes.io/projected/d20ddaa5-bf86-4dfa-9d5a-91fd4894ed34-kube-api-access-4mwq8\") pod \"praxis-route-d1d441-kserve-698cd98df-qv4m7\" (UID: \"d20ddaa5-bf86-4dfa-9d5a-91fd4894ed34\") " pod="ai-tenant-e2e-praxis-a-d1d441/praxis-route-d1d441-kserve-698cd98df-qv4m7" Oct 31 04:29:12.416508 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:12.416462 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d20ddaa5-bf86-4dfa-9d5a-91fd4894ed34-dshm\") pod \"praxis-route-d1d441-kserve-698cd98df-qv4m7\" (UID: \"d20ddaa5-bf86-4dfa-9d5a-91fd4894ed34\") " pod="ai-tenant-e2e-praxis-a-d1d441/praxis-route-d1d441-kserve-698cd98df-qv4m7" Oct 31 04:29:12.416591 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:12.416569 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d20ddaa5-bf86-4dfa-9d5a-91fd4894ed34-home\") pod \"praxis-route-d1d441-kserve-698cd98df-qv4m7\" (UID: \"d20ddaa5-bf86-4dfa-9d5a-91fd4894ed34\") " pod="ai-tenant-e2e-praxis-a-d1d441/praxis-route-d1d441-kserve-698cd98df-qv4m7" Oct 31 04:29:12.416625 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:12.416595 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d20ddaa5-bf86-4dfa-9d5a-91fd4894ed34-tmp-dir\") pod \"praxis-route-d1d441-kserve-698cd98df-qv4m7\" (UID: \"d20ddaa5-bf86-4dfa-9d5a-91fd4894ed34\") " pod="ai-tenant-e2e-praxis-a-d1d441/praxis-route-d1d441-kserve-698cd98df-qv4m7" Oct 31 04:29:12.416625 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:12.416610 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d20ddaa5-bf86-4dfa-9d5a-91fd4894ed34-tls-certs\") pod \"praxis-route-d1d441-kserve-698cd98df-qv4m7\" (UID: \"d20ddaa5-bf86-4dfa-9d5a-91fd4894ed34\") " pod="ai-tenant-e2e-praxis-a-d1d441/praxis-route-d1d441-kserve-698cd98df-qv4m7" Oct 31 04:29:12.444753 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:12.444685 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa104be0-8c9b-4345-bcaf-a0cc7d0bebb3" path="/var/lib/kubelet/pods/aa104be0-8c9b-4345-bcaf-a0cc7d0bebb3/volumes" Oct 31 04:29:12.446935 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:12.446913 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-praxis-a-d1d441\"/\"openshift-service-ca.crt\"" Oct 31 04:29:12.517830 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:12.517800 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d20ddaa5-bf86-4dfa-9d5a-91fd4894ed34-dshm\") pod \"praxis-route-d1d441-kserve-698cd98df-qv4m7\" (UID: \"d20ddaa5-bf86-4dfa-9d5a-91fd4894ed34\") " pod="ai-tenant-e2e-praxis-a-d1d441/praxis-route-d1d441-kserve-698cd98df-qv4m7" Oct 31 04:29:12.518009 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:12.517862 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d20ddaa5-bf86-4dfa-9d5a-91fd4894ed34-home\") pod \"praxis-route-d1d441-kserve-698cd98df-qv4m7\" (UID: \"d20ddaa5-bf86-4dfa-9d5a-91fd4894ed34\") " pod="ai-tenant-e2e-praxis-a-d1d441/praxis-route-d1d441-kserve-698cd98df-qv4m7" Oct 31 04:29:12.518009 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:12.517886 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d20ddaa5-bf86-4dfa-9d5a-91fd4894ed34-tmp-dir\") pod \"praxis-route-d1d441-kserve-698cd98df-qv4m7\" (UID: \"d20ddaa5-bf86-4dfa-9d5a-91fd4894ed34\") " pod="ai-tenant-e2e-praxis-a-d1d441/praxis-route-d1d441-kserve-698cd98df-qv4m7" Oct 31 04:29:12.518009 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:12.517901 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d20ddaa5-bf86-4dfa-9d5a-91fd4894ed34-tls-certs\") pod \"praxis-route-d1d441-kserve-698cd98df-qv4m7\" (UID: \"d20ddaa5-bf86-4dfa-9d5a-91fd4894ed34\") " pod="ai-tenant-e2e-praxis-a-d1d441/praxis-route-d1d441-kserve-698cd98df-qv4m7" Oct 31 04:29:12.518189 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:12.518043 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d20ddaa5-bf86-4dfa-9d5a-91fd4894ed34-model-cache\") pod \"praxis-route-d1d441-kserve-698cd98df-qv4m7\" (UID: \"d20ddaa5-bf86-4dfa-9d5a-91fd4894ed34\") " pod="ai-tenant-e2e-praxis-a-d1d441/praxis-route-d1d441-kserve-698cd98df-qv4m7" Oct 31 04:29:12.518189 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:12.518161 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mwq8\" (UniqueName: \"kubernetes.io/projected/d20ddaa5-bf86-4dfa-9d5a-91fd4894ed34-kube-api-access-4mwq8\") pod \"praxis-route-d1d441-kserve-698cd98df-qv4m7\" (UID: \"d20ddaa5-bf86-4dfa-9d5a-91fd4894ed34\") " pod="ai-tenant-e2e-praxis-a-d1d441/praxis-route-d1d441-kserve-698cd98df-qv4m7" Oct 31 04:29:12.518344 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:12.518301 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d20ddaa5-bf86-4dfa-9d5a-91fd4894ed34-home\") pod \"praxis-route-d1d441-kserve-698cd98df-qv4m7\" (UID: \"d20ddaa5-bf86-4dfa-9d5a-91fd4894ed34\") " pod="ai-tenant-e2e-praxis-a-d1d441/praxis-route-d1d441-kserve-698cd98df-qv4m7" Oct 31 04:29:12.518426 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:12.518365 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d20ddaa5-bf86-4dfa-9d5a-91fd4894ed34-tmp-dir\") pod \"praxis-route-d1d441-kserve-698cd98df-qv4m7\" (UID: \"d20ddaa5-bf86-4dfa-9d5a-91fd4894ed34\") " pod="ai-tenant-e2e-praxis-a-d1d441/praxis-route-d1d441-kserve-698cd98df-qv4m7" Oct 31 04:29:12.518488 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:12.518433 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d20ddaa5-bf86-4dfa-9d5a-91fd4894ed34-model-cache\") pod \"praxis-route-d1d441-kserve-698cd98df-qv4m7\" (UID: \"d20ddaa5-bf86-4dfa-9d5a-91fd4894ed34\") " pod="ai-tenant-e2e-praxis-a-d1d441/praxis-route-d1d441-kserve-698cd98df-qv4m7" Oct 31 04:29:12.520153 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:12.520129 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d20ddaa5-bf86-4dfa-9d5a-91fd4894ed34-dshm\") pod \"praxis-route-d1d441-kserve-698cd98df-qv4m7\" (UID: \"d20ddaa5-bf86-4dfa-9d5a-91fd4894ed34\") " pod="ai-tenant-e2e-praxis-a-d1d441/praxis-route-d1d441-kserve-698cd98df-qv4m7" Oct 31 04:29:12.532172 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:12.532153 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-praxis-a-d1d441\"/\"praxis-route-d1d441-kserve-self-signed-certs\"" Oct 31 04:29:12.571360 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:12.570047 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d20ddaa5-bf86-4dfa-9d5a-91fd4894ed34-tls-certs\") pod \"praxis-route-d1d441-kserve-698cd98df-qv4m7\" (UID: \"d20ddaa5-bf86-4dfa-9d5a-91fd4894ed34\") " pod="ai-tenant-e2e-praxis-a-d1d441/praxis-route-d1d441-kserve-698cd98df-qv4m7" Oct 31 04:29:12.571360 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:12.570448 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-praxis-a-d1d441\"/\"kube-root-ca.crt\"" Oct 31 04:29:12.591371 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:12.588760 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-praxis-a-d1d441\"/\"openshift-service-ca.crt\"" Oct 31 04:29:12.611407 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:12.606005 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mwq8\" (UniqueName: \"kubernetes.io/projected/d20ddaa5-bf86-4dfa-9d5a-91fd4894ed34-kube-api-access-4mwq8\") pod \"praxis-route-d1d441-kserve-698cd98df-qv4m7\" (UID: \"d20ddaa5-bf86-4dfa-9d5a-91fd4894ed34\") " pod="ai-tenant-e2e-praxis-a-d1d441/praxis-route-d1d441-kserve-698cd98df-qv4m7" Oct 31 04:29:12.667675 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:12.667633 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-praxis-a-d1d441\"/\"default-dockercfg-4pt7c\"" Oct 31 04:29:12.671379 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:12.670087 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-d1d441/praxis-route-d1d441-kserve-698cd98df-qv4m7" Oct 31 04:29:13.032790 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:29:13.032760 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd20ddaa5_bf86_4dfa_9d5a_91fd4894ed34.slice/crio-cc24908328282c321dc4860f3312aceb4ccac54eccb301c0ccb74237231f4e55 WatchSource:0}: Error finding container cc24908328282c321dc4860f3312aceb4ccac54eccb301c0ccb74237231f4e55: Status 404 returned error can't find the container with id cc24908328282c321dc4860f3312aceb4ccac54eccb301c0ccb74237231f4e55 Oct 31 04:29:13.032903 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:13.032867 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-praxis-a-d1d441/praxis-route-d1d441-kserve-698cd98df-qv4m7"] Oct 31 04:29:14.062358 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:14.061551 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-d1d441/praxis-route-d1d441-kserve-698cd98df-qv4m7" event={"ID":"d20ddaa5-bf86-4dfa-9d5a-91fd4894ed34","Type":"ContainerStarted","Data":"4697d65bd32a36600078af2510616c6be9c2c4bee91d778dd22f470a8977adec"} Oct 31 04:29:14.062358 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:14.061606 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-d1d441/praxis-route-d1d441-kserve-698cd98df-qv4m7" event={"ID":"d20ddaa5-bf86-4dfa-9d5a-91fd4894ed34","Type":"ContainerStarted","Data":"cc24908328282c321dc4860f3312aceb4ccac54eccb301c0ccb74237231f4e55"} Oct 31 04:29:14.109846 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:14.109779 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-praxis-a-d1d441/praxis-route-d1d441-kserve-698cd98df-qv4m7" podStartSLOduration=2.109759328 podStartE2EDuration="2.109759328s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:29:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:29:14.105170947 +0000 UTC m=+1501.819505561" watchObservedRunningTime="2026-10-31 04:29:14.109759328 +0000 UTC m=+1501.824093951" Oct 31 04:29:14.670708 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:14.670648 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-praxis-a-d1d441/praxis-route-d1d441-kserve-698cd98df-qv4m7" Oct 31 04:29:22.670987 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:22.670950 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-praxis-a-d1d441/praxis-route-d1d441-kserve-698cd98df-qv4m7" Oct 31 04:29:22.683760 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:22.683730 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-praxis-a-d1d441/praxis-route-d1d441-kserve-698cd98df-qv4m7" Oct 31 04:29:23.194544 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:23.194514 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-praxis-a-d1d441/praxis-route-d1d441-kserve-698cd98df-qv4m7" Oct 31 04:29:27.824055 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:27.824020 4376 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-cpb5g_83cfbbfb-a15c-4967-b20c-e102351c6b7b/payload-processing/0.log" Oct 31 04:29:28.043735 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:28.043707 4376 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-cpb5g_83cfbbfb-a15c-4967-b20c-e102351c6b7b/payload-processing/0.log" Oct 31 04:29:30.618946 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:30.618907 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-c6859a-5cbcf46d46-c4t77"] Oct 31 04:29:30.619449 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:30.619144 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c6859a-5cbcf46d46-c4t77" podUID="ef7c032f-cdf7-43e3-9c9e-ef46f21ac3e4" containerName="maas-api" containerID="cri-o://2ad44c477aad082341fc8196ef46f6ce13c72d97894fa89491608b7a39c19044" gracePeriod=30 Oct 31 04:29:30.877973 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:30.877911 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c6859a-5cbcf46d46-c4t77" Oct 31 04:29:30.965428 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:30.956493 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef7c032f-cdf7-43e3-9c9e-ef46f21ac3e4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef7c032f-cdf7-43e3-9c9e-ef46f21ac3e4-openshift-service-ca\") pod \"ef7c032f-cdf7-43e3-9c9e-ef46f21ac3e4\" (UID: \"ef7c032f-cdf7-43e3-9c9e-ef46f21ac3e4\") " Oct 31 04:29:30.965428 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:30.956574 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef7c032f-cdf7-43e3-9c9e-ef46f21ac3e4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef7c032f-cdf7-43e3-9c9e-ef46f21ac3e4-maas-api-tls\") pod \"ef7c032f-cdf7-43e3-9c9e-ef46f21ac3e4\" (UID: \"ef7c032f-cdf7-43e3-9c9e-ef46f21ac3e4\") " Oct 31 04:29:30.965428 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:30.956786 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef7c032f-cdf7-43e3-9c9e-ef46f21ac3e4-kube-api-access-nz5qq\" (UniqueName: \"kubernetes.io/projected/ef7c032f-cdf7-43e3-9c9e-ef46f21ac3e4-kube-api-access-nz5qq\") pod \"ef7c032f-cdf7-43e3-9c9e-ef46f21ac3e4\" (UID: \"ef7c032f-cdf7-43e3-9c9e-ef46f21ac3e4\") " Oct 31 04:29:30.965428 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:30.956840 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef7c032f-cdf7-43e3-9c9e-ef46f21ac3e4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef7c032f-cdf7-43e3-9c9e-ef46f21ac3e4-metrics-certs\") pod \"ef7c032f-cdf7-43e3-9c9e-ef46f21ac3e4\" (UID: \"ef7c032f-cdf7-43e3-9c9e-ef46f21ac3e4\") " Oct 31 04:29:30.965428 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:30.960679 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef7c032f-cdf7-43e3-9c9e-ef46f21ac3e4-openshift-service-ca" pod "ef7c032f-cdf7-43e3-9c9e-ef46f21ac3e4" (UID: "ef7c032f-cdf7-43e3-9c9e-ef46f21ac3e4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:29:31.021141 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:31.019913 4376 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef7c032f-cdf7-43e3-9c9e-ef46f21ac3e4-openshift-service-ca\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:29:31.041395 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:31.027465 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef7c032f-cdf7-43e3-9c9e-ef46f21ac3e4-metrics-certs" pod "ef7c032f-cdf7-43e3-9c9e-ef46f21ac3e4" (UID: "ef7c032f-cdf7-43e3-9c9e-ef46f21ac3e4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:29:31.041395 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:31.038869 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef7c032f-cdf7-43e3-9c9e-ef46f21ac3e4-maas-api-tls" pod "ef7c032f-cdf7-43e3-9c9e-ef46f21ac3e4" (UID: "ef7c032f-cdf7-43e3-9c9e-ef46f21ac3e4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:29:31.064361 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:31.063783 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef7c032f-cdf7-43e3-9c9e-ef46f21ac3e4-kube-api-access-nz5qq" pod "ef7c032f-cdf7-43e3-9c9e-ef46f21ac3e4" (UID: "ef7c032f-cdf7-43e3-9c9e-ef46f21ac3e4"). InnerVolumeSpecName "kube-api-access-nz5qq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:29:31.214467 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:31.213437 4376 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef7c032f-cdf7-43e3-9c9e-ef46f21ac3e4-maas-api-tls\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:29:31.214467 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:31.213472 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nz5qq\" (UniqueName: \"kubernetes.io/projected/ef7c032f-cdf7-43e3-9c9e-ef46f21ac3e4-kube-api-access-nz5qq\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:29:31.214467 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:31.213489 4376 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef7c032f-cdf7-43e3-9c9e-ef46f21ac3e4-metrics-certs\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:29:31.291783 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:31.291746 4376 generic.go:356] "Generic (PLEG): container finished" podID="ef7c032f-cdf7-43e3-9c9e-ef46f21ac3e4" containerID="2ad44c477aad082341fc8196ef46f6ce13c72d97894fa89491608b7a39c19044" exitCode=0 Oct 31 04:29:31.292006 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:31.291807 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c6859a-5cbcf46d46-c4t77" Oct 31 04:29:31.292006 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:31.291809 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c6859a-5cbcf46d46-c4t77" event={"ID":"ef7c032f-cdf7-43e3-9c9e-ef46f21ac3e4","Type":"ContainerDied","Data":"2ad44c477aad082341fc8196ef46f6ce13c72d97894fa89491608b7a39c19044"} Oct 31 04:29:31.292006 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:31.291943 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c6859a-5cbcf46d46-c4t77" event={"ID":"ef7c032f-cdf7-43e3-9c9e-ef46f21ac3e4","Type":"ContainerDied","Data":"df23409131c05267a8b8d1326bca1afaceb56ae56bcaec17d029061630e55316"} Oct 31 04:29:31.292006 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:31.291973 4376 scope.go:122] "RemoveContainer" containerID="2ad44c477aad082341fc8196ef46f6ce13c72d97894fa89491608b7a39c19044" Oct 31 04:29:31.302125 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:31.302105 4376 scope.go:122] "RemoveContainer" containerID="2ad44c477aad082341fc8196ef46f6ce13c72d97894fa89491608b7a39c19044" Oct 31 04:29:31.302415 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:29:31.302397 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ad44c477aad082341fc8196ef46f6ce13c72d97894fa89491608b7a39c19044\": container with ID starting with 2ad44c477aad082341fc8196ef46f6ce13c72d97894fa89491608b7a39c19044 not found: ID does not exist" containerID="2ad44c477aad082341fc8196ef46f6ce13c72d97894fa89491608b7a39c19044" Oct 31 04:29:31.302469 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:31.302423 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ad44c477aad082341fc8196ef46f6ce13c72d97894fa89491608b7a39c19044"} err="failed to get container status \"2ad44c477aad082341fc8196ef46f6ce13c72d97894fa89491608b7a39c19044\": rpc error: code = NotFound desc = could not find container \"2ad44c477aad082341fc8196ef46f6ce13c72d97894fa89491608b7a39c19044\": container with ID starting with 2ad44c477aad082341fc8196ef46f6ce13c72d97894fa89491608b7a39c19044 not found: ID does not exist" Oct 31 04:29:31.346838 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:31.346808 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-c6859a-5cbcf46d46-c4t77"] Oct 31 04:29:31.351022 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:31.350999 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-c6859a-5cbcf46d46-c4t77"] Oct 31 04:29:32.444948 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:32.444912 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef7c032f-cdf7-43e3-9c9e-ef46f21ac3e4" path="/var/lib/kubelet/pods/ef7c032f-cdf7-43e3-9c9e-ef46f21ac3e4/volumes" Oct 31 04:29:35.641346 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:35.641292 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-ff43e3-6cc76d6dc7-ndh7f"] Oct 31 04:29:35.641858 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:35.641616 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ff43e3-6cc76d6dc7-ndh7f" podUID="ab191222-92d2-4794-8317-e2ce18b8dcc8" containerName="maas-api" containerID="cri-o://6fb530781c01089ebb4f2e166275f21635f6375bc13ac0183b36c77225ad1d4f" gracePeriod=30 Oct 31 04:29:35.946067 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:35.946038 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ff43e3-6cc76d6dc7-ndh7f" Oct 31 04:29:36.083801 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:36.083764 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab191222-92d2-4794-8317-e2ce18b8dcc8-kube-api-access-kpw4c\" (UniqueName: \"kubernetes.io/projected/ab191222-92d2-4794-8317-e2ce18b8dcc8-kube-api-access-kpw4c\") pod \"ab191222-92d2-4794-8317-e2ce18b8dcc8\" (UID: \"ab191222-92d2-4794-8317-e2ce18b8dcc8\") " Oct 31 04:29:36.083978 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:36.083827 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab191222-92d2-4794-8317-e2ce18b8dcc8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab191222-92d2-4794-8317-e2ce18b8dcc8-metrics-certs\") pod \"ab191222-92d2-4794-8317-e2ce18b8dcc8\" (UID: \"ab191222-92d2-4794-8317-e2ce18b8dcc8\") " Oct 31 04:29:36.083978 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:36.083901 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab191222-92d2-4794-8317-e2ce18b8dcc8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab191222-92d2-4794-8317-e2ce18b8dcc8-openshift-service-ca\") pod \"ab191222-92d2-4794-8317-e2ce18b8dcc8\" (UID: \"ab191222-92d2-4794-8317-e2ce18b8dcc8\") " Oct 31 04:29:36.084066 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:36.083994 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab191222-92d2-4794-8317-e2ce18b8dcc8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab191222-92d2-4794-8317-e2ce18b8dcc8-maas-api-tls\") pod \"ab191222-92d2-4794-8317-e2ce18b8dcc8\" (UID: \"ab191222-92d2-4794-8317-e2ce18b8dcc8\") " Oct 31 04:29:36.084317 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:36.084287 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab191222-92d2-4794-8317-e2ce18b8dcc8-openshift-service-ca" pod "ab191222-92d2-4794-8317-e2ce18b8dcc8" (UID: "ab191222-92d2-4794-8317-e2ce18b8dcc8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:29:36.084668 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:36.084641 4376 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab191222-92d2-4794-8317-e2ce18b8dcc8-openshift-service-ca\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:29:36.086024 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:36.086002 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab191222-92d2-4794-8317-e2ce18b8dcc8-kube-api-access-kpw4c" pod "ab191222-92d2-4794-8317-e2ce18b8dcc8" (UID: "ab191222-92d2-4794-8317-e2ce18b8dcc8"). InnerVolumeSpecName "kube-api-access-kpw4c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:29:36.086131 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:36.086103 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab191222-92d2-4794-8317-e2ce18b8dcc8-maas-api-tls" pod "ab191222-92d2-4794-8317-e2ce18b8dcc8" (UID: "ab191222-92d2-4794-8317-e2ce18b8dcc8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:29:36.086212 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:36.086187 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab191222-92d2-4794-8317-e2ce18b8dcc8-metrics-certs" pod "ab191222-92d2-4794-8317-e2ce18b8dcc8" (UID: "ab191222-92d2-4794-8317-e2ce18b8dcc8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:29:36.189846 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:36.189706 4376 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab191222-92d2-4794-8317-e2ce18b8dcc8-maas-api-tls\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:29:36.189846 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:36.189747 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kpw4c\" (UniqueName: \"kubernetes.io/projected/ab191222-92d2-4794-8317-e2ce18b8dcc8-kube-api-access-kpw4c\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:29:36.189846 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:36.189765 4376 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab191222-92d2-4794-8317-e2ce18b8dcc8-metrics-certs\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:29:36.337636 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:36.337598 4376 generic.go:356] "Generic (PLEG): container finished" podID="ab191222-92d2-4794-8317-e2ce18b8dcc8" containerID="6fb530781c01089ebb4f2e166275f21635f6375bc13ac0183b36c77225ad1d4f" exitCode=0 Oct 31 04:29:36.337847 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:36.337662 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ff43e3-6cc76d6dc7-ndh7f" Oct 31 04:29:36.337847 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:36.337706 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ff43e3-6cc76d6dc7-ndh7f" event={"ID":"ab191222-92d2-4794-8317-e2ce18b8dcc8","Type":"ContainerDied","Data":"6fb530781c01089ebb4f2e166275f21635f6375bc13ac0183b36c77225ad1d4f"} Oct 31 04:29:36.337847 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:36.337778 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ff43e3-6cc76d6dc7-ndh7f" event={"ID":"ab191222-92d2-4794-8317-e2ce18b8dcc8","Type":"ContainerDied","Data":"bf56b40a2e3240ee92e7962573d200431013d3ab93d5478f4a73656d157c93ee"} Oct 31 04:29:36.337847 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:36.337808 4376 scope.go:122] "RemoveContainer" containerID="6fb530781c01089ebb4f2e166275f21635f6375bc13ac0183b36c77225ad1d4f" Oct 31 04:29:36.348517 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:36.348492 4376 scope.go:122] "RemoveContainer" containerID="6fb530781c01089ebb4f2e166275f21635f6375bc13ac0183b36c77225ad1d4f" Oct 31 04:29:36.348884 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:29:36.348839 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6fb530781c01089ebb4f2e166275f21635f6375bc13ac0183b36c77225ad1d4f\": container with ID starting with 6fb530781c01089ebb4f2e166275f21635f6375bc13ac0183b36c77225ad1d4f not found: ID does not exist" containerID="6fb530781c01089ebb4f2e166275f21635f6375bc13ac0183b36c77225ad1d4f" Oct 31 04:29:36.348884 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:36.348868 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fb530781c01089ebb4f2e166275f21635f6375bc13ac0183b36c77225ad1d4f"} err="failed to get container status \"6fb530781c01089ebb4f2e166275f21635f6375bc13ac0183b36c77225ad1d4f\": rpc error: code = NotFound desc = could not find container \"6fb530781c01089ebb4f2e166275f21635f6375bc13ac0183b36c77225ad1d4f\": container with ID starting with 6fb530781c01089ebb4f2e166275f21635f6375bc13ac0183b36c77225ad1d4f not found: ID does not exist" Oct 31 04:29:36.392638 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:36.392608 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-ff43e3-6cc76d6dc7-ndh7f"] Oct 31 04:29:36.405192 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:36.405160 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-ff43e3-6cc76d6dc7-ndh7f"] Oct 31 04:29:36.445105 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:36.445026 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab191222-92d2-4794-8317-e2ce18b8dcc8" path="/var/lib/kubelet/pods/ab191222-92d2-4794-8317-e2ce18b8dcc8/volumes" Oct 31 04:29:41.006753 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:29:41.006710 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 04:29:41.265816 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:41.265729 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 04:29:46.716477 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:46.716441 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-edc450-7b5666bb59-jnjrl"] Oct 31 04:29:50.609321 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:50.609281 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-7cf9ef-59747bbd5-tbh6h"] Oct 31 04:29:50.609946 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:50.609931 4376 cpu_manager.go:414] "containerMap: removing container" podUID="ef7c032f-cdf7-43e3-9c9e-ef46f21ac3e4" containerName="maas-api" Oct 31 04:29:50.609994 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:50.609948 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef7c032f-cdf7-43e3-9c9e-ef46f21ac3e4" containerName="maas-api" Oct 31 04:29:50.609994 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:50.609989 4376 cpu_manager.go:414] "containerMap: removing container" podUID="ab191222-92d2-4794-8317-e2ce18b8dcc8" containerName="maas-api" Oct 31 04:29:50.610056 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:50.609994 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab191222-92d2-4794-8317-e2ce18b8dcc8" containerName="maas-api" Oct 31 04:29:50.610122 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:50.610112 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab191222-92d2-4794-8317-e2ce18b8dcc8" containerName="maas-api" Oct 31 04:29:50.610162 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:50.610127 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef7c032f-cdf7-43e3-9c9e-ef46f21ac3e4" containerName="maas-api" Oct 31 04:29:50.614727 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:50.614709 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7cf9ef-59747bbd5-tbh6h" Oct 31 04:29:50.630540 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:50.630512 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-7cf9ef-serving-cert\"" Oct 31 04:29:50.649778 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:50.649740 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-7cf9ef-59747bbd5-tbh6h"] Oct 31 04:29:50.716399 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:50.716360 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/125f6f3a-41d5-48c3-bbe1-0d52682b0d11-metrics-certs\") pod \"maas-api-e2e-praxis-b-7cf9ef-59747bbd5-tbh6h\" (UID: \"125f6f3a-41d5-48c3-bbe1-0d52682b0d11\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7cf9ef-59747bbd5-tbh6h" Oct 31 04:29:50.716620 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:50.716464 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/125f6f3a-41d5-48c3-bbe1-0d52682b0d11-maas-api-tls\") pod \"maas-api-e2e-praxis-b-7cf9ef-59747bbd5-tbh6h\" (UID: \"125f6f3a-41d5-48c3-bbe1-0d52682b0d11\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7cf9ef-59747bbd5-tbh6h" Oct 31 04:29:50.716620 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:50.716579 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/125f6f3a-41d5-48c3-bbe1-0d52682b0d11-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-7cf9ef-59747bbd5-tbh6h\" (UID: \"125f6f3a-41d5-48c3-bbe1-0d52682b0d11\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7cf9ef-59747bbd5-tbh6h" Oct 31 04:29:50.716716 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:50.716648 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrdc4\" (UniqueName: \"kubernetes.io/projected/125f6f3a-41d5-48c3-bbe1-0d52682b0d11-kube-api-access-zrdc4\") pod \"maas-api-e2e-praxis-b-7cf9ef-59747bbd5-tbh6h\" (UID: \"125f6f3a-41d5-48c3-bbe1-0d52682b0d11\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7cf9ef-59747bbd5-tbh6h" Oct 31 04:29:50.818464 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:50.818427 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/125f6f3a-41d5-48c3-bbe1-0d52682b0d11-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-7cf9ef-59747bbd5-tbh6h\" (UID: \"125f6f3a-41d5-48c3-bbe1-0d52682b0d11\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7cf9ef-59747bbd5-tbh6h" Oct 31 04:29:50.818656 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:50.818521 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrdc4\" (UniqueName: \"kubernetes.io/projected/125f6f3a-41d5-48c3-bbe1-0d52682b0d11-kube-api-access-zrdc4\") pod \"maas-api-e2e-praxis-b-7cf9ef-59747bbd5-tbh6h\" (UID: \"125f6f3a-41d5-48c3-bbe1-0d52682b0d11\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7cf9ef-59747bbd5-tbh6h" Oct 31 04:29:50.818656 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:50.818647 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/125f6f3a-41d5-48c3-bbe1-0d52682b0d11-metrics-certs\") pod \"maas-api-e2e-praxis-b-7cf9ef-59747bbd5-tbh6h\" (UID: \"125f6f3a-41d5-48c3-bbe1-0d52682b0d11\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7cf9ef-59747bbd5-tbh6h" Oct 31 04:29:50.818731 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:50.818679 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/125f6f3a-41d5-48c3-bbe1-0d52682b0d11-maas-api-tls\") pod \"maas-api-e2e-praxis-b-7cf9ef-59747bbd5-tbh6h\" (UID: \"125f6f3a-41d5-48c3-bbe1-0d52682b0d11\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7cf9ef-59747bbd5-tbh6h" Oct 31 04:29:50.819243 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:50.819210 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/125f6f3a-41d5-48c3-bbe1-0d52682b0d11-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-7cf9ef-59747bbd5-tbh6h\" (UID: \"125f6f3a-41d5-48c3-bbe1-0d52682b0d11\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7cf9ef-59747bbd5-tbh6h" Oct 31 04:29:50.821252 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:50.821230 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/125f6f3a-41d5-48c3-bbe1-0d52682b0d11-maas-api-tls\") pod \"maas-api-e2e-praxis-b-7cf9ef-59747bbd5-tbh6h\" (UID: \"125f6f3a-41d5-48c3-bbe1-0d52682b0d11\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7cf9ef-59747bbd5-tbh6h" Oct 31 04:29:50.821252 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:50.821247 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/125f6f3a-41d5-48c3-bbe1-0d52682b0d11-metrics-certs\") pod \"maas-api-e2e-praxis-b-7cf9ef-59747bbd5-tbh6h\" (UID: \"125f6f3a-41d5-48c3-bbe1-0d52682b0d11\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7cf9ef-59747bbd5-tbh6h" Oct 31 04:29:50.842186 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:50.842155 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrdc4\" (UniqueName: \"kubernetes.io/projected/125f6f3a-41d5-48c3-bbe1-0d52682b0d11-kube-api-access-zrdc4\") pod \"maas-api-e2e-praxis-b-7cf9ef-59747bbd5-tbh6h\" (UID: \"125f6f3a-41d5-48c3-bbe1-0d52682b0d11\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7cf9ef-59747bbd5-tbh6h" Oct 31 04:29:50.933840 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:50.932677 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7cf9ef-59747bbd5-tbh6h" Oct 31 04:29:51.318512 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:29:51.318466 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod125f6f3a_41d5_48c3_bbe1_0d52682b0d11.slice/crio-70f6b6a2068b7983f787a9225a8fa1dc859445633a5992d04d444f4de3f6cd44 WatchSource:0}: Error finding container 70f6b6a2068b7983f787a9225a8fa1dc859445633a5992d04d444f4de3f6cd44: Status 404 returned error can't find the container with id 70f6b6a2068b7983f787a9225a8fa1dc859445633a5992d04d444f4de3f6cd44 Oct 31 04:29:51.319520 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:51.319493 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-7cf9ef-59747bbd5-tbh6h"] Oct 31 04:29:51.463116 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:51.463076 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7cf9ef-59747bbd5-tbh6h" event={"ID":"125f6f3a-41d5-48c3-bbe1-0d52682b0d11","Type":"ContainerStarted","Data":"70f6b6a2068b7983f787a9225a8fa1dc859445633a5992d04d444f4de3f6cd44"} Oct 31 04:29:51.717196 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:51.717151 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-edc450-7b5666bb59-jnjrl" podUID="03cd0818-43b9-4122-bb8b-d89063169326" containerName="payload-pre-processing" containerID="cri-o://6b6191984193535b65425777c6b06d836aa1bbcb782e8a483c6fdd9cb0c115f8" gracePeriod=25 Oct 31 04:29:52.124274 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:52.124246 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-edc450-7b5666bb59-jnjrl" Oct 31 04:29:52.249207 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:52.249175 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03cd0818-43b9-4122-bb8b-d89063169326-kube-api-access-6d87z\" (UniqueName: \"kubernetes.io/projected/03cd0818-43b9-4122-bb8b-d89063169326-kube-api-access-6d87z\") pod \"03cd0818-43b9-4122-bb8b-d89063169326\" (UID: \"03cd0818-43b9-4122-bb8b-d89063169326\") " Oct 31 04:29:52.249491 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:52.249479 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03cd0818-43b9-4122-bb8b-d89063169326-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03cd0818-43b9-4122-bb8b-d89063169326-plugins-config-volume\") pod \"03cd0818-43b9-4122-bb8b-d89063169326\" (UID: \"03cd0818-43b9-4122-bb8b-d89063169326\") " Oct 31 04:29:52.249846 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:52.249816 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03cd0818-43b9-4122-bb8b-d89063169326-plugins-config-volume" pod "03cd0818-43b9-4122-bb8b-d89063169326" (UID: "03cd0818-43b9-4122-bb8b-d89063169326"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:29:52.251437 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:52.251410 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03cd0818-43b9-4122-bb8b-d89063169326-kube-api-access-6d87z" pod "03cd0818-43b9-4122-bb8b-d89063169326" (UID: "03cd0818-43b9-4122-bb8b-d89063169326"). InnerVolumeSpecName "kube-api-access-6d87z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:29:52.351096 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:52.351009 4376 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03cd0818-43b9-4122-bb8b-d89063169326-plugins-config-volume\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:29:52.351096 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:52.351041 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6d87z\" (UniqueName: \"kubernetes.io/projected/03cd0818-43b9-4122-bb8b-d89063169326-kube-api-access-6d87z\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:29:52.472192 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:52.472152 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7cf9ef-59747bbd5-tbh6h" event={"ID":"125f6f3a-41d5-48c3-bbe1-0d52682b0d11","Type":"ContainerStarted","Data":"d0bd93308eb9d0b0b2ec78c726416696b000c760fadfca2aae031a087702be20"} Oct 31 04:29:52.472425 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:52.472250 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7cf9ef-59747bbd5-tbh6h" Oct 31 04:29:52.473983 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:52.473952 4376 generic.go:356] "Generic (PLEG): container finished" podID="03cd0818-43b9-4122-bb8b-d89063169326" containerID="6b6191984193535b65425777c6b06d836aa1bbcb782e8a483c6fdd9cb0c115f8" exitCode=0 Oct 31 04:29:52.474101 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:52.473999 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-edc450-7b5666bb59-jnjrl" Oct 31 04:29:52.474101 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:52.474018 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-edc450-7b5666bb59-jnjrl" event={"ID":"03cd0818-43b9-4122-bb8b-d89063169326","Type":"ContainerDied","Data":"6b6191984193535b65425777c6b06d836aa1bbcb782e8a483c6fdd9cb0c115f8"} Oct 31 04:29:52.474101 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:52.474065 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-edc450-7b5666bb59-jnjrl" event={"ID":"03cd0818-43b9-4122-bb8b-d89063169326","Type":"ContainerDied","Data":"b2b65f090320178f6c7dc7e01c0623dd504480ca82801ee81492f133e13aa698"} Oct 31 04:29:52.474101 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:52.474094 4376 scope.go:122] "RemoveContainer" containerID="6b6191984193535b65425777c6b06d836aa1bbcb782e8a483c6fdd9cb0c115f8" Oct 31 04:29:52.485185 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:52.485161 4376 scope.go:122] "RemoveContainer" containerID="6b6191984193535b65425777c6b06d836aa1bbcb782e8a483c6fdd9cb0c115f8" Oct 31 04:29:52.485589 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:29:52.485530 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b6191984193535b65425777c6b06d836aa1bbcb782e8a483c6fdd9cb0c115f8\": container with ID starting with 6b6191984193535b65425777c6b06d836aa1bbcb782e8a483c6fdd9cb0c115f8 not found: ID does not exist" containerID="6b6191984193535b65425777c6b06d836aa1bbcb782e8a483c6fdd9cb0c115f8" Oct 31 04:29:52.485589 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:52.485563 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b6191984193535b65425777c6b06d836aa1bbcb782e8a483c6fdd9cb0c115f8"} err="failed to get container status \"6b6191984193535b65425777c6b06d836aa1bbcb782e8a483c6fdd9cb0c115f8\": rpc error: code = NotFound desc = could not find container \"6b6191984193535b65425777c6b06d836aa1bbcb782e8a483c6fdd9cb0c115f8\": container with ID starting with 6b6191984193535b65425777c6b06d836aa1bbcb782e8a483c6fdd9cb0c115f8 not found: ID does not exist" Oct 31 04:29:52.516555 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:52.516500 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7cf9ef-59747bbd5-tbh6h" podStartSLOduration=2.128170709 podStartE2EDuration="2.516486396s" totalImagesPullingTime="388.315687ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:29:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:29:52.51470462 +0000 UTC m=+1540.229039238" watchObservedRunningTime="2026-10-31 04:29:52.516486396 +0000 UTC m=+1540.230821013" Oct 31 04:29:52.553531 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:52.553493 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-edc450-7b5666bb59-jnjrl"] Oct 31 04:29:52.557807 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:52.557777 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-edc450-7b5666bb59-jnjrl"] Oct 31 04:29:54.452824 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:54.452782 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03cd0818-43b9-4122-bb8b-d89063169326" path="/var/lib/kubelet/pods/03cd0818-43b9-4122-bb8b-d89063169326/volumes" Oct 31 04:29:55.302216 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:55.302181 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-wq9cv"] Oct 31 04:29:55.302807 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:55.302792 4376 cpu_manager.go:414] "containerMap: removing container" podUID="03cd0818-43b9-4122-bb8b-d89063169326" containerName="payload-pre-processing" Oct 31 04:29:55.302860 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:55.302808 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03cd0818-43b9-4122-bb8b-d89063169326" containerName="payload-pre-processing" Oct 31 04:29:55.302950 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:55.302940 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="03cd0818-43b9-4122-bb8b-d89063169326" containerName="payload-pre-processing" Oct 31 04:29:55.307843 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:55.307823 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-wq9cv" Oct 31 04:29:55.321314 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:55.321286 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-rw9sz"] Oct 31 04:29:55.326153 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:55.326125 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-rw9sz" Oct 31 04:29:55.329087 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:55.329056 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-wq9cv"] Oct 31 04:29:55.351279 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:55.351247 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-rw9sz"] Oct 31 04:29:55.406891 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:55.406848 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17ea2397-450e-4505-9a48-0cf93f76065e-istio-envoy\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-wq9cv\" (UID: \"17ea2397-450e-4505-9a48-0cf93f76065e\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-wq9cv" Oct 31 04:29:55.406891 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:55.406895 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17ea2397-450e-4505-9a48-0cf93f76065e-istio-podinfo\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-wq9cv\" (UID: \"17ea2397-450e-4505-9a48-0cf93f76065e\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-wq9cv" Oct 31 04:29:55.407173 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:55.407060 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17ea2397-450e-4505-9a48-0cf93f76065e-istio-token\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-wq9cv\" (UID: \"17ea2397-450e-4505-9a48-0cf93f76065e\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-wq9cv" Oct 31 04:29:55.407173 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:55.407114 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8qdc\" (UniqueName: \"kubernetes.io/projected/17ea2397-450e-4505-9a48-0cf93f76065e-kube-api-access-n8qdc\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-wq9cv\" (UID: \"17ea2397-450e-4505-9a48-0cf93f76065e\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-wq9cv" Oct 31 04:29:55.407285 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:55.407198 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17ea2397-450e-4505-9a48-0cf93f76065e-istiod-ca-cert\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-wq9cv\" (UID: \"17ea2397-450e-4505-9a48-0cf93f76065e\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-wq9cv" Oct 31 04:29:55.407285 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:55.407260 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17ea2397-450e-4505-9a48-0cf93f76065e-workload-certs\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-wq9cv\" (UID: \"17ea2397-450e-4505-9a48-0cf93f76065e\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-wq9cv" Oct 31 04:29:55.407285 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:55.407284 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17ea2397-450e-4505-9a48-0cf93f76065e-istio-data\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-wq9cv\" (UID: \"17ea2397-450e-4505-9a48-0cf93f76065e\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-wq9cv" Oct 31 04:29:55.407505 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:55.407440 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17ea2397-450e-4505-9a48-0cf93f76065e-workload-socket\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-wq9cv\" (UID: \"17ea2397-450e-4505-9a48-0cf93f76065e\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-wq9cv" Oct 31 04:29:55.407567 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:55.407532 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17ea2397-450e-4505-9a48-0cf93f76065e-credential-socket\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-wq9cv\" (UID: \"17ea2397-450e-4505-9a48-0cf93f76065e\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-wq9cv" Oct 31 04:29:55.509156 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:55.509113 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17ea2397-450e-4505-9a48-0cf93f76065e-workload-certs\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-wq9cv\" (UID: \"17ea2397-450e-4505-9a48-0cf93f76065e\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-wq9cv" Oct 31 04:29:55.509719 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:55.509175 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ead2084a-70a7-47d2-a757-f340d971bfb6-credential-socket\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-rw9sz\" (UID: \"ead2084a-70a7-47d2-a757-f340d971bfb6\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-rw9sz" Oct 31 04:29:55.509719 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:55.509214 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17ea2397-450e-4505-9a48-0cf93f76065e-istio-data\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-wq9cv\" (UID: \"17ea2397-450e-4505-9a48-0cf93f76065e\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-wq9cv" Oct 31 04:29:55.509719 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:55.509412 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17ea2397-450e-4505-9a48-0cf93f76065e-workload-socket\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-wq9cv\" (UID: \"17ea2397-450e-4505-9a48-0cf93f76065e\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-wq9cv" Oct 31 04:29:55.509719 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:55.509498 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ead2084a-70a7-47d2-a757-f340d971bfb6-istio-token\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-rw9sz\" (UID: \"ead2084a-70a7-47d2-a757-f340d971bfb6\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-rw9sz" Oct 31 04:29:55.509719 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:55.509536 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17ea2397-450e-4505-9a48-0cf93f76065e-credential-socket\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-wq9cv\" (UID: \"17ea2397-450e-4505-9a48-0cf93f76065e\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-wq9cv" Oct 31 04:29:55.509719 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:55.509563 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ead2084a-70a7-47d2-a757-f340d971bfb6-istiod-ca-cert\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-rw9sz\" (UID: \"ead2084a-70a7-47d2-a757-f340d971bfb6\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-rw9sz" Oct 31 04:29:55.509719 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:55.509578 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17ea2397-450e-4505-9a48-0cf93f76065e-workload-certs\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-wq9cv\" (UID: \"17ea2397-450e-4505-9a48-0cf93f76065e\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-wq9cv" Oct 31 04:29:55.510057 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:55.509720 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzhw7\" (UniqueName: \"kubernetes.io/projected/ead2084a-70a7-47d2-a757-f340d971bfb6-kube-api-access-mzhw7\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-rw9sz\" (UID: \"ead2084a-70a7-47d2-a757-f340d971bfb6\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-rw9sz" Oct 31 04:29:55.510057 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:55.509727 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17ea2397-450e-4505-9a48-0cf93f76065e-istio-data\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-wq9cv\" (UID: \"17ea2397-450e-4505-9a48-0cf93f76065e\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-wq9cv" Oct 31 04:29:55.510057 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:55.509788 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ead2084a-70a7-47d2-a757-f340d971bfb6-workload-certs\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-rw9sz\" (UID: \"ead2084a-70a7-47d2-a757-f340d971bfb6\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-rw9sz" Oct 31 04:29:55.510057 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:55.509801 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17ea2397-450e-4505-9a48-0cf93f76065e-workload-socket\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-wq9cv\" (UID: \"17ea2397-450e-4505-9a48-0cf93f76065e\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-wq9cv" Oct 31 04:29:55.510057 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:55.509876 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17ea2397-450e-4505-9a48-0cf93f76065e-credential-socket\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-wq9cv\" (UID: \"17ea2397-450e-4505-9a48-0cf93f76065e\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-wq9cv" Oct 31 04:29:55.510057 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:55.509896 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17ea2397-450e-4505-9a48-0cf93f76065e-istio-envoy\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-wq9cv\" (UID: \"17ea2397-450e-4505-9a48-0cf93f76065e\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-wq9cv" Oct 31 04:29:55.510057 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:55.509927 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17ea2397-450e-4505-9a48-0cf93f76065e-istio-podinfo\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-wq9cv\" (UID: \"17ea2397-450e-4505-9a48-0cf93f76065e\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-wq9cv" Oct 31 04:29:55.510057 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:55.509952 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ead2084a-70a7-47d2-a757-f340d971bfb6-workload-socket\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-rw9sz\" (UID: \"ead2084a-70a7-47d2-a757-f340d971bfb6\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-rw9sz" Oct 31 04:29:55.510057 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:55.509981 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ead2084a-70a7-47d2-a757-f340d971bfb6-istio-podinfo\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-rw9sz\" (UID: \"ead2084a-70a7-47d2-a757-f340d971bfb6\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-rw9sz" Oct 31 04:29:55.510542 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:55.510072 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ead2084a-70a7-47d2-a757-f340d971bfb6-istio-envoy\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-rw9sz\" (UID: \"ead2084a-70a7-47d2-a757-f340d971bfb6\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-rw9sz" Oct 31 04:29:55.510542 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:55.510103 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17ea2397-450e-4505-9a48-0cf93f76065e-istio-token\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-wq9cv\" (UID: \"17ea2397-450e-4505-9a48-0cf93f76065e\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-wq9cv" Oct 31 04:29:55.510542 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:55.510161 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8qdc\" (UniqueName: \"kubernetes.io/projected/17ea2397-450e-4505-9a48-0cf93f76065e-kube-api-access-n8qdc\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-wq9cv\" (UID: \"17ea2397-450e-4505-9a48-0cf93f76065e\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-wq9cv" Oct 31 04:29:55.510542 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:55.510220 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17ea2397-450e-4505-9a48-0cf93f76065e-istiod-ca-cert\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-wq9cv\" (UID: \"17ea2397-450e-4505-9a48-0cf93f76065e\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-wq9cv" Oct 31 04:29:55.510542 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:55.510262 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ead2084a-70a7-47d2-a757-f340d971bfb6-istio-data\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-rw9sz\" (UID: \"ead2084a-70a7-47d2-a757-f340d971bfb6\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-rw9sz" Oct 31 04:29:55.510850 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:55.510820 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17ea2397-450e-4505-9a48-0cf93f76065e-istiod-ca-cert\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-wq9cv\" (UID: \"17ea2397-450e-4505-9a48-0cf93f76065e\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-wq9cv" Oct 31 04:29:55.512069 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:55.512046 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17ea2397-450e-4505-9a48-0cf93f76065e-istio-envoy\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-wq9cv\" (UID: \"17ea2397-450e-4505-9a48-0cf93f76065e\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-wq9cv" Oct 31 04:29:55.512510 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:55.512490 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17ea2397-450e-4505-9a48-0cf93f76065e-istio-podinfo\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-wq9cv\" (UID: \"17ea2397-450e-4505-9a48-0cf93f76065e\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-wq9cv" Oct 31 04:29:55.533898 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:55.533866 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17ea2397-450e-4505-9a48-0cf93f76065e-istio-token\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-wq9cv\" (UID: \"17ea2397-450e-4505-9a48-0cf93f76065e\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-wq9cv" Oct 31 04:29:55.534108 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:55.534087 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8qdc\" (UniqueName: \"kubernetes.io/projected/17ea2397-450e-4505-9a48-0cf93f76065e-kube-api-access-n8qdc\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-wq9cv\" (UID: \"17ea2397-450e-4505-9a48-0cf93f76065e\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-wq9cv" Oct 31 04:29:55.611981 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:55.611894 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ead2084a-70a7-47d2-a757-f340d971bfb6-istio-envoy\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-rw9sz\" (UID: \"ead2084a-70a7-47d2-a757-f340d971bfb6\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-rw9sz" Oct 31 04:29:55.612141 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:55.611982 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ead2084a-70a7-47d2-a757-f340d971bfb6-istio-data\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-rw9sz\" (UID: \"ead2084a-70a7-47d2-a757-f340d971bfb6\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-rw9sz" Oct 31 04:29:55.612141 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:55.612014 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ead2084a-70a7-47d2-a757-f340d971bfb6-credential-socket\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-rw9sz\" (UID: \"ead2084a-70a7-47d2-a757-f340d971bfb6\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-rw9sz" Oct 31 04:29:55.612141 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:55.612098 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ead2084a-70a7-47d2-a757-f340d971bfb6-istio-token\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-rw9sz\" (UID: \"ead2084a-70a7-47d2-a757-f340d971bfb6\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-rw9sz" Oct 31 04:29:55.612141 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:55.612126 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ead2084a-70a7-47d2-a757-f340d971bfb6-istiod-ca-cert\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-rw9sz\" (UID: \"ead2084a-70a7-47d2-a757-f340d971bfb6\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-rw9sz" Oct 31 04:29:55.612620 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:55.612312 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzhw7\" (UniqueName: \"kubernetes.io/projected/ead2084a-70a7-47d2-a757-f340d971bfb6-kube-api-access-mzhw7\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-rw9sz\" (UID: \"ead2084a-70a7-47d2-a757-f340d971bfb6\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-rw9sz" Oct 31 04:29:55.612739 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:55.612646 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ead2084a-70a7-47d2-a757-f340d971bfb6-workload-certs\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-rw9sz\" (UID: \"ead2084a-70a7-47d2-a757-f340d971bfb6\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-rw9sz" Oct 31 04:29:55.612739 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:55.612431 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ead2084a-70a7-47d2-a757-f340d971bfb6-istio-data\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-rw9sz\" (UID: \"ead2084a-70a7-47d2-a757-f340d971bfb6\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-rw9sz" Oct 31 04:29:55.612739 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:55.612493 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ead2084a-70a7-47d2-a757-f340d971bfb6-credential-socket\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-rw9sz\" (UID: \"ead2084a-70a7-47d2-a757-f340d971bfb6\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-rw9sz" Oct 31 04:29:55.612926 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:55.612756 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ead2084a-70a7-47d2-a757-f340d971bfb6-istiod-ca-cert\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-rw9sz\" (UID: \"ead2084a-70a7-47d2-a757-f340d971bfb6\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-rw9sz" Oct 31 04:29:55.612926 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:55.612789 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ead2084a-70a7-47d2-a757-f340d971bfb6-workload-socket\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-rw9sz\" (UID: \"ead2084a-70a7-47d2-a757-f340d971bfb6\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-rw9sz" Oct 31 04:29:55.612926 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:55.612849 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ead2084a-70a7-47d2-a757-f340d971bfb6-istio-podinfo\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-rw9sz\" (UID: \"ead2084a-70a7-47d2-a757-f340d971bfb6\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-rw9sz" Oct 31 04:29:55.613068 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:55.612985 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ead2084a-70a7-47d2-a757-f340d971bfb6-workload-certs\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-rw9sz\" (UID: \"ead2084a-70a7-47d2-a757-f340d971bfb6\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-rw9sz" Oct 31 04:29:55.613068 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:55.613021 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ead2084a-70a7-47d2-a757-f340d971bfb6-workload-socket\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-rw9sz\" (UID: \"ead2084a-70a7-47d2-a757-f340d971bfb6\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-rw9sz" Oct 31 04:29:55.614369 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:55.614322 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ead2084a-70a7-47d2-a757-f340d971bfb6-istio-envoy\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-rw9sz\" (UID: \"ead2084a-70a7-47d2-a757-f340d971bfb6\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-rw9sz" Oct 31 04:29:55.615102 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:55.615087 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ead2084a-70a7-47d2-a757-f340d971bfb6-istio-podinfo\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-rw9sz\" (UID: \"ead2084a-70a7-47d2-a757-f340d971bfb6\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-rw9sz" Oct 31 04:29:55.623257 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:55.623229 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-wq9cv" Oct 31 04:29:55.635699 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:55.635670 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ead2084a-70a7-47d2-a757-f340d971bfb6-istio-token\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-rw9sz\" (UID: \"ead2084a-70a7-47d2-a757-f340d971bfb6\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-rw9sz" Oct 31 04:29:55.635845 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:55.635821 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzhw7\" (UniqueName: \"kubernetes.io/projected/ead2084a-70a7-47d2-a757-f340d971bfb6-kube-api-access-mzhw7\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-rw9sz\" (UID: \"ead2084a-70a7-47d2-a757-f340d971bfb6\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-rw9sz" Oct 31 04:29:55.652793 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:55.652764 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-rw9sz" Oct 31 04:29:55.942489 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:29:55.942461 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17ea2397_450e_4505_9a48_0cf93f76065e.slice/crio-eda7de36a1d9c178c94540398525c070e1ebea1992f5b73d330f3d423e1560af WatchSource:0}: Error finding container eda7de36a1d9c178c94540398525c070e1ebea1992f5b73d330f3d423e1560af: Status 404 returned error can't find the container with id eda7de36a1d9c178c94540398525c070e1ebea1992f5b73d330f3d423e1560af Oct 31 04:29:55.944048 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:55.944020 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-wq9cv"] Oct 31 04:29:55.945754 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:55.945712 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:29:55.945869 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:55.945802 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:29:55.945869 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:55.945850 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:29:55.978224 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:55.978198 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-rw9sz"] Oct 31 04:29:55.978773 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:29:55.978746 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podead2084a_70a7_47d2_a757_f340d971bfb6.slice/crio-d2f1ab09e3c9cf8f4282e5d066ba3e7eb6ae7c2c53d1e1d1bb69806df29f82d2 WatchSource:0}: Error finding container d2f1ab09e3c9cf8f4282e5d066ba3e7eb6ae7c2c53d1e1d1bb69806df29f82d2: Status 404 returned error can't find the container with id d2f1ab09e3c9cf8f4282e5d066ba3e7eb6ae7c2c53d1e1d1bb69806df29f82d2 Oct 31 04:29:55.981926 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:55.981885 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:29:55.981999 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:55.981983 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:29:55.982050 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:55.982024 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:29:56.524402 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:56.524363 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-rw9sz" event={"ID":"ead2084a-70a7-47d2-a757-f340d971bfb6","Type":"ContainerStarted","Data":"c4227668546e211ff6d5f7c2783bec5fc415a777c6fe37cf575f22e947f32ba2"} Oct 31 04:29:56.524862 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:56.524429 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-rw9sz" event={"ID":"ead2084a-70a7-47d2-a757-f340d971bfb6","Type":"ContainerStarted","Data":"d2f1ab09e3c9cf8f4282e5d066ba3e7eb6ae7c2c53d1e1d1bb69806df29f82d2"} Oct 31 04:29:56.530601 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:56.530568 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-wq9cv" event={"ID":"17ea2397-450e-4505-9a48-0cf93f76065e","Type":"ContainerStarted","Data":"ff9953f770bf9c4b20cfb7ca85ab92c25a709850cf99cd7c3ab370b9718fd576"} Oct 31 04:29:56.530758 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:56.530627 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-wq9cv" event={"ID":"17ea2397-450e-4505-9a48-0cf93f76065e","Type":"ContainerStarted","Data":"eda7de36a1d9c178c94540398525c070e1ebea1992f5b73d330f3d423e1560af"} Oct 31 04:29:56.623784 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:56.623750 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-wq9cv" Oct 31 04:29:56.668353 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:56.666114 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-rw9sz" podStartSLOduration=1.666090039 podStartE2EDuration="1.666090039s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:29:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:29:56.605755235 +0000 UTC m=+1544.320089875" watchObservedRunningTime="2026-10-31 04:29:56.666090039 +0000 UTC m=+1544.380424658" Oct 31 04:29:56.668353 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:56.667902 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-wq9cv" podStartSLOduration=1.667883408 podStartE2EDuration="1.667883408s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:29:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:29:56.663848815 +0000 UTC m=+1544.378183461" watchObservedRunningTime="2026-10-31 04:29:56.667883408 +0000 UTC m=+1544.382218029" Oct 31 04:29:56.678355 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:56.673110 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-rw9sz" Oct 31 04:29:57.625483 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:57.625443 4376 patch_prober.go:29] interesting pod/e2e-worker-w3-4c2387-openshift-default-5b68d44445-wq9cv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body= Oct 31 04:29:57.625872 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:57.625503 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-wq9cv" podUID="17ea2397-450e-4505-9a48-0cf93f76065e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" Oct 31 04:29:57.667352 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:57.666921 4376 patch_prober.go:29] interesting pod/e2e-worker-w3-4c2387-openshift-default-5b68d44445-rw9sz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body= Oct 31 04:29:57.667352 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:57.666987 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-rw9sz" podUID="ead2084a-70a7-47d2-a757-f340d971bfb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" Oct 31 04:29:58.499167 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:58.499133 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7cf9ef-59747bbd5-tbh6h" Oct 31 04:29:58.624361 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:58.624307 4376 patch_prober.go:29] interesting pod/e2e-worker-w3-4c2387-openshift-default-5b68d44445-wq9cv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body= Oct 31 04:29:58.624535 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:58.624391 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-wq9cv" podUID="17ea2397-450e-4505-9a48-0cf93f76065e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" Oct 31 04:29:58.690018 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:58.689983 4376 patch_prober.go:29] interesting pod/e2e-worker-w3-4c2387-openshift-default-5b68d44445-rw9sz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body= Oct 31 04:29:58.692322 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:58.692288 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-rw9sz" podUID="ead2084a-70a7-47d2-a757-f340d971bfb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" Oct 31 04:29:59.627060 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:59.627026 4376 patch_prober.go:29] interesting pod/e2e-worker-w3-4c2387-openshift-default-5b68d44445-wq9cv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body= Oct 31 04:29:59.627499 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:59.627471 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-wq9cv" podUID="17ea2397-450e-4505-9a48-0cf93f76065e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" Oct 31 04:29:59.656456 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:59.656418 4376 patch_prober.go:29] interesting pod/e2e-worker-w3-4c2387-openshift-default-5b68d44445-rw9sz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body= Oct 31 04:29:59.656733 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:29:59.656703 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-rw9sz" podUID="ead2084a-70a7-47d2-a757-f340d971bfb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" Oct 31 04:30:00.210400 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:00.195049 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-965f61-29890350-wlgk6"] Oct 31 04:30:00.262048 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:00.262012 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-965f61-29890350-wlgk6" Oct 31 04:30:00.326794 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:00.326730 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzb8r\"" Oct 31 04:30:00.327164 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:00.327128 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed72552d-53d8-43a8-8923-0098ab2a192b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-965f61-29890350-wlgk6\" (UID: \"ed72552d-53d8-43a8-8923-0098ab2a192b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-965f61-29890350-wlgk6" Oct 31 04:30:00.327464 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:00.327440 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2skt8\" (UniqueName: \"kubernetes.io/projected/ed72552d-53d8-43a8-8923-0098ab2a192b-kube-api-access-2skt8\") pod \"maas-api-key-cleanup-e2e-worker-w0-965f61-29890350-wlgk6\" (UID: \"ed72552d-53d8-43a8-8923-0098ab2a192b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-965f61-29890350-wlgk6" Oct 31 04:30:00.328632 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:00.328613 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-965f61-29890350-wlgk6"] Oct 31 04:30:00.406956 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:00.406918 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-6rs9r"] Oct 31 04:30:00.407195 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:00.407164 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-6rs9r" podUID="28a0b862-cb3b-4f82-990e-049381077d85" containerName="istio-proxy" containerID="cri-o://cca1910d69469194747b37b4b6fc5083ef9615228e12581e276f44c5a7d0363d" gracePeriod=30 Oct 31 04:30:00.430513 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:00.430477 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed72552d-53d8-43a8-8923-0098ab2a192b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-965f61-29890350-wlgk6\" (UID: \"ed72552d-53d8-43a8-8923-0098ab2a192b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-965f61-29890350-wlgk6" Oct 31 04:30:00.430662 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:00.430608 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2skt8\" (UniqueName: \"kubernetes.io/projected/ed72552d-53d8-43a8-8923-0098ab2a192b-kube-api-access-2skt8\") pod \"maas-api-key-cleanup-e2e-worker-w0-965f61-29890350-wlgk6\" (UID: \"ed72552d-53d8-43a8-8923-0098ab2a192b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-965f61-29890350-wlgk6" Oct 31 04:30:00.431667 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:00.431645 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed72552d-53d8-43a8-8923-0098ab2a192b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-965f61-29890350-wlgk6\" (UID: \"ed72552d-53d8-43a8-8923-0098ab2a192b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-965f61-29890350-wlgk6" Oct 31 04:30:00.531678 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:00.529933 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2skt8\" (UniqueName: \"kubernetes.io/projected/ed72552d-53d8-43a8-8923-0098ab2a192b-kube-api-access-2skt8\") pod \"maas-api-key-cleanup-e2e-worker-w0-965f61-29890350-wlgk6\" (UID: \"ed72552d-53d8-43a8-8923-0098ab2a192b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-965f61-29890350-wlgk6" Oct 31 04:30:00.624064 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:00.624026 4376 patch_prober.go:29] interesting pod/e2e-worker-w3-4c2387-openshift-default-5b68d44445-wq9cv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body= Oct 31 04:30:00.624229 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:00.624101 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-wq9cv" podUID="17ea2397-450e-4505-9a48-0cf93f76065e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" Oct 31 04:30:00.633701 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:00.632964 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-965f61-29890350-wlgk6" Oct 31 04:30:00.669244 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:00.669209 4376 patch_prober.go:29] interesting pod/e2e-worker-w3-4c2387-openshift-default-5b68d44445-rw9sz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body= Oct 31 04:30:00.677355 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:00.673399 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-rw9sz" podUID="ead2084a-70a7-47d2-a757-f340d971bfb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" Oct 31 04:30:01.086410 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:01.084696 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-965f61-29890350-wlgk6"] Oct 31 04:30:01.097207 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:30:01.097153 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded72552d_53d8_43a8_8923_0098ab2a192b.slice/crio-10622f90909171ef6cb3a93023c7e69930bc08086ce46ccc0361785c48342103 WatchSource:0}: Error finding container 10622f90909171ef6cb3a93023c7e69930bc08086ce46ccc0361785c48342103: Status 404 returned error can't find the container with id 10622f90909171ef6cb3a93023c7e69930bc08086ce46ccc0361785c48342103 Oct 31 04:30:01.624192 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:01.624155 4376 patch_prober.go:29] interesting pod/e2e-worker-w3-4c2387-openshift-default-5b68d44445-wq9cv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body= Oct 31 04:30:01.624601 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:01.624411 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-wq9cv" podUID="17ea2397-450e-4505-9a48-0cf93f76065e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" Oct 31 04:30:01.687588 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:01.687487 4376 patch_prober.go:29] interesting pod/e2e-worker-w3-4c2387-openshift-default-5b68d44445-rw9sz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body= Oct 31 04:30:01.687882 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:01.687852 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-rw9sz" podUID="ead2084a-70a7-47d2-a757-f340d971bfb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" Oct 31 04:30:01.691529 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:01.690739 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-965f61-29890350-wlgk6" event={"ID":"ed72552d-53d8-43a8-8923-0098ab2a192b","Type":"ContainerStarted","Data":"84d7beb0c43c053b9a9a90a0fa3b8c09943cb39769c3e1ebe2d52b3d29963f1e"} Oct 31 04:30:01.691781 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:01.691755 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-965f61-29890350-wlgk6" event={"ID":"ed72552d-53d8-43a8-8923-0098ab2a192b","Type":"ContainerStarted","Data":"10622f90909171ef6cb3a93023c7e69930bc08086ce46ccc0361785c48342103"} Oct 31 04:30:01.732937 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:01.732876 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-965f61-29890350-wlgk6" podStartSLOduration=1.732862431 podStartE2EDuration="1.732862431s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:30:01.732523965 +0000 UTC m=+1549.446858582" watchObservedRunningTime="2026-10-31 04:30:01.732862431 +0000 UTC m=+1549.447197047" Oct 31 04:30:02.634280 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:02.627728 4376 patch_prober.go:29] interesting pod/e2e-worker-w3-4c2387-openshift-default-5b68d44445-wq9cv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body= Oct 31 04:30:02.634280 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:02.627811 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-wq9cv" podUID="17ea2397-450e-4505-9a48-0cf93f76065e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" Oct 31 04:30:02.658352 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:02.657322 4376 patch_prober.go:29] interesting pod/e2e-worker-w3-4c2387-openshift-default-5b68d44445-rw9sz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body= Oct 31 04:30:02.658352 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:02.657401 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-rw9sz" podUID="ead2084a-70a7-47d2-a757-f340d971bfb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" Oct 31 04:30:02.699909 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:02.699878 4376 generic.go:356] "Generic (PLEG): container finished" podID="ed72552d-53d8-43a8-8923-0098ab2a192b" containerID="84d7beb0c43c053b9a9a90a0fa3b8c09943cb39769c3e1ebe2d52b3d29963f1e" exitCode=60 Oct 31 04:30:02.700091 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:02.699976 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-965f61-29890350-wlgk6" event={"ID":"ed72552d-53d8-43a8-8923-0098ab2a192b","Type":"ContainerDied","Data":"84d7beb0c43c053b9a9a90a0fa3b8c09943cb39769c3e1ebe2d52b3d29963f1e"} Oct 31 04:30:02.700287 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:02.700271 4376 scope.go:122] "RemoveContainer" containerID="84d7beb0c43c053b9a9a90a0fa3b8c09943cb39769c3e1ebe2d52b3d29963f1e" Oct 31 04:30:03.658463 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:03.658433 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-rw9sz" Oct 31 04:30:03.658960 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:03.658817 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-rw9sz" Oct 31 04:30:03.659525 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:03.659504 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-rw9sz" Oct 31 04:30:03.679789 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:03.679760 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-wq9cv" Oct 31 04:30:03.680212 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:03.680194 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-wq9cv" Oct 31 04:30:03.681134 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:03.681115 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-wq9cv" Oct 31 04:30:03.708779 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:03.708742 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-965f61-29890350-wlgk6" event={"ID":"ed72552d-53d8-43a8-8923-0098ab2a192b","Type":"ContainerStarted","Data":"25e1acaf2ccebb1ce570ceb78b70a0883c50b815d956550d89ac983700d4c2ef"} Oct 31 04:30:04.717580 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:04.717543 4376 generic.go:356] "Generic (PLEG): container finished" podID="ed72552d-53d8-43a8-8923-0098ab2a192b" containerID="25e1acaf2ccebb1ce570ceb78b70a0883c50b815d956550d89ac983700d4c2ef" exitCode=60 Oct 31 04:30:04.718072 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:04.717625 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-965f61-29890350-wlgk6" event={"ID":"ed72552d-53d8-43a8-8923-0098ab2a192b","Type":"ContainerDied","Data":"25e1acaf2ccebb1ce570ceb78b70a0883c50b815d956550d89ac983700d4c2ef"} Oct 31 04:30:04.718072 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:04.717731 4376 scope.go:122] "RemoveContainer" containerID="84d7beb0c43c053b9a9a90a0fa3b8c09943cb39769c3e1ebe2d52b3d29963f1e" Oct 31 04:30:04.718072 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:04.717944 4376 scope.go:122] "RemoveContainer" containerID="25e1acaf2ccebb1ce570ceb78b70a0883c50b815d956550d89ac983700d4c2ef" Oct 31 04:30:04.718361 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:30:04.718321 4376 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-965f61-29890350-wlgk6_odh-ai-gateway-infra(ed72552d-53d8-43a8-8923-0098ab2a192b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-965f61-29890350-wlgk6" podUID="ed72552d-53d8-43a8-8923-0098ab2a192b" Oct 31 04:30:05.726809 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:05.726787 4376 scope.go:122] "RemoveContainer" containerID="25e1acaf2ccebb1ce570ceb78b70a0883c50b815d956550d89ac983700d4c2ef" Oct 31 04:30:05.727255 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:30:05.727056 4376 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-965f61-29890350-wlgk6_odh-ai-gateway-infra(ed72552d-53d8-43a8-8923-0098ab2a192b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-965f61-29890350-wlgk6" podUID="ed72552d-53d8-43a8-8923-0098ab2a192b" Oct 31 04:30:05.728139 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:05.728114 4376 generic.go:356] "Generic (PLEG): container finished" podID="28a0b862-cb3b-4f82-990e-049381077d85" containerID="cca1910d69469194747b37b4b6fc5083ef9615228e12581e276f44c5a7d0363d" exitCode=0 Oct 31 04:30:05.728240 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:05.728190 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-6rs9r" event={"ID":"28a0b862-cb3b-4f82-990e-049381077d85","Type":"ContainerDied","Data":"cca1910d69469194747b37b4b6fc5083ef9615228e12581e276f44c5a7d0363d"} Oct 31 04:30:05.728303 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:05.728247 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-6rs9r" event={"ID":"28a0b862-cb3b-4f82-990e-049381077d85","Type":"ContainerDied","Data":"6d29eb77b394818b09e208f416d6b78193010755e7d4fcc5b3d3fad89733c585"} Oct 31 04:30:05.728303 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:05.728272 4376 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6d29eb77b394818b09e208f416d6b78193010755e7d4fcc5b3d3fad89733c585" Oct 31 04:30:05.745933 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:05.745899 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-6rs9r" Oct 31 04:30:05.826100 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:05.826058 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28a0b862-cb3b-4f82-990e-049381077d85-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28a0b862-cb3b-4f82-990e-049381077d85-workload-socket\") pod \"28a0b862-cb3b-4f82-990e-049381077d85\" (UID: \"28a0b862-cb3b-4f82-990e-049381077d85\") " Oct 31 04:30:05.826485 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:05.826402 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28a0b862-cb3b-4f82-990e-049381077d85-workload-socket" pod "28a0b862-cb3b-4f82-990e-049381077d85" (UID: "28a0b862-cb3b-4f82-990e-049381077d85"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:30:05.826485 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:05.826458 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28a0b862-cb3b-4f82-990e-049381077d85-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28a0b862-cb3b-4f82-990e-049381077d85-credential-socket\") pod \"28a0b862-cb3b-4f82-990e-049381077d85\" (UID: \"28a0b862-cb3b-4f82-990e-049381077d85\") " Oct 31 04:30:05.826797 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:05.826537 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28a0b862-cb3b-4f82-990e-049381077d85-kube-api-access-pxkjn\" (UniqueName: \"kubernetes.io/projected/28a0b862-cb3b-4f82-990e-049381077d85-kube-api-access-pxkjn\") pod \"28a0b862-cb3b-4f82-990e-049381077d85\" (UID: \"28a0b862-cb3b-4f82-990e-049381077d85\") " Oct 31 04:30:05.826797 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:05.826790 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28a0b862-cb3b-4f82-990e-049381077d85-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28a0b862-cb3b-4f82-990e-049381077d85-istio-data\") pod \"28a0b862-cb3b-4f82-990e-049381077d85\" (UID: \"28a0b862-cb3b-4f82-990e-049381077d85\") " Oct 31 04:30:05.828396 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:05.826909 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28a0b862-cb3b-4f82-990e-049381077d85-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28a0b862-cb3b-4f82-990e-049381077d85-workload-certs\") pod \"28a0b862-cb3b-4f82-990e-049381077d85\" (UID: \"28a0b862-cb3b-4f82-990e-049381077d85\") " Oct 31 04:30:05.828396 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:05.827029 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28a0b862-cb3b-4f82-990e-049381077d85-istio-token\" (UniqueName: \"kubernetes.io/projected/28a0b862-cb3b-4f82-990e-049381077d85-istio-token\") pod \"28a0b862-cb3b-4f82-990e-049381077d85\" (UID: \"28a0b862-cb3b-4f82-990e-049381077d85\") " Oct 31 04:30:05.828396 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:05.827108 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28a0b862-cb3b-4f82-990e-049381077d85-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28a0b862-cb3b-4f82-990e-049381077d85-istio-envoy\") pod \"28a0b862-cb3b-4f82-990e-049381077d85\" (UID: \"28a0b862-cb3b-4f82-990e-049381077d85\") " Oct 31 04:30:05.828396 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:05.827153 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/28a0b862-cb3b-4f82-990e-049381077d85-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28a0b862-cb3b-4f82-990e-049381077d85-istio-podinfo\") pod \"28a0b862-cb3b-4f82-990e-049381077d85\" (UID: \"28a0b862-cb3b-4f82-990e-049381077d85\") " Oct 31 04:30:05.828396 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:05.827196 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28a0b862-cb3b-4f82-990e-049381077d85-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28a0b862-cb3b-4f82-990e-049381077d85-istiod-ca-cert\") pod \"28a0b862-cb3b-4f82-990e-049381077d85\" (UID: \"28a0b862-cb3b-4f82-990e-049381077d85\") " Oct 31 04:30:05.828396 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:05.828283 4376 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28a0b862-cb3b-4f82-990e-049381077d85-workload-socket\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:30:05.829989 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:05.829963 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28a0b862-cb3b-4f82-990e-049381077d85-istiod-ca-cert" pod "28a0b862-cb3b-4f82-990e-049381077d85" (UID: "28a0b862-cb3b-4f82-990e-049381077d85"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:30:05.831277 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:05.831215 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28a0b862-cb3b-4f82-990e-049381077d85-istio-data" pod "28a0b862-cb3b-4f82-990e-049381077d85" (UID: "28a0b862-cb3b-4f82-990e-049381077d85"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:30:05.831277 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:05.831234 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28a0b862-cb3b-4f82-990e-049381077d85-credential-socket" pod "28a0b862-cb3b-4f82-990e-049381077d85" (UID: "28a0b862-cb3b-4f82-990e-049381077d85"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:30:05.831638 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:05.831604 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28a0b862-cb3b-4f82-990e-049381077d85-workload-certs" pod "28a0b862-cb3b-4f82-990e-049381077d85" (UID: "28a0b862-cb3b-4f82-990e-049381077d85"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:30:05.834738 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:05.834710 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28a0b862-cb3b-4f82-990e-049381077d85-istio-envoy" pod "28a0b862-cb3b-4f82-990e-049381077d85" (UID: "28a0b862-cb3b-4f82-990e-049381077d85"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:30:05.835785 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:05.835755 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28a0b862-cb3b-4f82-990e-049381077d85-istio-token" pod "28a0b862-cb3b-4f82-990e-049381077d85" (UID: "28a0b862-cb3b-4f82-990e-049381077d85"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:30:05.835994 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:05.835852 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28a0b862-cb3b-4f82-990e-049381077d85-kube-api-access-pxkjn" pod "28a0b862-cb3b-4f82-990e-049381077d85" (UID: "28a0b862-cb3b-4f82-990e-049381077d85"). InnerVolumeSpecName "kube-api-access-pxkjn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:30:05.835994 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:05.835927 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/28a0b862-cb3b-4f82-990e-049381077d85-istio-podinfo" pod "28a0b862-cb3b-4f82-990e-049381077d85" (UID: "28a0b862-cb3b-4f82-990e-049381077d85"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 04:30:05.929970 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:05.929936 4376 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28a0b862-cb3b-4f82-990e-049381077d85-workload-certs\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:30:05.929970 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:05.929965 4376 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28a0b862-cb3b-4f82-990e-049381077d85-istio-token\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:30:05.929970 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:05.929975 4376 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28a0b862-cb3b-4f82-990e-049381077d85-istio-envoy\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:30:05.930212 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:05.929983 4376 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28a0b862-cb3b-4f82-990e-049381077d85-istio-podinfo\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:30:05.930212 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:05.929991 4376 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28a0b862-cb3b-4f82-990e-049381077d85-istiod-ca-cert\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:30:05.930212 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:05.929999 4376 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28a0b862-cb3b-4f82-990e-049381077d85-credential-socket\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:30:05.930212 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:05.930007 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxkjn\" (UniqueName: \"kubernetes.io/projected/28a0b862-cb3b-4f82-990e-049381077d85-kube-api-access-pxkjn\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:30:05.930212 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:05.930017 4376 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28a0b862-cb3b-4f82-990e-049381077d85-istio-data\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:30:06.238860 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:06.238828 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-edc450-5c6c7f49c4-q2z97"] Oct 31 04:30:06.239046 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:06.239025 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-edc450-5c6c7f49c4-q2z97" podUID="2f3c1098-557f-4aae-90e3-b0e0869f5e30" containerName="maas-api" containerID="cri-o://d26c8f267da2a718fd61ece4cc85fd341f58fae7e29e868f5ef332ea6143422f" gracePeriod=30 Oct 31 04:30:06.385721 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:06.385679 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-edc450-5c6c7f49c4-q2z97" podUID="2f3c1098-557f-4aae-90e3-b0e0869f5e30" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.128:8443/health\": dial tcp 10.133.0.128:8443: connect: connection refused" Oct 31 04:30:06.395611 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:06.395579 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-96rgs"] Oct 31 04:30:06.395851 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:06.395808 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-96rgs" podUID="2b3b5806-b3d7-48a8-b2d5-fb202004605d" containerName="istio-proxy" containerID="cri-o://9c1b014c56cc8c247d4e85c151deb5101b818f50fee2a89082b85dea30d963ce" gracePeriod=30 Oct 31 04:30:06.465038 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:30:06.465002 4376 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28a0b862_cb3b_4f82_990e_049381077d85.slice/crio-6d29eb77b394818b09e208f416d6b78193010755e7d4fcc5b3d3fad89733c585\": RecentStats: unable to find data in memory cache]" Oct 31 04:30:06.727394 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:06.727355 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-edc450-5c6c7f49c4-q2z97" Oct 31 04:30:06.803351 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:06.802780 4376 generic.go:356] "Generic (PLEG): container finished" podID="2f3c1098-557f-4aae-90e3-b0e0869f5e30" containerID="d26c8f267da2a718fd61ece4cc85fd341f58fae7e29e868f5ef332ea6143422f" exitCode=0 Oct 31 04:30:06.803351 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:06.803147 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-6rs9r" Oct 31 04:30:06.810353 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:06.803984 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-edc450-5c6c7f49c4-q2z97" Oct 31 04:30:06.810353 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:06.804114 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-edc450-5c6c7f49c4-q2z97" event={"ID":"2f3c1098-557f-4aae-90e3-b0e0869f5e30","Type":"ContainerDied","Data":"d26c8f267da2a718fd61ece4cc85fd341f58fae7e29e868f5ef332ea6143422f"} Oct 31 04:30:06.810353 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:06.804161 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-edc450-5c6c7f49c4-q2z97" event={"ID":"2f3c1098-557f-4aae-90e3-b0e0869f5e30","Type":"ContainerDied","Data":"b4abd19f022408560be143564cf66abf3fd5c8fc1e4f57fc09fcd7f177723f13"} Oct 31 04:30:06.810353 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:06.804191 4376 scope.go:122] "RemoveContainer" containerID="d26c8f267da2a718fd61ece4cc85fd341f58fae7e29e868f5ef332ea6143422f" Oct 31 04:30:06.910479 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:06.908870 4376 scope.go:122] "RemoveContainer" containerID="d26c8f267da2a718fd61ece4cc85fd341f58fae7e29e868f5ef332ea6143422f" Oct 31 04:30:06.918200 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:30:06.914852 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d26c8f267da2a718fd61ece4cc85fd341f58fae7e29e868f5ef332ea6143422f\": container with ID starting with d26c8f267da2a718fd61ece4cc85fd341f58fae7e29e868f5ef332ea6143422f not found: ID does not exist" containerID="d26c8f267da2a718fd61ece4cc85fd341f58fae7e29e868f5ef332ea6143422f" Oct 31 04:30:06.918200 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:06.914902 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d26c8f267da2a718fd61ece4cc85fd341f58fae7e29e868f5ef332ea6143422f"} err="failed to get container status \"d26c8f267da2a718fd61ece4cc85fd341f58fae7e29e868f5ef332ea6143422f\": rpc error: code = NotFound desc = could not find container \"d26c8f267da2a718fd61ece4cc85fd341f58fae7e29e868f5ef332ea6143422f\": container with ID starting with d26c8f267da2a718fd61ece4cc85fd341f58fae7e29e868f5ef332ea6143422f not found: ID does not exist" Oct 31 04:30:06.924312 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:06.924027 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-zlcr5"] Oct 31 04:30:06.924312 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:06.924270 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-zlcr5" podUID="2697f74a-c2a2-4d6f-ba09-adb5825a8d85" containerName="istio-proxy" containerID="cri-o://47d9a120ee2b82fc6f02a7ea9c24f974f093b8068328e505c7ecee1dd113dc9d" gracePeriod=30 Oct 31 04:30:06.932916 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:06.932838 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f3c1098-557f-4aae-90e3-b0e0869f5e30-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f3c1098-557f-4aae-90e3-b0e0869f5e30-maas-api-tls\") pod \"2f3c1098-557f-4aae-90e3-b0e0869f5e30\" (UID: \"2f3c1098-557f-4aae-90e3-b0e0869f5e30\") " Oct 31 04:30:06.932916 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:06.932881 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f3c1098-557f-4aae-90e3-b0e0869f5e30-kube-api-access-22fnc\" (UniqueName: \"kubernetes.io/projected/2f3c1098-557f-4aae-90e3-b0e0869f5e30-kube-api-access-22fnc\") pod \"2f3c1098-557f-4aae-90e3-b0e0869f5e30\" (UID: \"2f3c1098-557f-4aae-90e3-b0e0869f5e30\") " Oct 31 04:30:06.933135 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:06.932961 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f3c1098-557f-4aae-90e3-b0e0869f5e30-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f3c1098-557f-4aae-90e3-b0e0869f5e30-metrics-certs\") pod \"2f3c1098-557f-4aae-90e3-b0e0869f5e30\" (UID: \"2f3c1098-557f-4aae-90e3-b0e0869f5e30\") " Oct 31 04:30:06.933135 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:06.933090 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f3c1098-557f-4aae-90e3-b0e0869f5e30-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f3c1098-557f-4aae-90e3-b0e0869f5e30-openshift-service-ca\") pod \"2f3c1098-557f-4aae-90e3-b0e0869f5e30\" (UID: \"2f3c1098-557f-4aae-90e3-b0e0869f5e30\") " Oct 31 04:30:06.934131 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:06.933892 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f3c1098-557f-4aae-90e3-b0e0869f5e30-openshift-service-ca" pod "2f3c1098-557f-4aae-90e3-b0e0869f5e30" (UID: "2f3c1098-557f-4aae-90e3-b0e0869f5e30"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:30:06.943694 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:06.943442 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-6rs9r"] Oct 31 04:30:06.944306 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:06.944271 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f3c1098-557f-4aae-90e3-b0e0869f5e30-kube-api-access-22fnc" pod "2f3c1098-557f-4aae-90e3-b0e0869f5e30" (UID: "2f3c1098-557f-4aae-90e3-b0e0869f5e30"). InnerVolumeSpecName "kube-api-access-22fnc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:30:06.945641 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:06.945610 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f3c1098-557f-4aae-90e3-b0e0869f5e30-maas-api-tls" pod "2f3c1098-557f-4aae-90e3-b0e0869f5e30" (UID: "2f3c1098-557f-4aae-90e3-b0e0869f5e30"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:30:06.946310 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:06.946283 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f3c1098-557f-4aae-90e3-b0e0869f5e30-metrics-certs" pod "2f3c1098-557f-4aae-90e3-b0e0869f5e30" (UID: "2f3c1098-557f-4aae-90e3-b0e0869f5e30"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:30:06.946648 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:06.946609 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-6rs9r"] Oct 31 04:30:07.034598 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:07.034553 4376 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f3c1098-557f-4aae-90e3-b0e0869f5e30-metrics-certs\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:30:07.034598 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:07.034591 4376 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f3c1098-557f-4aae-90e3-b0e0869f5e30-openshift-service-ca\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:30:07.034598 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:07.034605 4376 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f3c1098-557f-4aae-90e3-b0e0869f5e30-maas-api-tls\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:30:07.034840 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:07.034618 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22fnc\" (UniqueName: \"kubernetes.io/projected/2f3c1098-557f-4aae-90e3-b0e0869f5e30-kube-api-access-22fnc\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:30:07.153783 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:07.153754 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-edc450-5c6c7f49c4-q2z97"] Oct 31 04:30:07.156827 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:07.156792 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-edc450-5c6c7f49c4-q2z97"] Oct 31 04:30:08.444340 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:08.444299 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28a0b862-cb3b-4f82-990e-049381077d85" path="/var/lib/kubelet/pods/28a0b862-cb3b-4f82-990e-049381077d85/volumes" Oct 31 04:30:08.444814 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:08.444791 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f3c1098-557f-4aae-90e3-b0e0869f5e30" path="/var/lib/kubelet/pods/2f3c1098-557f-4aae-90e3-b0e0869f5e30/volumes" Oct 31 04:30:09.571775 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:09.571741 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 04:30:10.035229 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:10.035197 4376 patch_prober.go:29] interesting pod/e2e-worker-w1-439989-openshift-default-8448c6ff99-96rgs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 04:30:10.035416 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:10.035257 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-96rgs" podUID="2b3b5806-b3d7-48a8-b2d5-fb202004605d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 04:30:10.570741 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:10.570615 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-edc450-openshift-default-84c757994f-q4vwz"] Oct 31 04:30:10.603351 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:10.603139 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-edc450-openshift-default-84c757994f-q4vwz" podUID="c0cc41ad-7f5b-4a5c-a989-bfafee0857e8" containerName="istio-proxy" containerID="cri-o://20210477399c246dd0b73fbb8ffe62f810561347e339808647f65ad299caa95d" gracePeriod=30 Oct 31 04:30:11.015512 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:30:11.015462 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 04:30:12.136151 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:12.136121 4376 generic.go:356] "Generic (PLEG): container finished" podID="2b3b5806-b3d7-48a8-b2d5-fb202004605d" containerID="9c1b014c56cc8c247d4e85c151deb5101b818f50fee2a89082b85dea30d963ce" exitCode=0 Oct 31 04:30:12.136833 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:12.136796 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-96rgs" event={"ID":"2b3b5806-b3d7-48a8-b2d5-fb202004605d","Type":"ContainerDied","Data":"9c1b014c56cc8c247d4e85c151deb5101b818f50fee2a89082b85dea30d963ce"} Oct 31 04:30:12.136973 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:12.136950 4376 scope.go:122] "RemoveContainer" containerID="903f8ec8804650023256d4b08dbb9e0cf0299163173d3e83a7ae97003a6ac292" Oct 31 04:30:12.233831 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:12.233805 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-96rgs" Oct 31 04:30:12.361985 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:12.360787 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b3b5806-b3d7-48a8-b2d5-fb202004605d-istio-token\" (UniqueName: \"kubernetes.io/projected/2b3b5806-b3d7-48a8-b2d5-fb202004605d-istio-token\") pod \"2b3b5806-b3d7-48a8-b2d5-fb202004605d\" (UID: \"2b3b5806-b3d7-48a8-b2d5-fb202004605d\") " Oct 31 04:30:12.361985 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:12.360906 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b3b5806-b3d7-48a8-b2d5-fb202004605d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b3b5806-b3d7-48a8-b2d5-fb202004605d-credential-socket\") pod \"2b3b5806-b3d7-48a8-b2d5-fb202004605d\" (UID: \"2b3b5806-b3d7-48a8-b2d5-fb202004605d\") " Oct 31 04:30:12.361985 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:12.361032 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2b3b5806-b3d7-48a8-b2d5-fb202004605d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b3b5806-b3d7-48a8-b2d5-fb202004605d-istio-podinfo\") pod \"2b3b5806-b3d7-48a8-b2d5-fb202004605d\" (UID: \"2b3b5806-b3d7-48a8-b2d5-fb202004605d\") " Oct 31 04:30:12.361985 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:12.361084 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b3b5806-b3d7-48a8-b2d5-fb202004605d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b3b5806-b3d7-48a8-b2d5-fb202004605d-workload-socket\") pod \"2b3b5806-b3d7-48a8-b2d5-fb202004605d\" (UID: \"2b3b5806-b3d7-48a8-b2d5-fb202004605d\") " Oct 31 04:30:12.361985 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:12.361236 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b3b5806-b3d7-48a8-b2d5-fb202004605d-kube-api-access-tbjt2\" (UniqueName: \"kubernetes.io/projected/2b3b5806-b3d7-48a8-b2d5-fb202004605d-kube-api-access-tbjt2\") pod \"2b3b5806-b3d7-48a8-b2d5-fb202004605d\" (UID: \"2b3b5806-b3d7-48a8-b2d5-fb202004605d\") " Oct 31 04:30:12.361985 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:12.361277 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b3b5806-b3d7-48a8-b2d5-fb202004605d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b3b5806-b3d7-48a8-b2d5-fb202004605d-istio-envoy\") pod \"2b3b5806-b3d7-48a8-b2d5-fb202004605d\" (UID: \"2b3b5806-b3d7-48a8-b2d5-fb202004605d\") " Oct 31 04:30:12.361985 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:12.361306 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b3b5806-b3d7-48a8-b2d5-fb202004605d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b3b5806-b3d7-48a8-b2d5-fb202004605d-istiod-ca-cert\") pod \"2b3b5806-b3d7-48a8-b2d5-fb202004605d\" (UID: \"2b3b5806-b3d7-48a8-b2d5-fb202004605d\") " Oct 31 04:30:12.361985 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:12.361379 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b3b5806-b3d7-48a8-b2d5-fb202004605d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b3b5806-b3d7-48a8-b2d5-fb202004605d-workload-certs\") pod \"2b3b5806-b3d7-48a8-b2d5-fb202004605d\" (UID: \"2b3b5806-b3d7-48a8-b2d5-fb202004605d\") " Oct 31 04:30:12.361985 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:12.361433 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b3b5806-b3d7-48a8-b2d5-fb202004605d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b3b5806-b3d7-48a8-b2d5-fb202004605d-istio-data\") pod \"2b3b5806-b3d7-48a8-b2d5-fb202004605d\" (UID: \"2b3b5806-b3d7-48a8-b2d5-fb202004605d\") " Oct 31 04:30:12.364043 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:12.363628 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b3b5806-b3d7-48a8-b2d5-fb202004605d-istio-data" pod "2b3b5806-b3d7-48a8-b2d5-fb202004605d" (UID: "2b3b5806-b3d7-48a8-b2d5-fb202004605d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:30:12.372377 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:12.370484 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b3b5806-b3d7-48a8-b2d5-fb202004605d-workload-certs" pod "2b3b5806-b3d7-48a8-b2d5-fb202004605d" (UID: "2b3b5806-b3d7-48a8-b2d5-fb202004605d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:30:12.372377 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:12.370793 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b3b5806-b3d7-48a8-b2d5-fb202004605d-istiod-ca-cert" pod "2b3b5806-b3d7-48a8-b2d5-fb202004605d" (UID: "2b3b5806-b3d7-48a8-b2d5-fb202004605d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:30:12.372377 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:12.371062 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b3b5806-b3d7-48a8-b2d5-fb202004605d-credential-socket" pod "2b3b5806-b3d7-48a8-b2d5-fb202004605d" (UID: "2b3b5806-b3d7-48a8-b2d5-fb202004605d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:30:12.372377 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:12.371348 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b3b5806-b3d7-48a8-b2d5-fb202004605d-workload-socket" pod "2b3b5806-b3d7-48a8-b2d5-fb202004605d" (UID: "2b3b5806-b3d7-48a8-b2d5-fb202004605d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:30:12.377218 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:12.377178 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2b3b5806-b3d7-48a8-b2d5-fb202004605d-istio-podinfo" pod "2b3b5806-b3d7-48a8-b2d5-fb202004605d" (UID: "2b3b5806-b3d7-48a8-b2d5-fb202004605d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 04:30:12.382846 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:12.382050 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b3b5806-b3d7-48a8-b2d5-fb202004605d-istio-envoy" pod "2b3b5806-b3d7-48a8-b2d5-fb202004605d" (UID: "2b3b5806-b3d7-48a8-b2d5-fb202004605d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:30:12.382846 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:12.382583 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b3b5806-b3d7-48a8-b2d5-fb202004605d-kube-api-access-tbjt2" pod "2b3b5806-b3d7-48a8-b2d5-fb202004605d" (UID: "2b3b5806-b3d7-48a8-b2d5-fb202004605d"). InnerVolumeSpecName "kube-api-access-tbjt2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:30:12.385633 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:12.385606 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b3b5806-b3d7-48a8-b2d5-fb202004605d-istio-token" pod "2b3b5806-b3d7-48a8-b2d5-fb202004605d" (UID: "2b3b5806-b3d7-48a8-b2d5-fb202004605d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:30:12.464188 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:12.464159 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbjt2\" (UniqueName: \"kubernetes.io/projected/2b3b5806-b3d7-48a8-b2d5-fb202004605d-kube-api-access-tbjt2\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:30:12.464188 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:12.464190 4376 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b3b5806-b3d7-48a8-b2d5-fb202004605d-istio-envoy\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:30:12.464400 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:12.464199 4376 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b3b5806-b3d7-48a8-b2d5-fb202004605d-istiod-ca-cert\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:30:12.464400 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:12.464208 4376 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b3b5806-b3d7-48a8-b2d5-fb202004605d-workload-certs\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:30:12.464400 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:12.464215 4376 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b3b5806-b3d7-48a8-b2d5-fb202004605d-istio-data\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:30:12.464400 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:12.464225 4376 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b3b5806-b3d7-48a8-b2d5-fb202004605d-istio-token\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:30:12.464400 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:12.464236 4376 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b3b5806-b3d7-48a8-b2d5-fb202004605d-credential-socket\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:30:12.464400 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:12.464247 4376 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b3b5806-b3d7-48a8-b2d5-fb202004605d-istio-podinfo\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:30:12.464400 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:12.464256 4376 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b3b5806-b3d7-48a8-b2d5-fb202004605d-workload-socket\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:30:12.495830 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:12.495803 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-zlcr5" Oct 31 04:30:12.596279 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:12.596241 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2697f74a-c2a2-4d6f-ba09-adb5825a8d85-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2697f74a-c2a2-4d6f-ba09-adb5825a8d85-credential-socket\") pod \"2697f74a-c2a2-4d6f-ba09-adb5825a8d85\" (UID: \"2697f74a-c2a2-4d6f-ba09-adb5825a8d85\") " Oct 31 04:30:12.596491 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:12.596348 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2697f74a-c2a2-4d6f-ba09-adb5825a8d85-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2697f74a-c2a2-4d6f-ba09-adb5825a8d85-istiod-ca-cert\") pod \"2697f74a-c2a2-4d6f-ba09-adb5825a8d85\" (UID: \"2697f74a-c2a2-4d6f-ba09-adb5825a8d85\") " Oct 31 04:30:12.596491 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:12.596398 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2697f74a-c2a2-4d6f-ba09-adb5825a8d85-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2697f74a-c2a2-4d6f-ba09-adb5825a8d85-workload-socket\") pod \"2697f74a-c2a2-4d6f-ba09-adb5825a8d85\" (UID: \"2697f74a-c2a2-4d6f-ba09-adb5825a8d85\") " Oct 31 04:30:12.596641 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:12.596515 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2697f74a-c2a2-4d6f-ba09-adb5825a8d85-kube-api-access-vc9bv\" (UniqueName: \"kubernetes.io/projected/2697f74a-c2a2-4d6f-ba09-adb5825a8d85-kube-api-access-vc9bv\") pod \"2697f74a-c2a2-4d6f-ba09-adb5825a8d85\" (UID: \"2697f74a-c2a2-4d6f-ba09-adb5825a8d85\") " Oct 31 04:30:12.596641 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:12.596616 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2697f74a-c2a2-4d6f-ba09-adb5825a8d85-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2697f74a-c2a2-4d6f-ba09-adb5825a8d85-istio-podinfo\") pod \"2697f74a-c2a2-4d6f-ba09-adb5825a8d85\" (UID: \"2697f74a-c2a2-4d6f-ba09-adb5825a8d85\") " Oct 31 04:30:12.596755 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:12.596645 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2697f74a-c2a2-4d6f-ba09-adb5825a8d85-credential-socket" pod "2697f74a-c2a2-4d6f-ba09-adb5825a8d85" (UID: "2697f74a-c2a2-4d6f-ba09-adb5825a8d85"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:30:12.596755 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:12.596656 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2697f74a-c2a2-4d6f-ba09-adb5825a8d85-istio-token\" (UniqueName: \"kubernetes.io/projected/2697f74a-c2a2-4d6f-ba09-adb5825a8d85-istio-token\") pod \"2697f74a-c2a2-4d6f-ba09-adb5825a8d85\" (UID: \"2697f74a-c2a2-4d6f-ba09-adb5825a8d85\") " Oct 31 04:30:12.596858 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:12.596779 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2697f74a-c2a2-4d6f-ba09-adb5825a8d85-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2697f74a-c2a2-4d6f-ba09-adb5825a8d85-istio-envoy\") pod \"2697f74a-c2a2-4d6f-ba09-adb5825a8d85\" (UID: \"2697f74a-c2a2-4d6f-ba09-adb5825a8d85\") " Oct 31 04:30:12.596858 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:12.596839 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2697f74a-c2a2-4d6f-ba09-adb5825a8d85-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2697f74a-c2a2-4d6f-ba09-adb5825a8d85-workload-certs\") pod \"2697f74a-c2a2-4d6f-ba09-adb5825a8d85\" (UID: \"2697f74a-c2a2-4d6f-ba09-adb5825a8d85\") " Oct 31 04:30:12.596950 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:12.596867 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2697f74a-c2a2-4d6f-ba09-adb5825a8d85-workload-socket" pod "2697f74a-c2a2-4d6f-ba09-adb5825a8d85" (UID: "2697f74a-c2a2-4d6f-ba09-adb5825a8d85"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:30:12.596950 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:12.596901 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2697f74a-c2a2-4d6f-ba09-adb5825a8d85-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2697f74a-c2a2-4d6f-ba09-adb5825a8d85-istio-data\") pod \"2697f74a-c2a2-4d6f-ba09-adb5825a8d85\" (UID: \"2697f74a-c2a2-4d6f-ba09-adb5825a8d85\") " Oct 31 04:30:12.597155 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:12.597133 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2697f74a-c2a2-4d6f-ba09-adb5825a8d85-istiod-ca-cert" pod "2697f74a-c2a2-4d6f-ba09-adb5825a8d85" (UID: "2697f74a-c2a2-4d6f-ba09-adb5825a8d85"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:30:12.597877 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:12.597723 4376 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2697f74a-c2a2-4d6f-ba09-adb5825a8d85-credential-socket\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:30:12.597877 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:12.597751 4376 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2697f74a-c2a2-4d6f-ba09-adb5825a8d85-istiod-ca-cert\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:30:12.597877 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:12.597765 4376 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2697f74a-c2a2-4d6f-ba09-adb5825a8d85-workload-socket\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:30:12.600470 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:12.600445 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2697f74a-c2a2-4d6f-ba09-adb5825a8d85-workload-certs" pod "2697f74a-c2a2-4d6f-ba09-adb5825a8d85" (UID: "2697f74a-c2a2-4d6f-ba09-adb5825a8d85"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:30:12.600762 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:12.600745 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2697f74a-c2a2-4d6f-ba09-adb5825a8d85-istio-data" pod "2697f74a-c2a2-4d6f-ba09-adb5825a8d85" (UID: "2697f74a-c2a2-4d6f-ba09-adb5825a8d85"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:30:12.601141 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:12.601110 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2697f74a-c2a2-4d6f-ba09-adb5825a8d85-istio-envoy" pod "2697f74a-c2a2-4d6f-ba09-adb5825a8d85" (UID: "2697f74a-c2a2-4d6f-ba09-adb5825a8d85"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:30:12.610454 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:12.607651 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2697f74a-c2a2-4d6f-ba09-adb5825a8d85-istio-token" pod "2697f74a-c2a2-4d6f-ba09-adb5825a8d85" (UID: "2697f74a-c2a2-4d6f-ba09-adb5825a8d85"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:30:12.611759 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:12.611711 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2697f74a-c2a2-4d6f-ba09-adb5825a8d85-kube-api-access-vc9bv" pod "2697f74a-c2a2-4d6f-ba09-adb5825a8d85" (UID: "2697f74a-c2a2-4d6f-ba09-adb5825a8d85"). InnerVolumeSpecName "kube-api-access-vc9bv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:30:12.612099 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:12.612022 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2697f74a-c2a2-4d6f-ba09-adb5825a8d85-istio-podinfo" pod "2697f74a-c2a2-4d6f-ba09-adb5825a8d85" (UID: "2697f74a-c2a2-4d6f-ba09-adb5825a8d85"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 04:30:12.698911 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:12.698885 4376 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2697f74a-c2a2-4d6f-ba09-adb5825a8d85-workload-certs\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:30:12.698911 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:12.698909 4376 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2697f74a-c2a2-4d6f-ba09-adb5825a8d85-istio-data\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:30:12.698911 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:12.698918 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vc9bv\" (UniqueName: \"kubernetes.io/projected/2697f74a-c2a2-4d6f-ba09-adb5825a8d85-kube-api-access-vc9bv\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:30:12.699132 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:12.698927 4376 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2697f74a-c2a2-4d6f-ba09-adb5825a8d85-istio-podinfo\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:30:12.699132 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:12.698937 4376 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2697f74a-c2a2-4d6f-ba09-adb5825a8d85-istio-token\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:30:12.699132 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:12.698948 4376 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2697f74a-c2a2-4d6f-ba09-adb5825a8d85-istio-envoy\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:30:13.145577 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:13.145539 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-96rgs" event={"ID":"2b3b5806-b3d7-48a8-b2d5-fb202004605d","Type":"ContainerDied","Data":"8e2d0d016fc10ffc1a10ebeb873fdd6fa4f59e1a8aa687e691505622300ff885"} Oct 31 04:30:13.146167 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:13.145609 4376 scope.go:122] "RemoveContainer" containerID="9c1b014c56cc8c247d4e85c151deb5101b818f50fee2a89082b85dea30d963ce" Oct 31 04:30:13.146167 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:13.145671 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-96rgs" Oct 31 04:30:13.147702 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:13.147668 4376 generic.go:356] "Generic (PLEG): container finished" podID="2697f74a-c2a2-4d6f-ba09-adb5825a8d85" containerID="47d9a120ee2b82fc6f02a7ea9c24f974f093b8068328e505c7ecee1dd113dc9d" exitCode=0 Oct 31 04:30:13.147842 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:13.147738 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-zlcr5" Oct 31 04:30:13.147842 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:13.147771 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-zlcr5" event={"ID":"2697f74a-c2a2-4d6f-ba09-adb5825a8d85","Type":"ContainerDied","Data":"47d9a120ee2b82fc6f02a7ea9c24f974f093b8068328e505c7ecee1dd113dc9d"} Oct 31 04:30:13.147842 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:13.147812 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-zlcr5" event={"ID":"2697f74a-c2a2-4d6f-ba09-adb5825a8d85","Type":"ContainerDied","Data":"5fd870b7b77a36c598a327aa050afe4caf5f5ac6a8fbad0928fd405e997c3a42"} Oct 31 04:30:13.157192 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:13.157079 4376 scope.go:122] "RemoveContainer" containerID="47d9a120ee2b82fc6f02a7ea9c24f974f093b8068328e505c7ecee1dd113dc9d" Oct 31 04:30:13.167826 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:13.167797 4376 scope.go:122] "RemoveContainer" containerID="47d9a120ee2b82fc6f02a7ea9c24f974f093b8068328e505c7ecee1dd113dc9d" Oct 31 04:30:13.168109 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:30:13.168088 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47d9a120ee2b82fc6f02a7ea9c24f974f093b8068328e505c7ecee1dd113dc9d\": container with ID starting with 47d9a120ee2b82fc6f02a7ea9c24f974f093b8068328e505c7ecee1dd113dc9d not found: ID does not exist" containerID="47d9a120ee2b82fc6f02a7ea9c24f974f093b8068328e505c7ecee1dd113dc9d" Oct 31 04:30:13.168173 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:13.168117 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47d9a120ee2b82fc6f02a7ea9c24f974f093b8068328e505c7ecee1dd113dc9d"} err="failed to get container status \"47d9a120ee2b82fc6f02a7ea9c24f974f093b8068328e505c7ecee1dd113dc9d\": rpc error: code = NotFound desc = could not find container \"47d9a120ee2b82fc6f02a7ea9c24f974f093b8068328e505c7ecee1dd113dc9d\": container with ID starting with 47d9a120ee2b82fc6f02a7ea9c24f974f093b8068328e505c7ecee1dd113dc9d not found: ID does not exist" Oct 31 04:30:13.232499 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:13.232471 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-96rgs"] Oct 31 04:30:13.239376 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:13.239324 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-96rgs"] Oct 31 04:30:13.283074 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:13.283044 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-zlcr5"] Oct 31 04:30:13.288145 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:13.288118 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-zlcr5"] Oct 31 04:30:14.435919 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:14.435891 4376 scope.go:122] "RemoveContainer" containerID="25e1acaf2ccebb1ce570ceb78b70a0883c50b815d956550d89ac983700d4c2ef" Oct 31 04:30:14.454169 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:14.454143 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2697f74a-c2a2-4d6f-ba09-adb5825a8d85" path="/var/lib/kubelet/pods/2697f74a-c2a2-4d6f-ba09-adb5825a8d85/volumes" Oct 31 04:30:14.454635 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:14.454622 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b3b5806-b3d7-48a8-b2d5-fb202004605d" path="/var/lib/kubelet/pods/2b3b5806-b3d7-48a8-b2d5-fb202004605d/volumes" Oct 31 04:30:15.201756 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:15.195841 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41acf6c0-gxhc6"] Oct 31 04:30:15.201756 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:15.196698 4376 cpu_manager.go:414] "containerMap: removing container" podUID="2697f74a-c2a2-4d6f-ba09-adb5825a8d85" containerName="istio-proxy" Oct 31 04:30:15.201756 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:15.196717 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2697f74a-c2a2-4d6f-ba09-adb5825a8d85" containerName="istio-proxy" Oct 31 04:30:15.201756 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:15.196745 4376 cpu_manager.go:414] "containerMap: removing container" podUID="2f3c1098-557f-4aae-90e3-b0e0869f5e30" containerName="maas-api" Oct 31 04:30:15.201756 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:15.196752 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f3c1098-557f-4aae-90e3-b0e0869f5e30" containerName="maas-api" Oct 31 04:30:15.201756 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:15.196791 4376 cpu_manager.go:414] "containerMap: removing container" podUID="2b3b5806-b3d7-48a8-b2d5-fb202004605d" containerName="istio-proxy" Oct 31 04:30:15.201756 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:15.196799 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b3b5806-b3d7-48a8-b2d5-fb202004605d" containerName="istio-proxy" Oct 31 04:30:15.201756 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:15.196820 4376 cpu_manager.go:414] "containerMap: removing container" podUID="2b3b5806-b3d7-48a8-b2d5-fb202004605d" containerName="istio-proxy" Oct 31 04:30:15.201756 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:15.196827 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b3b5806-b3d7-48a8-b2d5-fb202004605d" containerName="istio-proxy" Oct 31 04:30:15.201756 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:15.196846 4376 cpu_manager.go:414] "containerMap: removing container" podUID="28a0b862-cb3b-4f82-990e-049381077d85" containerName="istio-proxy" Oct 31 04:30:15.201756 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:15.196852 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28a0b862-cb3b-4f82-990e-049381077d85" containerName="istio-proxy" Oct 31 04:30:15.201756 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:15.197000 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b3b5806-b3d7-48a8-b2d5-fb202004605d" containerName="istio-proxy" Oct 31 04:30:15.201756 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:15.197012 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b3b5806-b3d7-48a8-b2d5-fb202004605d" containerName="istio-proxy" Oct 31 04:30:15.201756 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:15.197025 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="28a0b862-cb3b-4f82-990e-049381077d85" containerName="istio-proxy" Oct 31 04:30:15.201756 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:15.197036 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="2697f74a-c2a2-4d6f-ba09-adb5825a8d85" containerName="istio-proxy" Oct 31 04:30:15.201756 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:15.197046 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f3c1098-557f-4aae-90e3-b0e0869f5e30" containerName="maas-api" Oct 31 04:30:15.201756 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:15.200887 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41acf6c0-gxhc6"] Oct 31 04:30:15.201756 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:15.200992 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41acf6c0-gxhc6" Oct 31 04:30:15.211580 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:15.211551 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-965f61-29890350-wlgk6" event={"ID":"ed72552d-53d8-43a8-8923-0098ab2a192b","Type":"ContainerStarted","Data":"5aa8654663c49dfafff84752fc9447c73549d4ce1651090d91980ef098253cf2"} Oct 31 04:30:15.218464 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:15.218439 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 04:30:15.297314 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:15.297290 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4vmx\" (UniqueName: \"kubernetes.io/projected/acc6c2f6-b64e-4940-b7f3-47a27f16a811-kube-api-access-m4vmx\") pod \"maas-api-revoke-sub-41acf6c0-gxhc6\" (UID: \"acc6c2f6-b64e-4940-b7f3-47a27f16a811\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41acf6c0-gxhc6" Oct 31 04:30:15.297472 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:15.297339 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acc6c2f6-b64e-4940-b7f3-47a27f16a811-maas-api-xks-ca\") pod \"maas-api-revoke-sub-41acf6c0-gxhc6\" (UID: \"acc6c2f6-b64e-4940-b7f3-47a27f16a811\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41acf6c0-gxhc6" Oct 31 04:30:15.297530 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:15.297484 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acc6c2f6-b64e-4940-b7f3-47a27f16a811-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-41acf6c0-gxhc6\" (UID: \"acc6c2f6-b64e-4940-b7f3-47a27f16a811\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41acf6c0-gxhc6" Oct 31 04:30:15.398717 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:15.398691 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acc6c2f6-b64e-4940-b7f3-47a27f16a811-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-41acf6c0-gxhc6\" (UID: \"acc6c2f6-b64e-4940-b7f3-47a27f16a811\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41acf6c0-gxhc6" Oct 31 04:30:15.398862 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:15.398849 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4vmx\" (UniqueName: \"kubernetes.io/projected/acc6c2f6-b64e-4940-b7f3-47a27f16a811-kube-api-access-m4vmx\") pod \"maas-api-revoke-sub-41acf6c0-gxhc6\" (UID: \"acc6c2f6-b64e-4940-b7f3-47a27f16a811\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41acf6c0-gxhc6" Oct 31 04:30:15.398906 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:15.398870 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acc6c2f6-b64e-4940-b7f3-47a27f16a811-maas-api-xks-ca\") pod \"maas-api-revoke-sub-41acf6c0-gxhc6\" (UID: \"acc6c2f6-b64e-4940-b7f3-47a27f16a811\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41acf6c0-gxhc6" Oct 31 04:30:15.399349 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:15.399310 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acc6c2f6-b64e-4940-b7f3-47a27f16a811-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-41acf6c0-gxhc6\" (UID: \"acc6c2f6-b64e-4940-b7f3-47a27f16a811\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41acf6c0-gxhc6" Oct 31 04:30:15.401193 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:15.401172 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acc6c2f6-b64e-4940-b7f3-47a27f16a811-maas-api-xks-ca\") pod \"maas-api-revoke-sub-41acf6c0-gxhc6\" (UID: \"acc6c2f6-b64e-4940-b7f3-47a27f16a811\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41acf6c0-gxhc6" Oct 31 04:30:15.422224 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:15.422198 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4vmx\" (UniqueName: \"kubernetes.io/projected/acc6c2f6-b64e-4940-b7f3-47a27f16a811-kube-api-access-m4vmx\") pod \"maas-api-revoke-sub-41acf6c0-gxhc6\" (UID: \"acc6c2f6-b64e-4940-b7f3-47a27f16a811\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41acf6c0-gxhc6" Oct 31 04:30:15.520124 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:15.520061 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41acf6c0-gxhc6" Oct 31 04:30:15.821162 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:30:15.821130 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacc6c2f6_b64e_4940_b7f3_47a27f16a811.slice/crio-100aa3fe17a1a882dee301118e3ba9873f0a5394807b971385003420f3561bab WatchSource:0}: Error finding container 100aa3fe17a1a882dee301118e3ba9873f0a5394807b971385003420f3561bab: Status 404 returned error can't find the container with id 100aa3fe17a1a882dee301118e3ba9873f0a5394807b971385003420f3561bab Oct 31 04:30:15.822499 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:15.822468 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41acf6c0-gxhc6"] Oct 31 04:30:15.848687 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:15.848661 4376 scope.go:122] "RemoveContainer" containerID="694e697cd50dcb0a7656d8013c1e7300b471e9c89bb1761a7bf85b1086b1f7a5" Oct 31 04:30:15.870475 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:15.870458 4376 scope.go:122] "RemoveContainer" containerID="cca1910d69469194747b37b4b6fc5083ef9615228e12581e276f44c5a7d0363d" Oct 31 04:30:16.226353 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:16.225685 4376 generic.go:356] "Generic (PLEG): container finished" podID="c0cc41ad-7f5b-4a5c-a989-bfafee0857e8" containerID="20210477399c246dd0b73fbb8ffe62f810561347e339808647f65ad299caa95d" exitCode=0 Oct 31 04:30:16.226353 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:16.225808 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-edc450-openshift-default-84c757994f-q4vwz" event={"ID":"c0cc41ad-7f5b-4a5c-a989-bfafee0857e8","Type":"ContainerDied","Data":"20210477399c246dd0b73fbb8ffe62f810561347e339808647f65ad299caa95d"} Oct 31 04:30:16.226353 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:16.225866 4376 scope.go:122] "RemoveContainer" containerID="28555215b5812fed9bd2c710a9a4ddd1010c7957faa478947a5a28fb3f010ad8" Oct 31 04:30:16.232003 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:16.231745 4376 generic.go:356] "Generic (PLEG): container finished" podID="ed72552d-53d8-43a8-8923-0098ab2a192b" containerID="5aa8654663c49dfafff84752fc9447c73549d4ce1651090d91980ef098253cf2" exitCode=60 Oct 31 04:30:16.232003 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:16.231911 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-965f61-29890350-wlgk6" event={"ID":"ed72552d-53d8-43a8-8923-0098ab2a192b","Type":"ContainerDied","Data":"5aa8654663c49dfafff84752fc9447c73549d4ce1651090d91980ef098253cf2"} Oct 31 04:30:16.232418 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:16.232392 4376 scope.go:122] "RemoveContainer" containerID="5aa8654663c49dfafff84752fc9447c73549d4ce1651090d91980ef098253cf2" Oct 31 04:30:16.232882 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:30:16.232853 4376 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-965f61-29890350-wlgk6_odh-ai-gateway-infra(ed72552d-53d8-43a8-8923-0098ab2a192b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-965f61-29890350-wlgk6" podUID="ed72552d-53d8-43a8-8923-0098ab2a192b" Oct 31 04:30:16.234232 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:16.234209 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41acf6c0-gxhc6" event={"ID":"acc6c2f6-b64e-4940-b7f3-47a27f16a811","Type":"ContainerStarted","Data":"100aa3fe17a1a882dee301118e3ba9873f0a5394807b971385003420f3561bab"} Oct 31 04:30:16.262406 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:16.262000 4376 scope.go:122] "RemoveContainer" containerID="25e1acaf2ccebb1ce570ceb78b70a0883c50b815d956550d89ac983700d4c2ef" Oct 31 04:30:16.292846 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:16.292821 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-965f61-29890350-wlgk6"] Oct 31 04:30:16.573468 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:16.573444 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-edc450-openshift-default-84c757994f-q4vwz" Oct 31 04:30:16.636352 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:16.630942 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0cc41ad-7f5b-4a5c-a989-bfafee0857e8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0cc41ad-7f5b-4a5c-a989-bfafee0857e8-credential-socket\") pod \"c0cc41ad-7f5b-4a5c-a989-bfafee0857e8\" (UID: \"c0cc41ad-7f5b-4a5c-a989-bfafee0857e8\") " Oct 31 04:30:16.636352 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:16.630999 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0cc41ad-7f5b-4a5c-a989-bfafee0857e8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0cc41ad-7f5b-4a5c-a989-bfafee0857e8-istio-data\") pod \"c0cc41ad-7f5b-4a5c-a989-bfafee0857e8\" (UID: \"c0cc41ad-7f5b-4a5c-a989-bfafee0857e8\") " Oct 31 04:30:16.636352 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:16.631094 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0cc41ad-7f5b-4a5c-a989-bfafee0857e8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0cc41ad-7f5b-4a5c-a989-bfafee0857e8-istio-envoy\") pod \"c0cc41ad-7f5b-4a5c-a989-bfafee0857e8\" (UID: \"c0cc41ad-7f5b-4a5c-a989-bfafee0857e8\") " Oct 31 04:30:16.636352 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:16.631178 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0cc41ad-7f5b-4a5c-a989-bfafee0857e8-kube-api-access-wm5l8\" (UniqueName: \"kubernetes.io/projected/c0cc41ad-7f5b-4a5c-a989-bfafee0857e8-kube-api-access-wm5l8\") pod \"c0cc41ad-7f5b-4a5c-a989-bfafee0857e8\" (UID: \"c0cc41ad-7f5b-4a5c-a989-bfafee0857e8\") " Oct 31 04:30:16.636352 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:16.631253 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0cc41ad-7f5b-4a5c-a989-bfafee0857e8-istio-token\" (UniqueName: \"kubernetes.io/projected/c0cc41ad-7f5b-4a5c-a989-bfafee0857e8-istio-token\") pod \"c0cc41ad-7f5b-4a5c-a989-bfafee0857e8\" (UID: \"c0cc41ad-7f5b-4a5c-a989-bfafee0857e8\") " Oct 31 04:30:16.636352 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:16.631288 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c0cc41ad-7f5b-4a5c-a989-bfafee0857e8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0cc41ad-7f5b-4a5c-a989-bfafee0857e8-istio-podinfo\") pod \"c0cc41ad-7f5b-4a5c-a989-bfafee0857e8\" (UID: \"c0cc41ad-7f5b-4a5c-a989-bfafee0857e8\") " Oct 31 04:30:16.636352 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:16.631490 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0cc41ad-7f5b-4a5c-a989-bfafee0857e8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0cc41ad-7f5b-4a5c-a989-bfafee0857e8-workload-certs\") pod \"c0cc41ad-7f5b-4a5c-a989-bfafee0857e8\" (UID: \"c0cc41ad-7f5b-4a5c-a989-bfafee0857e8\") " Oct 31 04:30:16.636352 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:16.631556 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0cc41ad-7f5b-4a5c-a989-bfafee0857e8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0cc41ad-7f5b-4a5c-a989-bfafee0857e8-istiod-ca-cert\") pod \"c0cc41ad-7f5b-4a5c-a989-bfafee0857e8\" (UID: \"c0cc41ad-7f5b-4a5c-a989-bfafee0857e8\") " Oct 31 04:30:16.636352 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:16.631623 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0cc41ad-7f5b-4a5c-a989-bfafee0857e8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0cc41ad-7f5b-4a5c-a989-bfafee0857e8-workload-socket\") pod \"c0cc41ad-7f5b-4a5c-a989-bfafee0857e8\" (UID: \"c0cc41ad-7f5b-4a5c-a989-bfafee0857e8\") " Oct 31 04:30:16.647351 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:16.646367 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0cc41ad-7f5b-4a5c-a989-bfafee0857e8-istio-data" pod "c0cc41ad-7f5b-4a5c-a989-bfafee0857e8" (UID: "c0cc41ad-7f5b-4a5c-a989-bfafee0857e8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:30:16.647351 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:16.646603 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0cc41ad-7f5b-4a5c-a989-bfafee0857e8-credential-socket" pod "c0cc41ad-7f5b-4a5c-a989-bfafee0857e8" (UID: "c0cc41ad-7f5b-4a5c-a989-bfafee0857e8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:30:16.647351 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:16.646796 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0cc41ad-7f5b-4a5c-a989-bfafee0857e8-workload-certs" pod "c0cc41ad-7f5b-4a5c-a989-bfafee0857e8" (UID: "c0cc41ad-7f5b-4a5c-a989-bfafee0857e8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:30:16.647351 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:16.646890 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0cc41ad-7f5b-4a5c-a989-bfafee0857e8-istiod-ca-cert" pod "c0cc41ad-7f5b-4a5c-a989-bfafee0857e8" (UID: "c0cc41ad-7f5b-4a5c-a989-bfafee0857e8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:30:16.647351 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:16.647150 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0cc41ad-7f5b-4a5c-a989-bfafee0857e8-workload-socket" pod "c0cc41ad-7f5b-4a5c-a989-bfafee0857e8" (UID: "c0cc41ad-7f5b-4a5c-a989-bfafee0857e8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:30:16.667439 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:16.667395 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0cc41ad-7f5b-4a5c-a989-bfafee0857e8-istio-token" pod "c0cc41ad-7f5b-4a5c-a989-bfafee0857e8" (UID: "c0cc41ad-7f5b-4a5c-a989-bfafee0857e8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:30:16.672405 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:16.671080 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c0cc41ad-7f5b-4a5c-a989-bfafee0857e8-istio-podinfo" pod "c0cc41ad-7f5b-4a5c-a989-bfafee0857e8" (UID: "c0cc41ad-7f5b-4a5c-a989-bfafee0857e8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 04:30:16.676351 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:16.675885 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0cc41ad-7f5b-4a5c-a989-bfafee0857e8-kube-api-access-wm5l8" pod "c0cc41ad-7f5b-4a5c-a989-bfafee0857e8" (UID: "c0cc41ad-7f5b-4a5c-a989-bfafee0857e8"). InnerVolumeSpecName "kube-api-access-wm5l8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:30:16.676944 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:16.676858 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0cc41ad-7f5b-4a5c-a989-bfafee0857e8-istio-envoy" pod "c0cc41ad-7f5b-4a5c-a989-bfafee0857e8" (UID: "c0cc41ad-7f5b-4a5c-a989-bfafee0857e8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:30:16.733586 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:16.733559 4376 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0cc41ad-7f5b-4a5c-a989-bfafee0857e8-workload-socket\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:30:16.733586 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:16.733586 4376 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0cc41ad-7f5b-4a5c-a989-bfafee0857e8-credential-socket\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:30:16.733731 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:16.733596 4376 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0cc41ad-7f5b-4a5c-a989-bfafee0857e8-istio-data\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:30:16.733731 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:16.733604 4376 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0cc41ad-7f5b-4a5c-a989-bfafee0857e8-istio-envoy\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:30:16.733731 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:16.733612 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wm5l8\" (UniqueName: \"kubernetes.io/projected/c0cc41ad-7f5b-4a5c-a989-bfafee0857e8-kube-api-access-wm5l8\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:30:16.733731 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:16.733620 4376 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0cc41ad-7f5b-4a5c-a989-bfafee0857e8-istio-token\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:30:16.733731 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:16.733630 4376 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0cc41ad-7f5b-4a5c-a989-bfafee0857e8-istio-podinfo\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:30:16.733731 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:16.733637 4376 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0cc41ad-7f5b-4a5c-a989-bfafee0857e8-workload-certs\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:30:16.733731 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:16.733645 4376 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0cc41ad-7f5b-4a5c-a989-bfafee0857e8-istiod-ca-cert\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:30:17.242126 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:17.242090 4376 generic.go:356] "Generic (PLEG): container finished" podID="acc6c2f6-b64e-4940-b7f3-47a27f16a811" containerID="95bf74fe482191a98de52d27b773dc593cf08baab1d4d40517fcc70694fb57e8" exitCode=0 Oct 31 04:30:17.242304 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:17.242176 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41acf6c0-gxhc6" event={"ID":"acc6c2f6-b64e-4940-b7f3-47a27f16a811","Type":"ContainerDied","Data":"95bf74fe482191a98de52d27b773dc593cf08baab1d4d40517fcc70694fb57e8"} Oct 31 04:30:17.243822 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:17.243790 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-edc450-openshift-default-84c757994f-q4vwz" event={"ID":"c0cc41ad-7f5b-4a5c-a989-bfafee0857e8","Type":"ContainerDied","Data":"5078d7ccfc62292358c6823e11662eeb291f4c0ebbec36a4ba2450b53b23f720"} Oct 31 04:30:17.243822 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:17.243807 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-edc450-openshift-default-84c757994f-q4vwz" Oct 31 04:30:17.243994 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:17.243845 4376 scope.go:122] "RemoveContainer" containerID="20210477399c246dd0b73fbb8ffe62f810561347e339808647f65ad299caa95d" Oct 31 04:30:17.379753 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:17.379723 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-edc450-openshift-default-84c757994f-q4vwz"] Oct 31 04:30:17.394367 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:17.394339 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-edc450-openshift-default-84c757994f-q4vwz"] Oct 31 04:30:17.425342 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:17.425312 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-965f61-29890350-wlgk6" Oct 31 04:30:17.544022 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:17.543939 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed72552d-53d8-43a8-8923-0098ab2a192b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed72552d-53d8-43a8-8923-0098ab2a192b-maas-api-service-ca\") pod \"ed72552d-53d8-43a8-8923-0098ab2a192b\" (UID: \"ed72552d-53d8-43a8-8923-0098ab2a192b\") " Oct 31 04:30:17.544022 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:17.544014 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed72552d-53d8-43a8-8923-0098ab2a192b-kube-api-access-2skt8\" (UniqueName: \"kubernetes.io/projected/ed72552d-53d8-43a8-8923-0098ab2a192b-kube-api-access-2skt8\") pod \"ed72552d-53d8-43a8-8923-0098ab2a192b\" (UID: \"ed72552d-53d8-43a8-8923-0098ab2a192b\") " Oct 31 04:30:17.544380 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:17.544355 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed72552d-53d8-43a8-8923-0098ab2a192b-maas-api-service-ca" pod "ed72552d-53d8-43a8-8923-0098ab2a192b" (UID: "ed72552d-53d8-43a8-8923-0098ab2a192b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:30:17.544652 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:17.544636 4376 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed72552d-53d8-43a8-8923-0098ab2a192b-maas-api-service-ca\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:30:17.546076 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:17.546054 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed72552d-53d8-43a8-8923-0098ab2a192b-kube-api-access-2skt8" pod "ed72552d-53d8-43a8-8923-0098ab2a192b" (UID: "ed72552d-53d8-43a8-8923-0098ab2a192b"). InnerVolumeSpecName "kube-api-access-2skt8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:30:17.646139 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:17.646106 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2skt8\" (UniqueName: \"kubernetes.io/projected/ed72552d-53d8-43a8-8923-0098ab2a192b-kube-api-access-2skt8\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:30:18.261445 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:18.261413 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-965f61-29890350-wlgk6" Oct 31 04:30:18.261445 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:18.261422 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-965f61-29890350-wlgk6" event={"ID":"ed72552d-53d8-43a8-8923-0098ab2a192b","Type":"ContainerDied","Data":"10622f90909171ef6cb3a93023c7e69930bc08086ce46ccc0361785c48342103"} Oct 31 04:30:18.261673 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:18.261507 4376 scope.go:122] "RemoveContainer" containerID="5aa8654663c49dfafff84752fc9447c73549d4ce1651090d91980ef098253cf2" Oct 31 04:30:18.315346 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:18.315306 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-965f61-29890350-wlgk6"] Oct 31 04:30:18.331773 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:18.331746 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-965f61-29890350-wlgk6"] Oct 31 04:30:18.416435 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:18.416406 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41acf6c0-gxhc6" Oct 31 04:30:18.443914 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:18.443888 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0cc41ad-7f5b-4a5c-a989-bfafee0857e8" path="/var/lib/kubelet/pods/c0cc41ad-7f5b-4a5c-a989-bfafee0857e8/volumes" Oct 31 04:30:18.444402 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:18.444388 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed72552d-53d8-43a8-8923-0098ab2a192b" path="/var/lib/kubelet/pods/ed72552d-53d8-43a8-8923-0098ab2a192b/volumes" Oct 31 04:30:18.466587 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:18.466553 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/acc6c2f6-b64e-4940-b7f3-47a27f16a811-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acc6c2f6-b64e-4940-b7f3-47a27f16a811-maas-api-openshift-ca\") pod \"acc6c2f6-b64e-4940-b7f3-47a27f16a811\" (UID: \"acc6c2f6-b64e-4940-b7f3-47a27f16a811\") " Oct 31 04:30:18.466754 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:18.466724 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/acc6c2f6-b64e-4940-b7f3-47a27f16a811-kube-api-access-m4vmx\" (UniqueName: \"kubernetes.io/projected/acc6c2f6-b64e-4940-b7f3-47a27f16a811-kube-api-access-m4vmx\") pod \"acc6c2f6-b64e-4940-b7f3-47a27f16a811\" (UID: \"acc6c2f6-b64e-4940-b7f3-47a27f16a811\") " Oct 31 04:30:18.467470 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:18.466881 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/acc6c2f6-b64e-4940-b7f3-47a27f16a811-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acc6c2f6-b64e-4940-b7f3-47a27f16a811-maas-api-xks-ca\") pod \"acc6c2f6-b64e-4940-b7f3-47a27f16a811\" (UID: \"acc6c2f6-b64e-4940-b7f3-47a27f16a811\") " Oct 31 04:30:18.467929 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:18.467907 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acc6c2f6-b64e-4940-b7f3-47a27f16a811-maas-api-openshift-ca" pod "acc6c2f6-b64e-4940-b7f3-47a27f16a811" (UID: "acc6c2f6-b64e-4940-b7f3-47a27f16a811"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:30:18.470241 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:18.470216 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/acc6c2f6-b64e-4940-b7f3-47a27f16a811-maas-api-xks-ca" pod "acc6c2f6-b64e-4940-b7f3-47a27f16a811" (UID: "acc6c2f6-b64e-4940-b7f3-47a27f16a811"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:30:18.470375 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:18.470286 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acc6c2f6-b64e-4940-b7f3-47a27f16a811-kube-api-access-m4vmx" pod "acc6c2f6-b64e-4940-b7f3-47a27f16a811" (UID: "acc6c2f6-b64e-4940-b7f3-47a27f16a811"). InnerVolumeSpecName "kube-api-access-m4vmx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:30:18.569165 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:18.569093 4376 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acc6c2f6-b64e-4940-b7f3-47a27f16a811-maas-api-xks-ca\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:30:18.569165 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:18.569129 4376 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acc6c2f6-b64e-4940-b7f3-47a27f16a811-maas-api-openshift-ca\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:30:18.569165 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:18.569144 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4vmx\" (UniqueName: \"kubernetes.io/projected/acc6c2f6-b64e-4940-b7f3-47a27f16a811-kube-api-access-m4vmx\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:30:19.271068 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:19.271020 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41acf6c0-gxhc6" event={"ID":"acc6c2f6-b64e-4940-b7f3-47a27f16a811","Type":"ContainerDied","Data":"100aa3fe17a1a882dee301118e3ba9873f0a5394807b971385003420f3561bab"} Oct 31 04:30:19.271467 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:19.271087 4376 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="100aa3fe17a1a882dee301118e3ba9873f0a5394807b971385003420f3561bab" Oct 31 04:30:19.271467 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:19.271040 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41acf6c0-gxhc6" Oct 31 04:30:25.662236 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:25.662155 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-b3ca79-776b5db7b5-8f4k2"] Oct 31 04:30:25.695946 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:25.695911 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-b3ca79-6f8bf456dc-t4295"] Oct 31 04:30:25.772415 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:25.772378 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-b3ca79-78445fb8df-792kj"] Oct 31 04:30:25.775343 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:25.775292 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b3ca79-78445fb8df-792kj" podUID="623914fe-4610-4eeb-ba51-15e3e55faa31" containerName="maas-api" containerID="cri-o://a297c0bb9eb4f3466d2e583a1e09e4f2bdb0d57e8143d06a9f7374bc33c048c1" gracePeriod=30 Oct 31 04:30:26.046927 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:26.046898 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b3ca79-78445fb8df-792kj" Oct 31 04:30:26.174959 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:26.174921 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/623914fe-4610-4eeb-ba51-15e3e55faa31-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/623914fe-4610-4eeb-ba51-15e3e55faa31-maas-api-tls\") pod \"623914fe-4610-4eeb-ba51-15e3e55faa31\" (UID: \"623914fe-4610-4eeb-ba51-15e3e55faa31\") " Oct 31 04:30:26.175164 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:26.175018 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/623914fe-4610-4eeb-ba51-15e3e55faa31-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/623914fe-4610-4eeb-ba51-15e3e55faa31-openshift-service-ca\") pod \"623914fe-4610-4eeb-ba51-15e3e55faa31\" (UID: \"623914fe-4610-4eeb-ba51-15e3e55faa31\") " Oct 31 04:30:26.175164 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:26.175110 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/623914fe-4610-4eeb-ba51-15e3e55faa31-kube-api-access-lr7mn\" (UniqueName: \"kubernetes.io/projected/623914fe-4610-4eeb-ba51-15e3e55faa31-kube-api-access-lr7mn\") pod \"623914fe-4610-4eeb-ba51-15e3e55faa31\" (UID: \"623914fe-4610-4eeb-ba51-15e3e55faa31\") " Oct 31 04:30:26.175164 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:26.175139 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/623914fe-4610-4eeb-ba51-15e3e55faa31-metrics-certs\" (UniqueName: \"kubernetes.io/secret/623914fe-4610-4eeb-ba51-15e3e55faa31-metrics-certs\") pod \"623914fe-4610-4eeb-ba51-15e3e55faa31\" (UID: \"623914fe-4610-4eeb-ba51-15e3e55faa31\") " Oct 31 04:30:26.175486 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:26.175454 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/623914fe-4610-4eeb-ba51-15e3e55faa31-openshift-service-ca" pod "623914fe-4610-4eeb-ba51-15e3e55faa31" (UID: "623914fe-4610-4eeb-ba51-15e3e55faa31"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:30:26.175773 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:26.175748 4376 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/623914fe-4610-4eeb-ba51-15e3e55faa31-openshift-service-ca\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:30:26.177465 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:26.177435 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/623914fe-4610-4eeb-ba51-15e3e55faa31-maas-api-tls" pod "623914fe-4610-4eeb-ba51-15e3e55faa31" (UID: "623914fe-4610-4eeb-ba51-15e3e55faa31"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:30:26.177465 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:26.177458 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/623914fe-4610-4eeb-ba51-15e3e55faa31-metrics-certs" pod "623914fe-4610-4eeb-ba51-15e3e55faa31" (UID: "623914fe-4610-4eeb-ba51-15e3e55faa31"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:30:26.177617 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:26.177488 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/623914fe-4610-4eeb-ba51-15e3e55faa31-kube-api-access-lr7mn" pod "623914fe-4610-4eeb-ba51-15e3e55faa31" (UID: "623914fe-4610-4eeb-ba51-15e3e55faa31"). InnerVolumeSpecName "kube-api-access-lr7mn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:30:26.287406 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:26.277749 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lr7mn\" (UniqueName: \"kubernetes.io/projected/623914fe-4610-4eeb-ba51-15e3e55faa31-kube-api-access-lr7mn\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:30:26.287406 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:26.277788 4376 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/623914fe-4610-4eeb-ba51-15e3e55faa31-metrics-certs\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:30:26.287406 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:26.277804 4376 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/623914fe-4610-4eeb-ba51-15e3e55faa31-maas-api-tls\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:30:26.385373 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:26.380997 4376 generic.go:356] "Generic (PLEG): container finished" podID="623914fe-4610-4eeb-ba51-15e3e55faa31" containerID="a297c0bb9eb4f3466d2e583a1e09e4f2bdb0d57e8143d06a9f7374bc33c048c1" exitCode=0 Oct 31 04:30:26.385373 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:26.381116 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b3ca79-78445fb8df-792kj" event={"ID":"623914fe-4610-4eeb-ba51-15e3e55faa31","Type":"ContainerDied","Data":"a297c0bb9eb4f3466d2e583a1e09e4f2bdb0d57e8143d06a9f7374bc33c048c1"} Oct 31 04:30:26.385373 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:26.381177 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b3ca79-78445fb8df-792kj" event={"ID":"623914fe-4610-4eeb-ba51-15e3e55faa31","Type":"ContainerDied","Data":"0a2f3d609369c2e4a7c09249482d14139e5d49603095b02e0060a2c887ffa42b"} Oct 31 04:30:26.385373 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:26.381208 4376 scope.go:122] "RemoveContainer" containerID="a297c0bb9eb4f3466d2e583a1e09e4f2bdb0d57e8143d06a9f7374bc33c048c1" Oct 31 04:30:26.385373 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:26.381467 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b3ca79-78445fb8df-792kj" Oct 31 04:30:26.407616 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:26.407596 4376 scope.go:122] "RemoveContainer" containerID="a297c0bb9eb4f3466d2e583a1e09e4f2bdb0d57e8143d06a9f7374bc33c048c1" Oct 31 04:30:26.407937 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:30:26.407918 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a297c0bb9eb4f3466d2e583a1e09e4f2bdb0d57e8143d06a9f7374bc33c048c1\": container with ID starting with a297c0bb9eb4f3466d2e583a1e09e4f2bdb0d57e8143d06a9f7374bc33c048c1 not found: ID does not exist" containerID="a297c0bb9eb4f3466d2e583a1e09e4f2bdb0d57e8143d06a9f7374bc33c048c1" Oct 31 04:30:26.408039 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:26.407945 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a297c0bb9eb4f3466d2e583a1e09e4f2bdb0d57e8143d06a9f7374bc33c048c1"} err="failed to get container status \"a297c0bb9eb4f3466d2e583a1e09e4f2bdb0d57e8143d06a9f7374bc33c048c1\": rpc error: code = NotFound desc = could not find container \"a297c0bb9eb4f3466d2e583a1e09e4f2bdb0d57e8143d06a9f7374bc33c048c1\": container with ID starting with a297c0bb9eb4f3466d2e583a1e09e4f2bdb0d57e8143d06a9f7374bc33c048c1 not found: ID does not exist" Oct 31 04:30:26.446296 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:26.446028 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-b3ca79-78445fb8df-792kj"] Oct 31 04:30:26.446787 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:26.446686 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-b3ca79-78445fb8df-792kj"] Oct 31 04:30:28.445357 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:28.445302 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="623914fe-4610-4eeb-ba51-15e3e55faa31" path="/var/lib/kubelet/pods/623914fe-4610-4eeb-ba51-15e3e55faa31/volumes" Oct 31 04:30:30.484837 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:30.484796 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-e37d27-75bf88d5c4-mxc78"] Oct 31 04:30:30.485617 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:30.485601 4376 cpu_manager.go:414] "containerMap: removing container" podUID="acc6c2f6-b64e-4940-b7f3-47a27f16a811" containerName="revoke-keys" Oct 31 04:30:30.485674 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:30.485620 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="acc6c2f6-b64e-4940-b7f3-47a27f16a811" containerName="revoke-keys" Oct 31 04:30:30.485674 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:30.485639 4376 cpu_manager.go:414] "containerMap: removing container" podUID="ed72552d-53d8-43a8-8923-0098ab2a192b" containerName="cleanup" Oct 31 04:30:30.485674 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:30.485647 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed72552d-53d8-43a8-8923-0098ab2a192b" containerName="cleanup" Oct 31 04:30:30.485674 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:30.485667 4376 cpu_manager.go:414] "containerMap: removing container" podUID="c0cc41ad-7f5b-4a5c-a989-bfafee0857e8" containerName="istio-proxy" Oct 31 04:30:30.485802 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:30.485674 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0cc41ad-7f5b-4a5c-a989-bfafee0857e8" containerName="istio-proxy" Oct 31 04:30:30.485802 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:30.485727 4376 cpu_manager.go:414] "containerMap: removing container" podUID="c0cc41ad-7f5b-4a5c-a989-bfafee0857e8" containerName="istio-proxy" Oct 31 04:30:30.485802 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:30.485735 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0cc41ad-7f5b-4a5c-a989-bfafee0857e8" containerName="istio-proxy" Oct 31 04:30:30.485802 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:30.485745 4376 cpu_manager.go:414] "containerMap: removing container" podUID="ed72552d-53d8-43a8-8923-0098ab2a192b" containerName="cleanup" Oct 31 04:30:30.485802 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:30.485753 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed72552d-53d8-43a8-8923-0098ab2a192b" containerName="cleanup" Oct 31 04:30:30.485802 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:30.485764 4376 cpu_manager.go:414] "containerMap: removing container" podUID="623914fe-4610-4eeb-ba51-15e3e55faa31" containerName="maas-api" Oct 31 04:30:30.485802 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:30.485772 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="623914fe-4610-4eeb-ba51-15e3e55faa31" containerName="maas-api" Oct 31 04:30:30.486019 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:30.485880 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0cc41ad-7f5b-4a5c-a989-bfafee0857e8" containerName="istio-proxy" Oct 31 04:30:30.486019 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:30.485892 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed72552d-53d8-43a8-8923-0098ab2a192b" containerName="cleanup" Oct 31 04:30:30.486019 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:30.485903 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="623914fe-4610-4eeb-ba51-15e3e55faa31" containerName="maas-api" Oct 31 04:30:30.486019 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:30.485912 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed72552d-53d8-43a8-8923-0098ab2a192b" containerName="cleanup" Oct 31 04:30:30.486019 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:30.485924 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="acc6c2f6-b64e-4940-b7f3-47a27f16a811" containerName="revoke-keys" Oct 31 04:30:30.486019 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:30.485933 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed72552d-53d8-43a8-8923-0098ab2a192b" containerName="cleanup" Oct 31 04:30:30.486019 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:30.485944 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0cc41ad-7f5b-4a5c-a989-bfafee0857e8" containerName="istio-proxy" Oct 31 04:30:30.491309 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:30.491290 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-e37d27-75bf88d5c4-mxc78" Oct 31 04:30:30.507649 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:30.507624 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-e37d27-75bf88d5c4-mxc78"] Oct 31 04:30:30.524060 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:30.524037 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-e37d27-serving-cert\"" Oct 31 04:30:30.558414 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:30.558384 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/216d467a-46f9-4fbe-ab89-899e7d9d9f60-metrics-certs\") pod \"maas-api-e2e-praxis-a-e37d27-75bf88d5c4-mxc78\" (UID: \"216d467a-46f9-4fbe-ab89-899e7d9d9f60\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-e37d27-75bf88d5c4-mxc78" Oct 31 04:30:30.558576 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:30.558434 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnx4l\" (UniqueName: \"kubernetes.io/projected/216d467a-46f9-4fbe-ab89-899e7d9d9f60-kube-api-access-rnx4l\") pod \"maas-api-e2e-praxis-a-e37d27-75bf88d5c4-mxc78\" (UID: \"216d467a-46f9-4fbe-ab89-899e7d9d9f60\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-e37d27-75bf88d5c4-mxc78" Oct 31 04:30:30.558576 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:30.558505 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/216d467a-46f9-4fbe-ab89-899e7d9d9f60-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-e37d27-75bf88d5c4-mxc78\" (UID: \"216d467a-46f9-4fbe-ab89-899e7d9d9f60\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-e37d27-75bf88d5c4-mxc78" Oct 31 04:30:30.558754 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:30.558735 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/216d467a-46f9-4fbe-ab89-899e7d9d9f60-maas-api-tls\") pod \"maas-api-e2e-praxis-a-e37d27-75bf88d5c4-mxc78\" (UID: \"216d467a-46f9-4fbe-ab89-899e7d9d9f60\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-e37d27-75bf88d5c4-mxc78" Oct 31 04:30:30.660181 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:30.660147 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/216d467a-46f9-4fbe-ab89-899e7d9d9f60-maas-api-tls\") pod \"maas-api-e2e-praxis-a-e37d27-75bf88d5c4-mxc78\" (UID: \"216d467a-46f9-4fbe-ab89-899e7d9d9f60\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-e37d27-75bf88d5c4-mxc78" Oct 31 04:30:30.660395 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:30.660259 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/216d467a-46f9-4fbe-ab89-899e7d9d9f60-metrics-certs\") pod \"maas-api-e2e-praxis-a-e37d27-75bf88d5c4-mxc78\" (UID: \"216d467a-46f9-4fbe-ab89-899e7d9d9f60\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-e37d27-75bf88d5c4-mxc78" Oct 31 04:30:30.660395 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:30.660291 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnx4l\" (UniqueName: \"kubernetes.io/projected/216d467a-46f9-4fbe-ab89-899e7d9d9f60-kube-api-access-rnx4l\") pod \"maas-api-e2e-praxis-a-e37d27-75bf88d5c4-mxc78\" (UID: \"216d467a-46f9-4fbe-ab89-899e7d9d9f60\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-e37d27-75bf88d5c4-mxc78" Oct 31 04:30:30.660395 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:30.660322 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/216d467a-46f9-4fbe-ab89-899e7d9d9f60-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-e37d27-75bf88d5c4-mxc78\" (UID: \"216d467a-46f9-4fbe-ab89-899e7d9d9f60\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-e37d27-75bf88d5c4-mxc78" Oct 31 04:30:30.661005 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:30.660980 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/216d467a-46f9-4fbe-ab89-899e7d9d9f60-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-e37d27-75bf88d5c4-mxc78\" (UID: \"216d467a-46f9-4fbe-ab89-899e7d9d9f60\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-e37d27-75bf88d5c4-mxc78" Oct 31 04:30:30.662615 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:30.662593 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/216d467a-46f9-4fbe-ab89-899e7d9d9f60-metrics-certs\") pod \"maas-api-e2e-praxis-a-e37d27-75bf88d5c4-mxc78\" (UID: \"216d467a-46f9-4fbe-ab89-899e7d9d9f60\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-e37d27-75bf88d5c4-mxc78" Oct 31 04:30:30.662734 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:30.662660 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/216d467a-46f9-4fbe-ab89-899e7d9d9f60-maas-api-tls\") pod \"maas-api-e2e-praxis-a-e37d27-75bf88d5c4-mxc78\" (UID: \"216d467a-46f9-4fbe-ab89-899e7d9d9f60\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-e37d27-75bf88d5c4-mxc78" Oct 31 04:30:30.662734 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:30.662650 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-b3ca79-776b5db7b5-8f4k2" podUID="45ff2ef8-aff5-4df7-8b70-6cea9e0b9332" containerName="payload-processing" containerID="cri-o://c7197a13aefa66039a025bd6c77e313b8880e05a79cf716885bb47769a373ad9" gracePeriod=25 Oct 31 04:30:30.684158 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:30.684134 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnx4l\" (UniqueName: \"kubernetes.io/projected/216d467a-46f9-4fbe-ab89-899e7d9d9f60-kube-api-access-rnx4l\") pod \"maas-api-e2e-praxis-a-e37d27-75bf88d5c4-mxc78\" (UID: \"216d467a-46f9-4fbe-ab89-899e7d9d9f60\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-e37d27-75bf88d5c4-mxc78" Oct 31 04:30:30.696943 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:30.696899 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b3ca79-6f8bf456dc-t4295" podUID="25409dbd-265a-4852-9b66-a7689871e139" containerName="payload-pre-processing" containerID="cri-o://d23098ddb93b43c5232d7156e15d9632b9f579252f1ea63a47072952b0bcf11d" gracePeriod=25 Oct 31 04:30:30.820070 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:30.819682 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-e37d27-75bf88d5c4-mxc78" Oct 31 04:30:31.045599 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:31.045573 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-b3ca79-776b5db7b5-8f4k2" Oct 31 04:30:31.047508 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:31.047491 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b3ca79-6f8bf456dc-t4295" Oct 31 04:30:31.067239 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:31.067216 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45ff2ef8-aff5-4df7-8b70-6cea9e0b9332-kube-api-access-4ngms\" (UniqueName: \"kubernetes.io/projected/45ff2ef8-aff5-4df7-8b70-6cea9e0b9332-kube-api-access-4ngms\") pod \"45ff2ef8-aff5-4df7-8b70-6cea9e0b9332\" (UID: \"45ff2ef8-aff5-4df7-8b70-6cea9e0b9332\") " Oct 31 04:30:31.067364 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:31.067339 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25409dbd-265a-4852-9b66-a7689871e139-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25409dbd-265a-4852-9b66-a7689871e139-plugins-config-volume\") pod \"25409dbd-265a-4852-9b66-a7689871e139\" (UID: \"25409dbd-265a-4852-9b66-a7689871e139\") " Oct 31 04:30:31.067408 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:31.067383 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25409dbd-265a-4852-9b66-a7689871e139-kube-api-access-z9spv\" (UniqueName: \"kubernetes.io/projected/25409dbd-265a-4852-9b66-a7689871e139-kube-api-access-z9spv\") pod \"25409dbd-265a-4852-9b66-a7689871e139\" (UID: \"25409dbd-265a-4852-9b66-a7689871e139\") " Oct 31 04:30:31.067446 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:31.067429 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45ff2ef8-aff5-4df7-8b70-6cea9e0b9332-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45ff2ef8-aff5-4df7-8b70-6cea9e0b9332-plugins-config-volume\") pod \"45ff2ef8-aff5-4df7-8b70-6cea9e0b9332\" (UID: \"45ff2ef8-aff5-4df7-8b70-6cea9e0b9332\") " Oct 31 04:30:31.067735 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:31.067710 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25409dbd-265a-4852-9b66-a7689871e139-plugins-config-volume" pod "25409dbd-265a-4852-9b66-a7689871e139" (UID: "25409dbd-265a-4852-9b66-a7689871e139"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:30:31.067851 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:31.067832 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45ff2ef8-aff5-4df7-8b70-6cea9e0b9332-plugins-config-volume" pod "45ff2ef8-aff5-4df7-8b70-6cea9e0b9332" (UID: "45ff2ef8-aff5-4df7-8b70-6cea9e0b9332"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:30:31.067930 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:31.067916 4376 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25409dbd-265a-4852-9b66-a7689871e139-plugins-config-volume\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:30:31.069321 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:31.069301 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45ff2ef8-aff5-4df7-8b70-6cea9e0b9332-kube-api-access-4ngms" pod "45ff2ef8-aff5-4df7-8b70-6cea9e0b9332" (UID: "45ff2ef8-aff5-4df7-8b70-6cea9e0b9332"). InnerVolumeSpecName "kube-api-access-4ngms". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:30:31.069522 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:31.069506 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25409dbd-265a-4852-9b66-a7689871e139-kube-api-access-z9spv" pod "25409dbd-265a-4852-9b66-a7689871e139" (UID: "25409dbd-265a-4852-9b66-a7689871e139"). InnerVolumeSpecName "kube-api-access-z9spv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:30:31.136083 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:30:31.136048 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod216d467a_46f9_4fbe_ab89_899e7d9d9f60.slice/crio-918c68930ab1d9c95105b468bd9b63147066557566f97e5115606f37e25bb0f4 WatchSource:0}: Error finding container 918c68930ab1d9c95105b468bd9b63147066557566f97e5115606f37e25bb0f4: Status 404 returned error can't find the container with id 918c68930ab1d9c95105b468bd9b63147066557566f97e5115606f37e25bb0f4 Oct 31 04:30:31.136521 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:31.136497 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-e37d27-75bf88d5c4-mxc78"] Oct 31 04:30:31.168879 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:31.168851 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ngms\" (UniqueName: \"kubernetes.io/projected/45ff2ef8-aff5-4df7-8b70-6cea9e0b9332-kube-api-access-4ngms\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:30:31.168879 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:31.168879 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9spv\" (UniqueName: \"kubernetes.io/projected/25409dbd-265a-4852-9b66-a7689871e139-kube-api-access-z9spv\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:30:31.169039 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:31.168888 4376 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45ff2ef8-aff5-4df7-8b70-6cea9e0b9332-plugins-config-volume\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:30:31.457613 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:31.457573 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-e37d27-75bf88d5c4-mxc78" event={"ID":"216d467a-46f9-4fbe-ab89-899e7d9d9f60","Type":"ContainerStarted","Data":"918c68930ab1d9c95105b468bd9b63147066557566f97e5115606f37e25bb0f4"} Oct 31 04:30:31.459415 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:31.459371 4376 generic.go:356] "Generic (PLEG): container finished" podID="45ff2ef8-aff5-4df7-8b70-6cea9e0b9332" containerID="c7197a13aefa66039a025bd6c77e313b8880e05a79cf716885bb47769a373ad9" exitCode=0 Oct 31 04:30:31.459415 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:31.459426 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-b3ca79-776b5db7b5-8f4k2" Oct 31 04:30:31.459630 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:31.459462 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-b3ca79-776b5db7b5-8f4k2" event={"ID":"45ff2ef8-aff5-4df7-8b70-6cea9e0b9332","Type":"ContainerDied","Data":"c7197a13aefa66039a025bd6c77e313b8880e05a79cf716885bb47769a373ad9"} Oct 31 04:30:31.459630 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:31.459507 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-b3ca79-776b5db7b5-8f4k2" event={"ID":"45ff2ef8-aff5-4df7-8b70-6cea9e0b9332","Type":"ContainerDied","Data":"91456758cab9d968eb5a38ef3d9810e35ae7484c6b02e5ac2644f08c1ad2218e"} Oct 31 04:30:31.459630 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:31.459521 4376 scope.go:122] "RemoveContainer" containerID="c7197a13aefa66039a025bd6c77e313b8880e05a79cf716885bb47769a373ad9" Oct 31 04:30:31.460834 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:31.460810 4376 generic.go:356] "Generic (PLEG): container finished" podID="25409dbd-265a-4852-9b66-a7689871e139" containerID="d23098ddb93b43c5232d7156e15d9632b9f579252f1ea63a47072952b0bcf11d" exitCode=0 Oct 31 04:30:31.460950 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:31.460848 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b3ca79-6f8bf456dc-t4295" event={"ID":"25409dbd-265a-4852-9b66-a7689871e139","Type":"ContainerDied","Data":"d23098ddb93b43c5232d7156e15d9632b9f579252f1ea63a47072952b0bcf11d"} Oct 31 04:30:31.460950 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:31.460864 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b3ca79-6f8bf456dc-t4295" Oct 31 04:30:31.460950 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:31.460888 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b3ca79-6f8bf456dc-t4295" event={"ID":"25409dbd-265a-4852-9b66-a7689871e139","Type":"ContainerDied","Data":"49edf5e78905662852cb533faafb2a3fd5e05136b41005e4729e82c352eb0f08"} Oct 31 04:30:31.469525 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:31.469471 4376 scope.go:122] "RemoveContainer" containerID="c7197a13aefa66039a025bd6c77e313b8880e05a79cf716885bb47769a373ad9" Oct 31 04:30:31.469995 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:30:31.469976 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7197a13aefa66039a025bd6c77e313b8880e05a79cf716885bb47769a373ad9\": container with ID starting with c7197a13aefa66039a025bd6c77e313b8880e05a79cf716885bb47769a373ad9 not found: ID does not exist" containerID="c7197a13aefa66039a025bd6c77e313b8880e05a79cf716885bb47769a373ad9" Oct 31 04:30:31.470072 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:31.470001 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7197a13aefa66039a025bd6c77e313b8880e05a79cf716885bb47769a373ad9"} err="failed to get container status \"c7197a13aefa66039a025bd6c77e313b8880e05a79cf716885bb47769a373ad9\": rpc error: code = NotFound desc = could not find container \"c7197a13aefa66039a025bd6c77e313b8880e05a79cf716885bb47769a373ad9\": container with ID starting with c7197a13aefa66039a025bd6c77e313b8880e05a79cf716885bb47769a373ad9 not found: ID does not exist" Oct 31 04:30:31.470072 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:31.470018 4376 scope.go:122] "RemoveContainer" containerID="d23098ddb93b43c5232d7156e15d9632b9f579252f1ea63a47072952b0bcf11d" Oct 31 04:30:31.478976 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:31.478959 4376 scope.go:122] "RemoveContainer" containerID="d23098ddb93b43c5232d7156e15d9632b9f579252f1ea63a47072952b0bcf11d" Oct 31 04:30:31.479220 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:30:31.479203 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d23098ddb93b43c5232d7156e15d9632b9f579252f1ea63a47072952b0bcf11d\": container with ID starting with d23098ddb93b43c5232d7156e15d9632b9f579252f1ea63a47072952b0bcf11d not found: ID does not exist" containerID="d23098ddb93b43c5232d7156e15d9632b9f579252f1ea63a47072952b0bcf11d" Oct 31 04:30:31.479276 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:31.479226 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d23098ddb93b43c5232d7156e15d9632b9f579252f1ea63a47072952b0bcf11d"} err="failed to get container status \"d23098ddb93b43c5232d7156e15d9632b9f579252f1ea63a47072952b0bcf11d\": rpc error: code = NotFound desc = could not find container \"d23098ddb93b43c5232d7156e15d9632b9f579252f1ea63a47072952b0bcf11d\": container with ID starting with d23098ddb93b43c5232d7156e15d9632b9f579252f1ea63a47072952b0bcf11d not found: ID does not exist" Oct 31 04:30:31.516255 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:31.512950 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-b3ca79-776b5db7b5-8f4k2"] Oct 31 04:30:31.519891 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:31.519865 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-b3ca79-776b5db7b5-8f4k2"] Oct 31 04:30:31.553018 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:31.552990 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-b3ca79-6f8bf456dc-t4295"] Oct 31 04:30:31.556384 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:31.556361 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-b3ca79-6f8bf456dc-t4295"] Oct 31 04:30:32.444189 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:32.444156 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25409dbd-265a-4852-9b66-a7689871e139" path="/var/lib/kubelet/pods/25409dbd-265a-4852-9b66-a7689871e139/volumes" Oct 31 04:30:32.444606 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:32.444590 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45ff2ef8-aff5-4df7-8b70-6cea9e0b9332" path="/var/lib/kubelet/pods/45ff2ef8-aff5-4df7-8b70-6cea9e0b9332/volumes" Oct 31 04:30:32.469813 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:32.469787 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-e37d27-75bf88d5c4-mxc78" event={"ID":"216d467a-46f9-4fbe-ab89-899e7d9d9f60","Type":"ContainerStarted","Data":"a1c7a967629979edfb4ff8b47ac615baf1e0c1b9d763deaf69110bde5ca9994c"} Oct 31 04:30:32.469951 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:32.469903 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-e37d27-75bf88d5c4-mxc78" Oct 31 04:30:32.523594 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:32.523544 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-e37d27-75bf88d5c4-mxc78" podStartSLOduration=2.133922194 podStartE2EDuration="2.523527547s" totalImagesPullingTime="389.605353ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:30:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:30:32.522631118 +0000 UTC m=+1580.236965753" watchObservedRunningTime="2026-10-31 04:30:32.523527547 +0000 UTC m=+1580.237862166" Oct 31 04:30:38.544352 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:38.542719 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-e37d27-75bf88d5c4-mxc78" Oct 31 04:30:41.001683 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:30:41.001647 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 04:30:53.416688 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:53.416654 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-7cf9ef-59747bbd5-tbh6h"] Oct 31 04:30:53.417202 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:53.416915 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7cf9ef-59747bbd5-tbh6h" podUID="125f6f3a-41d5-48c3-bbe1-0d52682b0d11" containerName="maas-api" containerID="cri-o://d0bd93308eb9d0b0b2ec78c726416696b000c760fadfca2aae031a087702be20" gracePeriod=30 Oct 31 04:30:53.481718 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:53.481677 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7cf9ef-59747bbd5-tbh6h" podUID="125f6f3a-41d5-48c3-bbe1-0d52682b0d11" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.133:8443/health\": dial tcp 10.133.0.133:8443: connect: connection refused" Oct 31 04:30:53.749350 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:53.747913 4376 generic.go:356] "Generic (PLEG): container finished" podID="125f6f3a-41d5-48c3-bbe1-0d52682b0d11" containerID="d0bd93308eb9d0b0b2ec78c726416696b000c760fadfca2aae031a087702be20" exitCode=0 Oct 31 04:30:53.749350 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:53.748211 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7cf9ef-59747bbd5-tbh6h" event={"ID":"125f6f3a-41d5-48c3-bbe1-0d52682b0d11","Type":"ContainerDied","Data":"d0bd93308eb9d0b0b2ec78c726416696b000c760fadfca2aae031a087702be20"} Oct 31 04:30:53.749350 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:53.748293 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7cf9ef-59747bbd5-tbh6h" event={"ID":"125f6f3a-41d5-48c3-bbe1-0d52682b0d11","Type":"ContainerDied","Data":"70f6b6a2068b7983f787a9225a8fa1dc859445633a5992d04d444f4de3f6cd44"} Oct 31 04:30:53.749350 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:53.748317 4376 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="70f6b6a2068b7983f787a9225a8fa1dc859445633a5992d04d444f4de3f6cd44" Oct 31 04:30:53.765020 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:53.763437 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7cf9ef-59747bbd5-tbh6h" Oct 31 04:30:53.904151 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:53.904122 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/125f6f3a-41d5-48c3-bbe1-0d52682b0d11-metrics-certs\" (UniqueName: \"kubernetes.io/secret/125f6f3a-41d5-48c3-bbe1-0d52682b0d11-metrics-certs\") pod \"125f6f3a-41d5-48c3-bbe1-0d52682b0d11\" (UID: \"125f6f3a-41d5-48c3-bbe1-0d52682b0d11\") " Oct 31 04:30:53.904358 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:53.904168 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/125f6f3a-41d5-48c3-bbe1-0d52682b0d11-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/125f6f3a-41d5-48c3-bbe1-0d52682b0d11-maas-api-tls\") pod \"125f6f3a-41d5-48c3-bbe1-0d52682b0d11\" (UID: \"125f6f3a-41d5-48c3-bbe1-0d52682b0d11\") " Oct 31 04:30:53.904358 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:53.904195 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/125f6f3a-41d5-48c3-bbe1-0d52682b0d11-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/125f6f3a-41d5-48c3-bbe1-0d52682b0d11-openshift-service-ca\") pod \"125f6f3a-41d5-48c3-bbe1-0d52682b0d11\" (UID: \"125f6f3a-41d5-48c3-bbe1-0d52682b0d11\") " Oct 31 04:30:53.904358 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:53.904312 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/125f6f3a-41d5-48c3-bbe1-0d52682b0d11-kube-api-access-zrdc4\" (UniqueName: \"kubernetes.io/projected/125f6f3a-41d5-48c3-bbe1-0d52682b0d11-kube-api-access-zrdc4\") pod \"125f6f3a-41d5-48c3-bbe1-0d52682b0d11\" (UID: \"125f6f3a-41d5-48c3-bbe1-0d52682b0d11\") " Oct 31 04:30:53.904612 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:53.904580 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/125f6f3a-41d5-48c3-bbe1-0d52682b0d11-openshift-service-ca" pod "125f6f3a-41d5-48c3-bbe1-0d52682b0d11" (UID: "125f6f3a-41d5-48c3-bbe1-0d52682b0d11"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:30:53.904967 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:53.904944 4376 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/125f6f3a-41d5-48c3-bbe1-0d52682b0d11-openshift-service-ca\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:30:53.906311 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:53.906288 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/125f6f3a-41d5-48c3-bbe1-0d52682b0d11-metrics-certs" pod "125f6f3a-41d5-48c3-bbe1-0d52682b0d11" (UID: "125f6f3a-41d5-48c3-bbe1-0d52682b0d11"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:30:53.906463 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:53.906445 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/125f6f3a-41d5-48c3-bbe1-0d52682b0d11-kube-api-access-zrdc4" pod "125f6f3a-41d5-48c3-bbe1-0d52682b0d11" (UID: "125f6f3a-41d5-48c3-bbe1-0d52682b0d11"). InnerVolumeSpecName "kube-api-access-zrdc4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:30:53.906642 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:53.906615 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/125f6f3a-41d5-48c3-bbe1-0d52682b0d11-maas-api-tls" pod "125f6f3a-41d5-48c3-bbe1-0d52682b0d11" (UID: "125f6f3a-41d5-48c3-bbe1-0d52682b0d11"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:30:54.017416 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:54.016526 4376 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/125f6f3a-41d5-48c3-bbe1-0d52682b0d11-metrics-certs\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:30:54.017416 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:54.016564 4376 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/125f6f3a-41d5-48c3-bbe1-0d52682b0d11-maas-api-tls\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:30:54.017416 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:54.016579 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrdc4\" (UniqueName: \"kubernetes.io/projected/125f6f3a-41d5-48c3-bbe1-0d52682b0d11-kube-api-access-zrdc4\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:30:54.755702 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:54.755625 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7cf9ef-59747bbd5-tbh6h" Oct 31 04:30:54.808968 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:54.808932 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-7cf9ef-59747bbd5-tbh6h"] Oct 31 04:30:54.821467 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:54.821444 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-7cf9ef-59747bbd5-tbh6h"] Oct 31 04:30:56.443979 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:56.443944 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="125f6f3a-41d5-48c3-bbe1-0d52682b0d11" path="/var/lib/kubelet/pods/125f6f3a-41d5-48c3-bbe1-0d52682b0d11/volumes" Oct 31 04:30:58.887368 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:30:58.887318 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 04:31:04.576277 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:04.576241 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-e6a0de-5b64759d6c-qqgln"] Oct 31 04:31:04.577036 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:04.577013 4376 cpu_manager.go:414] "containerMap: removing container" podUID="25409dbd-265a-4852-9b66-a7689871e139" containerName="payload-pre-processing" Oct 31 04:31:04.577138 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:04.577036 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25409dbd-265a-4852-9b66-a7689871e139" containerName="payload-pre-processing" Oct 31 04:31:04.577138 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:04.577073 4376 cpu_manager.go:414] "containerMap: removing container" podUID="ed72552d-53d8-43a8-8923-0098ab2a192b" containerName="cleanup" Oct 31 04:31:04.577138 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:04.577081 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed72552d-53d8-43a8-8923-0098ab2a192b" containerName="cleanup" Oct 31 04:31:04.577138 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:04.577120 4376 cpu_manager.go:414] "containerMap: removing container" podUID="125f6f3a-41d5-48c3-bbe1-0d52682b0d11" containerName="maas-api" Oct 31 04:31:04.577138 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:04.577127 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="125f6f3a-41d5-48c3-bbe1-0d52682b0d11" containerName="maas-api" Oct 31 04:31:04.577412 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:04.577143 4376 cpu_manager.go:414] "containerMap: removing container" podUID="45ff2ef8-aff5-4df7-8b70-6cea9e0b9332" containerName="payload-processing" Oct 31 04:31:04.577412 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:04.577152 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45ff2ef8-aff5-4df7-8b70-6cea9e0b9332" containerName="payload-processing" Oct 31 04:31:04.577412 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:04.577280 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="125f6f3a-41d5-48c3-bbe1-0d52682b0d11" containerName="maas-api" Oct 31 04:31:04.577412 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:04.577295 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="45ff2ef8-aff5-4df7-8b70-6cea9e0b9332" containerName="payload-processing" Oct 31 04:31:04.577412 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:04.577307 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="25409dbd-265a-4852-9b66-a7689871e139" containerName="payload-pre-processing" Oct 31 04:31:04.580845 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:04.580826 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-e6a0de-5b64759d6c-qqgln" Oct 31 04:31:04.601359 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:04.601312 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-e6a0de-5b64759d6c-qqgln"] Oct 31 04:31:04.609607 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:04.609586 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-e6a0de\"" Oct 31 04:31:04.640583 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:04.640555 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-e6a0de-5987ffc46c-4mxw7"] Oct 31 04:31:04.644460 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:04.644432 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e6a0de-5987ffc46c-4mxw7" Oct 31 04:31:04.659946 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:04.659926 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-e6a0de-dockercfg-2p4pv\"" Oct 31 04:31:04.662299 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:04.662274 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-e6a0de-5987ffc46c-4mxw7"] Oct 31 04:31:04.667938 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:04.667890 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-km5f6\" (UniqueName: \"kubernetes.io/projected/dfd97100-7d93-4575-bb69-43e6f8f462d9-kube-api-access-km5f6\") pod \"payload-processing-e2e-praxis-b-e6a0de-5b64759d6c-qqgln\" (UID: \"dfd97100-7d93-4575-bb69-43e6f8f462d9\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-e6a0de-5b64759d6c-qqgln" Oct 31 04:31:04.668208 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:04.668040 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a3b2925-c12d-4014-9781-067ed8da9b1e-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-e6a0de-5987ffc46c-4mxw7\" (UID: \"7a3b2925-c12d-4014-9781-067ed8da9b1e\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e6a0de-5987ffc46c-4mxw7" Oct 31 04:31:04.668604 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:04.668580 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svd9w\" (UniqueName: \"kubernetes.io/projected/7a3b2925-c12d-4014-9781-067ed8da9b1e-kube-api-access-svd9w\") pod \"payload-pre-processing-e2e-praxis-b-e6a0de-5987ffc46c-4mxw7\" (UID: \"7a3b2925-c12d-4014-9781-067ed8da9b1e\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e6a0de-5987ffc46c-4mxw7" Oct 31 04:31:04.668705 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:04.668630 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfd97100-7d93-4575-bb69-43e6f8f462d9-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-e6a0de-5b64759d6c-qqgln\" (UID: \"dfd97100-7d93-4575-bb69-43e6f8f462d9\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-e6a0de-5b64759d6c-qqgln" Oct 31 04:31:04.778353 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:04.776200 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-km5f6\" (UniqueName: \"kubernetes.io/projected/dfd97100-7d93-4575-bb69-43e6f8f462d9-kube-api-access-km5f6\") pod \"payload-processing-e2e-praxis-b-e6a0de-5b64759d6c-qqgln\" (UID: \"dfd97100-7d93-4575-bb69-43e6f8f462d9\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-e6a0de-5b64759d6c-qqgln" Oct 31 04:31:04.778353 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:04.776395 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a3b2925-c12d-4014-9781-067ed8da9b1e-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-e6a0de-5987ffc46c-4mxw7\" (UID: \"7a3b2925-c12d-4014-9781-067ed8da9b1e\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e6a0de-5987ffc46c-4mxw7" Oct 31 04:31:04.778353 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:04.776457 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svd9w\" (UniqueName: \"kubernetes.io/projected/7a3b2925-c12d-4014-9781-067ed8da9b1e-kube-api-access-svd9w\") pod \"payload-pre-processing-e2e-praxis-b-e6a0de-5987ffc46c-4mxw7\" (UID: \"7a3b2925-c12d-4014-9781-067ed8da9b1e\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e6a0de-5987ffc46c-4mxw7" Oct 31 04:31:04.778353 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:04.776499 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfd97100-7d93-4575-bb69-43e6f8f462d9-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-e6a0de-5b64759d6c-qqgln\" (UID: \"dfd97100-7d93-4575-bb69-43e6f8f462d9\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-e6a0de-5b64759d6c-qqgln" Oct 31 04:31:04.778353 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:04.777483 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfd97100-7d93-4575-bb69-43e6f8f462d9-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-e6a0de-5b64759d6c-qqgln\" (UID: \"dfd97100-7d93-4575-bb69-43e6f8f462d9\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-e6a0de-5b64759d6c-qqgln" Oct 31 04:31:04.778770 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:04.778550 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a3b2925-c12d-4014-9781-067ed8da9b1e-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-e6a0de-5987ffc46c-4mxw7\" (UID: \"7a3b2925-c12d-4014-9781-067ed8da9b1e\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e6a0de-5987ffc46c-4mxw7" Oct 31 04:31:04.845155 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:04.845014 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-svd9w\" (UniqueName: \"kubernetes.io/projected/7a3b2925-c12d-4014-9781-067ed8da9b1e-kube-api-access-svd9w\") pod \"payload-pre-processing-e2e-praxis-b-e6a0de-5987ffc46c-4mxw7\" (UID: \"7a3b2925-c12d-4014-9781-067ed8da9b1e\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e6a0de-5987ffc46c-4mxw7" Oct 31 04:31:04.852430 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:04.845962 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-km5f6\" (UniqueName: \"kubernetes.io/projected/dfd97100-7d93-4575-bb69-43e6f8f462d9-kube-api-access-km5f6\") pod \"payload-processing-e2e-praxis-b-e6a0de-5b64759d6c-qqgln\" (UID: \"dfd97100-7d93-4575-bb69-43e6f8f462d9\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-e6a0de-5b64759d6c-qqgln" Oct 31 04:31:04.902938 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:04.902905 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-e6a0de-5b64759d6c-qqgln" Oct 31 04:31:04.958114 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:04.958086 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e6a0de-5987ffc46c-4mxw7" Oct 31 04:31:05.203449 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:31:05.203417 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfd97100_7d93_4575_bb69_43e6f8f462d9.slice/crio-c107a3087bb1266bce0eb83826fb841b1f5bb8caf73f03533aab30c3dc50525e WatchSource:0}: Error finding container c107a3087bb1266bce0eb83826fb841b1f5bb8caf73f03533aab30c3dc50525e: Status 404 returned error can't find the container with id c107a3087bb1266bce0eb83826fb841b1f5bb8caf73f03533aab30c3dc50525e Oct 31 04:31:05.204615 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:05.204589 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-e6a0de-5b64759d6c-qqgln"] Oct 31 04:31:05.288745 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:31:05.288712 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a3b2925_c12d_4014_9781_067ed8da9b1e.slice/crio-b1d60d5364e2ed367892fb8e31aab83de42946734733a4d923ab9fdd775d8e1c WatchSource:0}: Error finding container b1d60d5364e2ed367892fb8e31aab83de42946734733a4d923ab9fdd775d8e1c: Status 404 returned error can't find the container with id b1d60d5364e2ed367892fb8e31aab83de42946734733a4d923ab9fdd775d8e1c Oct 31 04:31:05.289886 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:05.289862 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-e6a0de-5987ffc46c-4mxw7"] Oct 31 04:31:05.883787 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:05.883748 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-e6a0de-5b64759d6c-qqgln" event={"ID":"dfd97100-7d93-4575-bb69-43e6f8f462d9","Type":"ContainerStarted","Data":"2803e9d05fc94dbd96f6099e2e8708abfc95a19535bdf425791ad77c9cb565fc"} Oct 31 04:31:05.884201 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:05.883811 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-e6a0de-5b64759d6c-qqgln" event={"ID":"dfd97100-7d93-4575-bb69-43e6f8f462d9","Type":"ContainerStarted","Data":"c107a3087bb1266bce0eb83826fb841b1f5bb8caf73f03533aab30c3dc50525e"} Oct 31 04:31:05.884201 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:05.883933 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-e6a0de-5b64759d6c-qqgln" Oct 31 04:31:05.885387 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:05.885361 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e6a0de-5987ffc46c-4mxw7" event={"ID":"7a3b2925-c12d-4014-9781-067ed8da9b1e","Type":"ContainerStarted","Data":"0dc2d5283b97c25821054f79c4b38d4620e0db700ce79ba4c6e6b427037aeff7"} Oct 31 04:31:05.885521 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:05.885397 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e6a0de-5987ffc46c-4mxw7" event={"ID":"7a3b2925-c12d-4014-9781-067ed8da9b1e","Type":"ContainerStarted","Data":"b1d60d5364e2ed367892fb8e31aab83de42946734733a4d923ab9fdd775d8e1c"} Oct 31 04:31:05.885521 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:05.885419 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e6a0de-5987ffc46c-4mxw7" Oct 31 04:31:05.926488 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:05.926441 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-e6a0de-5b64759d6c-qqgln" podStartSLOduration=1.926428782 podStartE2EDuration="1.926428782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:31:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:31:05.924704545 +0000 UTC m=+1613.639039162" watchObservedRunningTime="2026-10-31 04:31:05.926428782 +0000 UTC m=+1613.640763398" Oct 31 04:31:05.967149 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:05.967094 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e6a0de-5987ffc46c-4mxw7" podStartSLOduration=1.9670792590000001 podStartE2EDuration="1.967079259s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:31:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:31:05.96545878 +0000 UTC m=+1613.679793394" watchObservedRunningTime="2026-10-31 04:31:05.967079259 +0000 UTC m=+1613.681413883" Oct 31 04:31:06.929476 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:06.929440 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-d5gfx"] Oct 31 04:31:06.934082 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:06.934060 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-d5gfx" Oct 31 04:31:06.952678 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:06.952654 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-9x5tt"] Oct 31 04:31:06.957170 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:06.957111 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-9x5tt" Oct 31 04:31:06.976260 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:06.976209 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-d5gfx"] Oct 31 04:31:06.983648 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:06.983621 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-9x5tt"] Oct 31 04:31:07.014290 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:07.014260 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d2c5af6-838a-4bb6-bec8-8803bbb03c22-istiod-ca-cert\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-d5gfx\" (UID: \"4d2c5af6-838a-4bb6-bec8-8803bbb03c22\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-d5gfx" Oct 31 04:31:07.014474 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:07.014451 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d2c5af6-838a-4bb6-bec8-8803bbb03c22-istio-token\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-d5gfx\" (UID: \"4d2c5af6-838a-4bb6-bec8-8803bbb03c22\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-d5gfx" Oct 31 04:31:07.014551 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:07.014504 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d2c5af6-838a-4bb6-bec8-8803bbb03c22-istio-podinfo\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-d5gfx\" (UID: \"4d2c5af6-838a-4bb6-bec8-8803bbb03c22\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-d5gfx" Oct 31 04:31:07.014646 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:07.014623 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d2c5af6-838a-4bb6-bec8-8803bbb03c22-istio-envoy\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-d5gfx\" (UID: \"4d2c5af6-838a-4bb6-bec8-8803bbb03c22\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-d5gfx" Oct 31 04:31:07.014815 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:07.014769 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4dtb\" (UniqueName: \"kubernetes.io/projected/4d2c5af6-838a-4bb6-bec8-8803bbb03c22-kube-api-access-m4dtb\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-d5gfx\" (UID: \"4d2c5af6-838a-4bb6-bec8-8803bbb03c22\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-d5gfx" Oct 31 04:31:07.014940 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:07.014891 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d2c5af6-838a-4bb6-bec8-8803bbb03c22-credential-socket\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-d5gfx\" (UID: \"4d2c5af6-838a-4bb6-bec8-8803bbb03c22\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-d5gfx" Oct 31 04:31:07.015074 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:07.015056 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d2c5af6-838a-4bb6-bec8-8803bbb03c22-workload-certs\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-d5gfx\" (UID: \"4d2c5af6-838a-4bb6-bec8-8803bbb03c22\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-d5gfx" Oct 31 04:31:07.015170 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:07.015082 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d2c5af6-838a-4bb6-bec8-8803bbb03c22-istio-data\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-d5gfx\" (UID: \"4d2c5af6-838a-4bb6-bec8-8803bbb03c22\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-d5gfx" Oct 31 04:31:07.015234 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:07.015166 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d2c5af6-838a-4bb6-bec8-8803bbb03c22-workload-socket\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-d5gfx\" (UID: \"4d2c5af6-838a-4bb6-bec8-8803bbb03c22\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-d5gfx" Oct 31 04:31:07.116674 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:07.116628 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b61c092a-60dc-4bca-b24d-aa63f10dbf14-istiod-ca-cert\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-9x5tt\" (UID: \"b61c092a-60dc-4bca-b24d-aa63f10dbf14\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-9x5tt" Oct 31 04:31:07.116884 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:07.116730 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b61c092a-60dc-4bca-b24d-aa63f10dbf14-istio-podinfo\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-9x5tt\" (UID: \"b61c092a-60dc-4bca-b24d-aa63f10dbf14\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-9x5tt" Oct 31 04:31:07.116884 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:07.116767 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4dtb\" (UniqueName: \"kubernetes.io/projected/4d2c5af6-838a-4bb6-bec8-8803bbb03c22-kube-api-access-m4dtb\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-d5gfx\" (UID: \"4d2c5af6-838a-4bb6-bec8-8803bbb03c22\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-d5gfx" Oct 31 04:31:07.116884 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:07.116796 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b61c092a-60dc-4bca-b24d-aa63f10dbf14-workload-certs\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-9x5tt\" (UID: \"b61c092a-60dc-4bca-b24d-aa63f10dbf14\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-9x5tt" Oct 31 04:31:07.117092 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:07.116987 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d2c5af6-838a-4bb6-bec8-8803bbb03c22-credential-socket\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-d5gfx\" (UID: \"4d2c5af6-838a-4bb6-bec8-8803bbb03c22\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-d5gfx" Oct 31 04:31:07.117092 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:07.117025 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b61c092a-60dc-4bca-b24d-aa63f10dbf14-credential-socket\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-9x5tt\" (UID: \"b61c092a-60dc-4bca-b24d-aa63f10dbf14\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-9x5tt" Oct 31 04:31:07.117092 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:07.117056 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d2c5af6-838a-4bb6-bec8-8803bbb03c22-workload-certs\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-d5gfx\" (UID: \"4d2c5af6-838a-4bb6-bec8-8803bbb03c22\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-d5gfx" Oct 31 04:31:07.117262 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:07.117096 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d2c5af6-838a-4bb6-bec8-8803bbb03c22-istio-data\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-d5gfx\" (UID: \"4d2c5af6-838a-4bb6-bec8-8803bbb03c22\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-d5gfx" Oct 31 04:31:07.117262 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:07.117127 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b61c092a-60dc-4bca-b24d-aa63f10dbf14-istio-token\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-9x5tt\" (UID: \"b61c092a-60dc-4bca-b24d-aa63f10dbf14\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-9x5tt" Oct 31 04:31:07.117262 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:07.117161 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b61c092a-60dc-4bca-b24d-aa63f10dbf14-istio-data\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-9x5tt\" (UID: \"b61c092a-60dc-4bca-b24d-aa63f10dbf14\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-9x5tt" Oct 31 04:31:07.117262 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:07.117191 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ns2l6\" (UniqueName: \"kubernetes.io/projected/b61c092a-60dc-4bca-b24d-aa63f10dbf14-kube-api-access-ns2l6\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-9x5tt\" (UID: \"b61c092a-60dc-4bca-b24d-aa63f10dbf14\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-9x5tt" Oct 31 04:31:07.117262 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:07.117252 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d2c5af6-838a-4bb6-bec8-8803bbb03c22-workload-socket\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-d5gfx\" (UID: \"4d2c5af6-838a-4bb6-bec8-8803bbb03c22\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-d5gfx" Oct 31 04:31:07.117557 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:07.117345 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b61c092a-60dc-4bca-b24d-aa63f10dbf14-workload-socket\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-9x5tt\" (UID: \"b61c092a-60dc-4bca-b24d-aa63f10dbf14\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-9x5tt" Oct 31 04:31:07.117557 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:07.117392 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d2c5af6-838a-4bb6-bec8-8803bbb03c22-istiod-ca-cert\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-d5gfx\" (UID: \"4d2c5af6-838a-4bb6-bec8-8803bbb03c22\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-d5gfx" Oct 31 04:31:07.117557 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:07.117421 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d2c5af6-838a-4bb6-bec8-8803bbb03c22-istio-data\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-d5gfx\" (UID: \"4d2c5af6-838a-4bb6-bec8-8803bbb03c22\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-d5gfx" Oct 31 04:31:07.117557 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:07.117464 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d2c5af6-838a-4bb6-bec8-8803bbb03c22-credential-socket\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-d5gfx\" (UID: \"4d2c5af6-838a-4bb6-bec8-8803bbb03c22\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-d5gfx" Oct 31 04:31:07.117557 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:07.117505 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d2c5af6-838a-4bb6-bec8-8803bbb03c22-istio-token\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-d5gfx\" (UID: \"4d2c5af6-838a-4bb6-bec8-8803bbb03c22\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-d5gfx" Oct 31 04:31:07.117557 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:07.117503 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d2c5af6-838a-4bb6-bec8-8803bbb03c22-workload-certs\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-d5gfx\" (UID: \"4d2c5af6-838a-4bb6-bec8-8803bbb03c22\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-d5gfx" Oct 31 04:31:07.117910 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:07.117699 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d2c5af6-838a-4bb6-bec8-8803bbb03c22-workload-socket\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-d5gfx\" (UID: \"4d2c5af6-838a-4bb6-bec8-8803bbb03c22\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-d5gfx" Oct 31 04:31:07.117910 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:07.117768 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d2c5af6-838a-4bb6-bec8-8803bbb03c22-istio-podinfo\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-d5gfx\" (UID: \"4d2c5af6-838a-4bb6-bec8-8803bbb03c22\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-d5gfx" Oct 31 04:31:07.117910 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:07.117794 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d2c5af6-838a-4bb6-bec8-8803bbb03c22-istio-envoy\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-d5gfx\" (UID: \"4d2c5af6-838a-4bb6-bec8-8803bbb03c22\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-d5gfx" Oct 31 04:31:07.117910 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:07.117830 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b61c092a-60dc-4bca-b24d-aa63f10dbf14-istio-envoy\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-9x5tt\" (UID: \"b61c092a-60dc-4bca-b24d-aa63f10dbf14\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-9x5tt" Oct 31 04:31:07.118123 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:07.118068 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d2c5af6-838a-4bb6-bec8-8803bbb03c22-istiod-ca-cert\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-d5gfx\" (UID: \"4d2c5af6-838a-4bb6-bec8-8803bbb03c22\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-d5gfx" Oct 31 04:31:07.120081 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:07.120057 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d2c5af6-838a-4bb6-bec8-8803bbb03c22-istio-envoy\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-d5gfx\" (UID: \"4d2c5af6-838a-4bb6-bec8-8803bbb03c22\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-d5gfx" Oct 31 04:31:07.120256 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:07.120236 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d2c5af6-838a-4bb6-bec8-8803bbb03c22-istio-podinfo\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-d5gfx\" (UID: \"4d2c5af6-838a-4bb6-bec8-8803bbb03c22\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-d5gfx" Oct 31 04:31:07.140268 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:07.140236 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4dtb\" (UniqueName: \"kubernetes.io/projected/4d2c5af6-838a-4bb6-bec8-8803bbb03c22-kube-api-access-m4dtb\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-d5gfx\" (UID: \"4d2c5af6-838a-4bb6-bec8-8803bbb03c22\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-d5gfx" Oct 31 04:31:07.140373 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:07.140267 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d2c5af6-838a-4bb6-bec8-8803bbb03c22-istio-token\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-d5gfx\" (UID: \"4d2c5af6-838a-4bb6-bec8-8803bbb03c22\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-d5gfx" Oct 31 04:31:07.219011 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:07.218955 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b61c092a-60dc-4bca-b24d-aa63f10dbf14-istiod-ca-cert\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-9x5tt\" (UID: \"b61c092a-60dc-4bca-b24d-aa63f10dbf14\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-9x5tt" Oct 31 04:31:07.219011 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:07.218991 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b61c092a-60dc-4bca-b24d-aa63f10dbf14-istio-podinfo\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-9x5tt\" (UID: \"b61c092a-60dc-4bca-b24d-aa63f10dbf14\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-9x5tt" Oct 31 04:31:07.219160 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:07.219015 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b61c092a-60dc-4bca-b24d-aa63f10dbf14-workload-certs\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-9x5tt\" (UID: \"b61c092a-60dc-4bca-b24d-aa63f10dbf14\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-9x5tt" Oct 31 04:31:07.219160 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:07.219055 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b61c092a-60dc-4bca-b24d-aa63f10dbf14-credential-socket\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-9x5tt\" (UID: \"b61c092a-60dc-4bca-b24d-aa63f10dbf14\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-9x5tt" Oct 31 04:31:07.219160 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:07.219079 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b61c092a-60dc-4bca-b24d-aa63f10dbf14-istio-token\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-9x5tt\" (UID: \"b61c092a-60dc-4bca-b24d-aa63f10dbf14\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-9x5tt" Oct 31 04:31:07.219160 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:07.219097 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b61c092a-60dc-4bca-b24d-aa63f10dbf14-istio-data\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-9x5tt\" (UID: \"b61c092a-60dc-4bca-b24d-aa63f10dbf14\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-9x5tt" Oct 31 04:31:07.219160 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:07.219125 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ns2l6\" (UniqueName: \"kubernetes.io/projected/b61c092a-60dc-4bca-b24d-aa63f10dbf14-kube-api-access-ns2l6\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-9x5tt\" (UID: \"b61c092a-60dc-4bca-b24d-aa63f10dbf14\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-9x5tt" Oct 31 04:31:07.219430 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:07.219214 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b61c092a-60dc-4bca-b24d-aa63f10dbf14-workload-socket\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-9x5tt\" (UID: \"b61c092a-60dc-4bca-b24d-aa63f10dbf14\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-9x5tt" Oct 31 04:31:07.219430 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:07.219409 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b61c092a-60dc-4bca-b24d-aa63f10dbf14-istio-envoy\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-9x5tt\" (UID: \"b61c092a-60dc-4bca-b24d-aa63f10dbf14\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-9x5tt" Oct 31 04:31:07.219612 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:07.219585 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b61c092a-60dc-4bca-b24d-aa63f10dbf14-istio-data\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-9x5tt\" (UID: \"b61c092a-60dc-4bca-b24d-aa63f10dbf14\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-9x5tt" Oct 31 04:31:07.219672 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:07.219622 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b61c092a-60dc-4bca-b24d-aa63f10dbf14-credential-socket\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-9x5tt\" (UID: \"b61c092a-60dc-4bca-b24d-aa63f10dbf14\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-9x5tt" Oct 31 04:31:07.219757 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:07.219737 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b61c092a-60dc-4bca-b24d-aa63f10dbf14-workload-socket\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-9x5tt\" (UID: \"b61c092a-60dc-4bca-b24d-aa63f10dbf14\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-9x5tt" Oct 31 04:31:07.219810 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:07.219794 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b61c092a-60dc-4bca-b24d-aa63f10dbf14-istiod-ca-cert\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-9x5tt\" (UID: \"b61c092a-60dc-4bca-b24d-aa63f10dbf14\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-9x5tt" Oct 31 04:31:07.219851 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:07.219830 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b61c092a-60dc-4bca-b24d-aa63f10dbf14-workload-certs\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-9x5tt\" (UID: \"b61c092a-60dc-4bca-b24d-aa63f10dbf14\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-9x5tt" Oct 31 04:31:07.221357 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:07.221322 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b61c092a-60dc-4bca-b24d-aa63f10dbf14-istio-podinfo\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-9x5tt\" (UID: \"b61c092a-60dc-4bca-b24d-aa63f10dbf14\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-9x5tt" Oct 31 04:31:07.221446 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:07.221409 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b61c092a-60dc-4bca-b24d-aa63f10dbf14-istio-envoy\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-9x5tt\" (UID: \"b61c092a-60dc-4bca-b24d-aa63f10dbf14\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-9x5tt" Oct 31 04:31:07.242988 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:07.242963 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b61c092a-60dc-4bca-b24d-aa63f10dbf14-istio-token\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-9x5tt\" (UID: \"b61c092a-60dc-4bca-b24d-aa63f10dbf14\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-9x5tt" Oct 31 04:31:07.243170 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:07.243152 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ns2l6\" (UniqueName: \"kubernetes.io/projected/b61c092a-60dc-4bca-b24d-aa63f10dbf14-kube-api-access-ns2l6\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-9x5tt\" (UID: \"b61c092a-60dc-4bca-b24d-aa63f10dbf14\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-9x5tt" Oct 31 04:31:07.251194 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:07.251176 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-d5gfx" Oct 31 04:31:07.286126 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:07.286099 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-9x5tt" Oct 31 04:31:07.565417 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:31:07.565389 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d2c5af6_838a_4bb6_bec8_8803bbb03c22.slice/crio-c04f735da9f18ed1c55cac7e514a3d8bf6c00a3128eb25842cf67fd7629de22c WatchSource:0}: Error finding container c04f735da9f18ed1c55cac7e514a3d8bf6c00a3128eb25842cf67fd7629de22c: Status 404 returned error can't find the container with id c04f735da9f18ed1c55cac7e514a3d8bf6c00a3128eb25842cf67fd7629de22c Oct 31 04:31:07.567999 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:07.567972 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-d5gfx"] Oct 31 04:31:07.568677 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:07.568644 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:31:07.568742 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:07.568711 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:31:07.568788 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:07.568742 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:31:07.604142 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:31:07.604118 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb61c092a_60dc_4bca_b24d_aa63f10dbf14.slice/crio-6627681a17954fa1a19dac8f0c16173849ac80f07cfb33bae81320fd2e30e67c WatchSource:0}: Error finding container 6627681a17954fa1a19dac8f0c16173849ac80f07cfb33bae81320fd2e30e67c: Status 404 returned error can't find the container with id 6627681a17954fa1a19dac8f0c16173849ac80f07cfb33bae81320fd2e30e67c Oct 31 04:31:07.605723 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:07.605699 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-9x5tt"] Oct 31 04:31:07.606815 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:07.606784 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:31:07.606880 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:07.606848 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:31:07.606925 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:07.606878 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:31:07.903358 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:07.903278 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-9x5tt" event={"ID":"b61c092a-60dc-4bca-b24d-aa63f10dbf14","Type":"ContainerStarted","Data":"3563a5c957fb617b8bac32d8eaf616f37ceda13f239d12a496f054b98d6d9613"} Oct 31 04:31:07.903553 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:07.903379 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-9x5tt" event={"ID":"b61c092a-60dc-4bca-b24d-aa63f10dbf14","Type":"ContainerStarted","Data":"6627681a17954fa1a19dac8f0c16173849ac80f07cfb33bae81320fd2e30e67c"} Oct 31 04:31:07.905315 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:07.905279 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-d5gfx" event={"ID":"4d2c5af6-838a-4bb6-bec8-8803bbb03c22","Type":"ContainerStarted","Data":"4a2b8b2ebb49be8a92bbb62db2239c1c63aed46c083c7407995c1a3d273240e9"} Oct 31 04:31:07.905470 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:07.905372 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-d5gfx" event={"ID":"4d2c5af6-838a-4bb6-bec8-8803bbb03c22","Type":"ContainerStarted","Data":"c04f735da9f18ed1c55cac7e514a3d8bf6c00a3128eb25842cf67fd7629de22c"} Oct 31 04:31:07.951803 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:07.951753 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-9x5tt" podStartSLOduration=1.951738866 podStartE2EDuration="1.951738866s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:31:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:31:07.949774436 +0000 UTC m=+1615.664109054" watchObservedRunningTime="2026-10-31 04:31:07.951738866 +0000 UTC m=+1615.666073483" Oct 31 04:31:08.003845 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:08.003795 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-d5gfx" podStartSLOduration=2.003780575 podStartE2EDuration="2.003780575s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:31:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:31:08.000533644 +0000 UTC m=+1615.714868288" watchObservedRunningTime="2026-10-31 04:31:08.003780575 +0000 UTC m=+1615.718115191" Oct 31 04:31:08.251735 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:08.251657 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-d5gfx" Oct 31 04:31:08.253552 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:08.253521 4376 patch_prober.go:29] interesting pod/e2e-worker-w2-09a20a-openshift-default-55d8647ff-d5gfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body= Oct 31 04:31:08.253679 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:08.253569 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-d5gfx" podUID="4d2c5af6-838a-4bb6-bec8-8803bbb03c22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" Oct 31 04:31:08.286584 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:08.286552 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-9x5tt" Oct 31 04:31:08.288084 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:08.288058 4376 patch_prober.go:29] interesting pod/e2e-worker-w2-09a20a-openshift-default-55d8647ff-9x5tt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body= Oct 31 04:31:08.288210 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:08.288109 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-9x5tt" podUID="b61c092a-60dc-4bca-b24d-aa63f10dbf14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" Oct 31 04:31:09.263350 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:09.262765 4376 patch_prober.go:29] interesting pod/e2e-worker-w2-09a20a-openshift-default-55d8647ff-d5gfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body= Oct 31 04:31:09.263350 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:09.262830 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-d5gfx" podUID="4d2c5af6-838a-4bb6-bec8-8803bbb03c22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" Oct 31 04:31:09.286954 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:09.286799 4376 patch_prober.go:29] interesting pod/e2e-worker-w2-09a20a-openshift-default-55d8647ff-9x5tt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body= Oct 31 04:31:09.286954 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:09.286859 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-9x5tt" podUID="b61c092a-60dc-4bca-b24d-aa63f10dbf14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" Oct 31 04:31:10.266353 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:10.254689 4376 patch_prober.go:29] interesting pod/e2e-worker-w2-09a20a-openshift-default-55d8647ff-d5gfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body= Oct 31 04:31:10.266353 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:10.254752 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-d5gfx" podUID="4d2c5af6-838a-4bb6-bec8-8803bbb03c22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" Oct 31 04:31:10.289353 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:10.287985 4376 patch_prober.go:29] interesting pod/e2e-worker-w2-09a20a-openshift-default-55d8647ff-9x5tt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body= Oct 31 04:31:10.289353 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:10.288040 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-9x5tt" podUID="b61c092a-60dc-4bca-b24d-aa63f10dbf14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" Oct 31 04:31:10.620402 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:10.611899 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-praxis-a-d1d441/praxis-route-d1d441-kserve-698cd98df-qv4m7"] Oct 31 04:31:11.017749 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:31:11.017671 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 04:31:11.252702 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:11.252668 4376 patch_prober.go:29] interesting pod/e2e-worker-w2-09a20a-openshift-default-55d8647ff-d5gfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body= Oct 31 04:31:11.252981 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:11.252952 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-d5gfx" podUID="4d2c5af6-838a-4bb6-bec8-8803bbb03c22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" Oct 31 04:31:11.287797 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:11.287650 4376 patch_prober.go:29] interesting pod/e2e-worker-w2-09a20a-openshift-default-55d8647ff-9x5tt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body= Oct 31 04:31:11.287797 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:11.287712 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-9x5tt" podUID="b61c092a-60dc-4bca-b24d-aa63f10dbf14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" Oct 31 04:31:12.253353 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:12.251714 4376 patch_prober.go:29] interesting pod/e2e-worker-w2-09a20a-openshift-default-55d8647ff-d5gfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body= Oct 31 04:31:12.253353 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:12.251809 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-d5gfx" podUID="4d2c5af6-838a-4bb6-bec8-8803bbb03c22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" Oct 31 04:31:12.288351 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:12.287713 4376 patch_prober.go:29] interesting pod/e2e-worker-w2-09a20a-openshift-default-55d8647ff-9x5tt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body= Oct 31 04:31:12.288351 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:12.287772 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-9x5tt" podUID="b61c092a-60dc-4bca-b24d-aa63f10dbf14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" Oct 31 04:31:13.259353 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:13.251690 4376 patch_prober.go:29] interesting pod/e2e-worker-w2-09a20a-openshift-default-55d8647ff-d5gfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body= Oct 31 04:31:13.259353 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:13.251763 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-d5gfx" podUID="4d2c5af6-838a-4bb6-bec8-8803bbb03c22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" Oct 31 04:31:13.286748 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:13.286697 4376 patch_prober.go:29] interesting pod/e2e-worker-w2-09a20a-openshift-default-55d8647ff-9x5tt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body= Oct 31 04:31:13.286968 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:13.286757 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-9x5tt" podUID="b61c092a-60dc-4bca-b24d-aa63f10dbf14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" Oct 31 04:31:14.252343 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:14.252289 4376 patch_prober.go:29] interesting pod/e2e-worker-w2-09a20a-openshift-default-55d8647ff-d5gfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body= Oct 31 04:31:14.252851 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:14.252368 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-d5gfx" podUID="4d2c5af6-838a-4bb6-bec8-8803bbb03c22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" Oct 31 04:31:14.286535 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:14.286499 4376 patch_prober.go:29] interesting pod/e2e-worker-w2-09a20a-openshift-default-55d8647ff-9x5tt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body= Oct 31 04:31:14.286801 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:14.286564 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-9x5tt" podUID="b61c092a-60dc-4bca-b24d-aa63f10dbf14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" Oct 31 04:31:15.251780 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:15.251745 4376 patch_prober.go:29] interesting pod/e2e-worker-w2-09a20a-openshift-default-55d8647ff-d5gfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body= Oct 31 04:31:15.251960 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:15.251817 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-d5gfx" podUID="4d2c5af6-838a-4bb6-bec8-8803bbb03c22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" Oct 31 04:31:15.554213 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:15.554126 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-9x5tt" Oct 31 04:31:15.555167 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:15.555145 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-9x5tt" Oct 31 04:31:15.556940 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:15.556918 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-9x5tt" Oct 31 04:31:16.255629 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:16.255553 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-d5gfx" Oct 31 04:31:16.256036 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:16.256021 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-d5gfx" Oct 31 04:31:16.256688 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:16.256666 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-d5gfx" Oct 31 04:31:16.895771 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:16.895745 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e6a0de-5987ffc46c-4mxw7" Oct 31 04:31:16.896273 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:16.895845 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-e6a0de-5b64759d6c-qqgln" Oct 31 04:31:19.707088 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:19.704636 4376 patch_prober.go:29] interesting pod/e2e-worker-w3-4c2387-openshift-default-5b68d44445-rw9sz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 04:31:19.707088 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:19.704715 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-rw9sz" podUID="ead2084a-70a7-47d2-a757-f340d971bfb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 04:31:19.730131 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:19.729980 4376 patch_prober.go:29] interesting pod/e2e-worker-w3-4c2387-openshift-default-5b68d44445-wq9cv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 04:31:19.730131 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:19.730052 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-wq9cv" podUID="17ea2397-450e-4505-9a48-0cf93f76065e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 04:31:19.797474 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:19.797414 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c4006d/simulator-w4-c4006d-kserve-595455485b-4h46q" podUID="576e2c1f-1c80-43a9-a136-3692776642ae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded" Oct 31 04:31:19.958530 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:19.958428 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4c2387/distinct-w3-4c2387-kserve-768bdd84c7-6jmtl" podUID="b2c75691-4cf4-484b-96e2-3c2503b8d78c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:31:19.972880 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:19.972814 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c4006d/distinct-2-w4-c4006d-kserve-776b5b899-kpl6z" podUID="dc53083d-eadd-4cb8-8633-35e02d400b9c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:31:19.990770 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:19.990737 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-09a20a/distinct-2-w2-09a20a-kserve-779d648d75-mshl7" podUID="c81f5d58-c5c7-46c1-b832-c550b092f02b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:31:25.707100 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:25.707036 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-praxis-a-d1d441/praxis-route-d1d441-kserve-698cd98df-qv4m7" podUID="d20ddaa5-bf86-4dfa-9d5a-91fd4894ed34" containerName="main" containerID="cri-o://4697d65bd32a36600078af2510616c6be9c2c4bee91d778dd22f470a8977adec" gracePeriod=45 Oct 31 04:31:25.973080 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:25.973016 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-d1d441/praxis-route-d1d441-kserve-698cd98df-qv4m7" Oct 31 04:31:26.082305 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:26.082279 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d20ddaa5-bf86-4dfa-9d5a-91fd4894ed34-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d20ddaa5-bf86-4dfa-9d5a-91fd4894ed34-dshm\") pod \"d20ddaa5-bf86-4dfa-9d5a-91fd4894ed34\" (UID: \"d20ddaa5-bf86-4dfa-9d5a-91fd4894ed34\") " Oct 31 04:31:26.082484 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:26.082369 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d20ddaa5-bf86-4dfa-9d5a-91fd4894ed34-home\" (UniqueName: \"kubernetes.io/empty-dir/d20ddaa5-bf86-4dfa-9d5a-91fd4894ed34-home\") pod \"d20ddaa5-bf86-4dfa-9d5a-91fd4894ed34\" (UID: \"d20ddaa5-bf86-4dfa-9d5a-91fd4894ed34\") " Oct 31 04:31:26.082484 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:26.082418 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d20ddaa5-bf86-4dfa-9d5a-91fd4894ed34-kube-api-access-4mwq8\" (UniqueName: \"kubernetes.io/projected/d20ddaa5-bf86-4dfa-9d5a-91fd4894ed34-kube-api-access-4mwq8\") pod \"d20ddaa5-bf86-4dfa-9d5a-91fd4894ed34\" (UID: \"d20ddaa5-bf86-4dfa-9d5a-91fd4894ed34\") " Oct 31 04:31:26.082600 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:26.082546 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d20ddaa5-bf86-4dfa-9d5a-91fd4894ed34-tls-certs\" (UniqueName: \"kubernetes.io/secret/d20ddaa5-bf86-4dfa-9d5a-91fd4894ed34-tls-certs\") pod \"d20ddaa5-bf86-4dfa-9d5a-91fd4894ed34\" (UID: \"d20ddaa5-bf86-4dfa-9d5a-91fd4894ed34\") " Oct 31 04:31:26.082661 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:26.082610 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d20ddaa5-bf86-4dfa-9d5a-91fd4894ed34-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d20ddaa5-bf86-4dfa-9d5a-91fd4894ed34-model-cache\") pod \"d20ddaa5-bf86-4dfa-9d5a-91fd4894ed34\" (UID: \"d20ddaa5-bf86-4dfa-9d5a-91fd4894ed34\") " Oct 31 04:31:26.082719 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:26.082655 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d20ddaa5-bf86-4dfa-9d5a-91fd4894ed34-home" pod "d20ddaa5-bf86-4dfa-9d5a-91fd4894ed34" (UID: "d20ddaa5-bf86-4dfa-9d5a-91fd4894ed34"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:31:26.082778 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:26.082735 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d20ddaa5-bf86-4dfa-9d5a-91fd4894ed34-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d20ddaa5-bf86-4dfa-9d5a-91fd4894ed34-tmp-dir\") pod \"d20ddaa5-bf86-4dfa-9d5a-91fd4894ed34\" (UID: \"d20ddaa5-bf86-4dfa-9d5a-91fd4894ed34\") " Oct 31 04:31:26.082897 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:26.082866 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d20ddaa5-bf86-4dfa-9d5a-91fd4894ed34-model-cache" pod "d20ddaa5-bf86-4dfa-9d5a-91fd4894ed34" (UID: "d20ddaa5-bf86-4dfa-9d5a-91fd4894ed34"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:31:26.082975 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:26.082961 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d20ddaa5-bf86-4dfa-9d5a-91fd4894ed34-tmp-dir" pod "d20ddaa5-bf86-4dfa-9d5a-91fd4894ed34" (UID: "d20ddaa5-bf86-4dfa-9d5a-91fd4894ed34"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:31:26.083291 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:26.083270 4376 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d20ddaa5-bf86-4dfa-9d5a-91fd4894ed34-model-cache\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:31:26.083291 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:26.083288 4376 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d20ddaa5-bf86-4dfa-9d5a-91fd4894ed34-tmp-dir\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:31:26.083291 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:26.083297 4376 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d20ddaa5-bf86-4dfa-9d5a-91fd4894ed34-home\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:31:26.084814 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:26.084788 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d20ddaa5-bf86-4dfa-9d5a-91fd4894ed34-kube-api-access-4mwq8" pod "d20ddaa5-bf86-4dfa-9d5a-91fd4894ed34" (UID: "d20ddaa5-bf86-4dfa-9d5a-91fd4894ed34"). InnerVolumeSpecName "kube-api-access-4mwq8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:31:26.084814 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:26.084801 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d20ddaa5-bf86-4dfa-9d5a-91fd4894ed34-tls-certs" pod "d20ddaa5-bf86-4dfa-9d5a-91fd4894ed34" (UID: "d20ddaa5-bf86-4dfa-9d5a-91fd4894ed34"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:31:26.084935 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:26.084856 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d20ddaa5-bf86-4dfa-9d5a-91fd4894ed34-dshm" pod "d20ddaa5-bf86-4dfa-9d5a-91fd4894ed34" (UID: "d20ddaa5-bf86-4dfa-9d5a-91fd4894ed34"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:31:26.184940 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:26.184903 4376 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d20ddaa5-bf86-4dfa-9d5a-91fd4894ed34-dshm\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:31:26.184940 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:26.184934 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mwq8\" (UniqueName: \"kubernetes.io/projected/d20ddaa5-bf86-4dfa-9d5a-91fd4894ed34-kube-api-access-4mwq8\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:31:26.184940 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:26.184949 4376 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d20ddaa5-bf86-4dfa-9d5a-91fd4894ed34-tls-certs\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:31:26.247123 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:26.247033 4376 generic.go:356] "Generic (PLEG): container finished" podID="d20ddaa5-bf86-4dfa-9d5a-91fd4894ed34" containerID="4697d65bd32a36600078af2510616c6be9c2c4bee91d778dd22f470a8977adec" exitCode=0 Oct 31 04:31:26.247123 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:26.247076 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-d1d441/praxis-route-d1d441-kserve-698cd98df-qv4m7" Oct 31 04:31:26.247123 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:26.247111 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-d1d441/praxis-route-d1d441-kserve-698cd98df-qv4m7" event={"ID":"d20ddaa5-bf86-4dfa-9d5a-91fd4894ed34","Type":"ContainerDied","Data":"4697d65bd32a36600078af2510616c6be9c2c4bee91d778dd22f470a8977adec"} Oct 31 04:31:26.247426 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:26.247156 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-d1d441/praxis-route-d1d441-kserve-698cd98df-qv4m7" event={"ID":"d20ddaa5-bf86-4dfa-9d5a-91fd4894ed34","Type":"ContainerDied","Data":"cc24908328282c321dc4860f3312aceb4ccac54eccb301c0ccb74237231f4e55"} Oct 31 04:31:26.247426 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:26.247174 4376 scope.go:122] "RemoveContainer" containerID="4697d65bd32a36600078af2510616c6be9c2c4bee91d778dd22f470a8977adec" Oct 31 04:31:26.257324 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:26.257306 4376 scope.go:122] "RemoveContainer" containerID="4697d65bd32a36600078af2510616c6be9c2c4bee91d778dd22f470a8977adec" Oct 31 04:31:26.257596 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:31:26.257575 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4697d65bd32a36600078af2510616c6be9c2c4bee91d778dd22f470a8977adec\": container with ID starting with 4697d65bd32a36600078af2510616c6be9c2c4bee91d778dd22f470a8977adec not found: ID does not exist" containerID="4697d65bd32a36600078af2510616c6be9c2c4bee91d778dd22f470a8977adec" Oct 31 04:31:26.257662 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:26.257606 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4697d65bd32a36600078af2510616c6be9c2c4bee91d778dd22f470a8977adec"} err="failed to get container status \"4697d65bd32a36600078af2510616c6be9c2c4bee91d778dd22f470a8977adec\": rpc error: code = NotFound desc = could not find container \"4697d65bd32a36600078af2510616c6be9c2c4bee91d778dd22f470a8977adec\": container with ID starting with 4697d65bd32a36600078af2510616c6be9c2c4bee91d778dd22f470a8977adec not found: ID does not exist" Oct 31 04:31:26.301468 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:26.301439 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-praxis-a-d1d441/praxis-route-d1d441-kserve-698cd98df-qv4m7"] Oct 31 04:31:26.313804 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:26.313783 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-praxis-a-d1d441/praxis-route-d1d441-kserve-698cd98df-qv4m7"] Oct 31 04:31:26.444915 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:26.444881 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d20ddaa5-bf86-4dfa-9d5a-91fd4894ed34" path="/var/lib/kubelet/pods/d20ddaa5-bf86-4dfa-9d5a-91fd4894ed34/volumes" Oct 31 04:31:38.092171 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:38.092131 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 04:31:41.123956 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:31:41.123890 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 04:31:55.647731 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:55.647282 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-zs49c"] Oct 31 04:31:55.648256 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:55.648227 4376 cpu_manager.go:414] "containerMap: removing container" podUID="d20ddaa5-bf86-4dfa-9d5a-91fd4894ed34" containerName="main" Oct 31 04:31:55.648448 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:55.648429 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d20ddaa5-bf86-4dfa-9d5a-91fd4894ed34" containerName="main" Oct 31 04:31:55.648753 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:55.648737 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="d20ddaa5-bf86-4dfa-9d5a-91fd4894ed34" containerName="main" Oct 31 04:31:55.669470 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:55.669436 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-zs49c" Oct 31 04:31:55.673547 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:55.673514 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-zs49c"] Oct 31 04:31:55.728018 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:55.727985 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4sqww\" (UniqueName: \"kubernetes.io/projected/d0c24766-7a33-47fa-a162-cf7593c2d41c-kube-api-access-4sqww\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-zs49c\" (UID: \"d0c24766-7a33-47fa-a162-cf7593c2d41c\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-zs49c" Oct 31 04:31:55.728214 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:55.728025 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0c24766-7a33-47fa-a162-cf7593c2d41c-istiod-ca-cert\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-zs49c\" (UID: \"d0c24766-7a33-47fa-a162-cf7593c2d41c\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-zs49c" Oct 31 04:31:55.728214 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:55.728099 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0c24766-7a33-47fa-a162-cf7593c2d41c-workload-certs\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-zs49c\" (UID: \"d0c24766-7a33-47fa-a162-cf7593c2d41c\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-zs49c" Oct 31 04:31:55.728214 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:55.728172 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0c24766-7a33-47fa-a162-cf7593c2d41c-istio-token\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-zs49c\" (UID: \"d0c24766-7a33-47fa-a162-cf7593c2d41c\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-zs49c" Oct 31 04:31:55.728410 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:55.728217 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0c24766-7a33-47fa-a162-cf7593c2d41c-istio-data\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-zs49c\" (UID: \"d0c24766-7a33-47fa-a162-cf7593c2d41c\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-zs49c" Oct 31 04:31:55.728410 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:55.728396 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0c24766-7a33-47fa-a162-cf7593c2d41c-istio-envoy\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-zs49c\" (UID: \"d0c24766-7a33-47fa-a162-cf7593c2d41c\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-zs49c" Oct 31 04:31:55.728516 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:55.728423 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0c24766-7a33-47fa-a162-cf7593c2d41c-workload-socket\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-zs49c\" (UID: \"d0c24766-7a33-47fa-a162-cf7593c2d41c\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-zs49c" Oct 31 04:31:55.728575 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:55.728542 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0c24766-7a33-47fa-a162-cf7593c2d41c-credential-socket\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-zs49c\" (UID: \"d0c24766-7a33-47fa-a162-cf7593c2d41c\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-zs49c" Oct 31 04:31:55.728628 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:55.728580 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0c24766-7a33-47fa-a162-cf7593c2d41c-istio-podinfo\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-zs49c\" (UID: \"d0c24766-7a33-47fa-a162-cf7593c2d41c\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-zs49c" Oct 31 04:31:55.829855 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:55.829818 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0c24766-7a33-47fa-a162-cf7593c2d41c-istio-envoy\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-zs49c\" (UID: \"d0c24766-7a33-47fa-a162-cf7593c2d41c\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-zs49c" Oct 31 04:31:55.830058 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:55.829864 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0c24766-7a33-47fa-a162-cf7593c2d41c-workload-socket\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-zs49c\" (UID: \"d0c24766-7a33-47fa-a162-cf7593c2d41c\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-zs49c" Oct 31 04:31:55.830058 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:55.829908 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0c24766-7a33-47fa-a162-cf7593c2d41c-credential-socket\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-zs49c\" (UID: \"d0c24766-7a33-47fa-a162-cf7593c2d41c\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-zs49c" Oct 31 04:31:55.830058 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:55.829930 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0c24766-7a33-47fa-a162-cf7593c2d41c-istio-podinfo\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-zs49c\" (UID: \"d0c24766-7a33-47fa-a162-cf7593c2d41c\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-zs49c" Oct 31 04:31:55.830234 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:55.830073 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4sqww\" (UniqueName: \"kubernetes.io/projected/d0c24766-7a33-47fa-a162-cf7593c2d41c-kube-api-access-4sqww\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-zs49c\" (UID: \"d0c24766-7a33-47fa-a162-cf7593c2d41c\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-zs49c" Oct 31 04:31:55.830234 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:55.830110 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0c24766-7a33-47fa-a162-cf7593c2d41c-istiod-ca-cert\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-zs49c\" (UID: \"d0c24766-7a33-47fa-a162-cf7593c2d41c\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-zs49c" Oct 31 04:31:55.830234 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:55.830164 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0c24766-7a33-47fa-a162-cf7593c2d41c-workload-certs\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-zs49c\" (UID: \"d0c24766-7a33-47fa-a162-cf7593c2d41c\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-zs49c" Oct 31 04:31:55.830417 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:55.830294 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0c24766-7a33-47fa-a162-cf7593c2d41c-istio-token\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-zs49c\" (UID: \"d0c24766-7a33-47fa-a162-cf7593c2d41c\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-zs49c" Oct 31 04:31:55.830417 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:55.830318 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0c24766-7a33-47fa-a162-cf7593c2d41c-workload-socket\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-zs49c\" (UID: \"d0c24766-7a33-47fa-a162-cf7593c2d41c\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-zs49c" Oct 31 04:31:55.830417 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:55.830387 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0c24766-7a33-47fa-a162-cf7593c2d41c-credential-socket\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-zs49c\" (UID: \"d0c24766-7a33-47fa-a162-cf7593c2d41c\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-zs49c" Oct 31 04:31:55.830585 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:55.830425 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0c24766-7a33-47fa-a162-cf7593c2d41c-istio-data\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-zs49c\" (UID: \"d0c24766-7a33-47fa-a162-cf7593c2d41c\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-zs49c" Oct 31 04:31:55.830585 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:55.830557 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0c24766-7a33-47fa-a162-cf7593c2d41c-workload-certs\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-zs49c\" (UID: \"d0c24766-7a33-47fa-a162-cf7593c2d41c\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-zs49c" Oct 31 04:31:55.830775 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:55.830735 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0c24766-7a33-47fa-a162-cf7593c2d41c-istio-data\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-zs49c\" (UID: \"d0c24766-7a33-47fa-a162-cf7593c2d41c\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-zs49c" Oct 31 04:31:55.830883 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:55.830863 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0c24766-7a33-47fa-a162-cf7593c2d41c-istiod-ca-cert\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-zs49c\" (UID: \"d0c24766-7a33-47fa-a162-cf7593c2d41c\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-zs49c" Oct 31 04:31:55.832675 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:55.832646 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0c24766-7a33-47fa-a162-cf7593c2d41c-istio-podinfo\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-zs49c\" (UID: \"d0c24766-7a33-47fa-a162-cf7593c2d41c\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-zs49c" Oct 31 04:31:55.832795 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:55.832784 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0c24766-7a33-47fa-a162-cf7593c2d41c-istio-envoy\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-zs49c\" (UID: \"d0c24766-7a33-47fa-a162-cf7593c2d41c\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-zs49c" Oct 31 04:31:55.854778 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:55.854747 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0c24766-7a33-47fa-a162-cf7593c2d41c-istio-token\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-zs49c\" (UID: \"d0c24766-7a33-47fa-a162-cf7593c2d41c\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-zs49c" Oct 31 04:31:55.854941 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:55.854911 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4sqww\" (UniqueName: \"kubernetes.io/projected/d0c24766-7a33-47fa-a162-cf7593c2d41c-kube-api-access-4sqww\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-zs49c\" (UID: \"d0c24766-7a33-47fa-a162-cf7593c2d41c\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-zs49c" Oct 31 04:31:55.997427 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:55.992908 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-zs49c" Oct 31 04:31:56.622957 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:56.622753 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-zs49c"] Oct 31 04:31:56.635353 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:56.627070 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:31:56.635353 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:56.627166 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:31:56.635353 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:56.627212 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:31:57.633350 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:57.633082 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-zs49c" event={"ID":"d0c24766-7a33-47fa-a162-cf7593c2d41c","Type":"ContainerStarted","Data":"a62e3e3373a2a3d2897a18b4aeb2c57154ba16a90b8e42d0b99b49e5fa64cc21"} Oct 31 04:31:57.633350 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:57.633246 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-zs49c" event={"ID":"d0c24766-7a33-47fa-a162-cf7593c2d41c","Type":"ContainerStarted","Data":"39f66c91db5e588d948c0718addc4a9f9989d9afa209a0ce675240236ceee9c6"} Oct 31 04:31:57.722350 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:57.721568 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-zs49c" podStartSLOduration=2.721546202 podStartE2EDuration="2.721546202s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:31:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:31:57.700994206 +0000 UTC m=+1665.415328825" watchObservedRunningTime="2026-10-31 04:31:57.721546202 +0000 UTC m=+1665.435880818" Oct 31 04:31:57.994536 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:57.994406 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-zs49c" Oct 31 04:31:58.474357 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:58.469429 4376 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-09a20a/distinct-2-w2-09a20a-kserve-779d648d75-mshl7" podUID="c81f5d58-c5c7-46c1-b832-c550b092f02b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:31:58.653482 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:58.653429 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-09a20a/premium-w2-09a20a-kserve-57c45db74f-s8glf" podUID="a3f9d0a0-a106-48bb-b6f2-a980c3aff4e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:31:58.869418 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:58.794667 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c4006d/simulator-w4-c4006d-kserve-595455485b-4h46q" podUID="576e2c1f-1c80-43a9-a136-3692776642ae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded" Oct 31 04:31:58.887353 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:58.871007 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-d7npm" podUID="174171e7-100c-4b84-becd-0176672f1805" containerName="main" probeResult="failure" output="Get \"https://10.133.0.34:8000/ready\": context deadline exceeded" Oct 31 04:31:58.956985 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:58.956940 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4c2387/distinct-w3-4c2387-kserve-768bdd84c7-6jmtl" podUID="b2c75691-4cf4-484b-96e2-3c2503b8d78c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:31:58.984352 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:58.974283 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c4006d/distinct-2-w4-c4006d-kserve-776b5b899-kpl6z" podUID="dc53083d-eadd-4cb8-8633-35e02d400b9c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:31:58.998351 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:58.992418 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-09a20a/distinct-2-w2-09a20a-kserve-779d648d75-mshl7" podUID="c81f5d58-c5c7-46c1-b832-c550b092f02b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:31:59.003355 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:58.999375 4376 patch_prober.go:29] interesting pod/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-zs49c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Oct 31 04:31:59.003355 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:58.999426 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-zs49c" podUID="d0c24766-7a33-47fa-a162-cf7593c2d41c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Oct 31 04:31:59.056616 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:59.056572 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-439989/premium-w1-439989-kserve-f9fc54cdf-sn2pp" podUID="17b2f181-2e66-4ce2-95b0-cb213f11a0e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:31:59.111467 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:59.111423 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-09a20a/distinct-w2-09a20a-kserve-76557f6448-l8d69" podUID="7d298000-b65b-45fd-a1ee-092e47a7eb24" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded" Oct 31 04:31:59.430574 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:59.430164 4376 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-09a20a/premium-w2-09a20a-kserve-57c45db74f-s8glf" podUID="a3f9d0a0-a106-48bb-b6f2-a980c3aff4e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:31:59.993824 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:59.993792 4376 patch_prober.go:29] interesting pod/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-zs49c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Oct 31 04:31:59.994300 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:31:59.993842 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-zs49c" podUID="d0c24766-7a33-47fa-a162-cf7593c2d41c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Oct 31 04:32:00.994145 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:00.994112 4376 patch_prober.go:29] interesting pod/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-zs49c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Oct 31 04:32:00.994580 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:00.994164 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-zs49c" podUID="d0c24766-7a33-47fa-a162-cf7593c2d41c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Oct 31 04:32:01.993570 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:01.993532 4376 patch_prober.go:29] interesting pod/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-zs49c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Oct 31 04:32:01.993762 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:01.993593 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-zs49c" podUID="d0c24766-7a33-47fa-a162-cf7593c2d41c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Oct 31 04:32:02.993837 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:02.993795 4376 patch_prober.go:29] interesting pod/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-zs49c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Oct 31 04:32:02.994366 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:02.993862 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-zs49c" podUID="d0c24766-7a33-47fa-a162-cf7593c2d41c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Oct 31 04:32:03.993588 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:03.993547 4376 patch_prober.go:29] interesting pod/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-zs49c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Oct 31 04:32:03.993773 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:03.993614 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-zs49c" podUID="d0c24766-7a33-47fa-a162-cf7593c2d41c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Oct 31 04:32:04.994506 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:04.994461 4376 patch_prober.go:29] interesting pod/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-zs49c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Oct 31 04:32:04.995020 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:04.994524 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-zs49c" podUID="d0c24766-7a33-47fa-a162-cf7593c2d41c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Oct 31 04:32:05.999571 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:05.999537 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-zs49c" Oct 31 04:32:06.000083 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:05.999831 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-zs49c" Oct 31 04:32:06.000718 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:06.000701 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-zs49c" Oct 31 04:32:06.803613 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:06.803577 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-d1d441-664bf44798-dr568"] Oct 31 04:32:06.803829 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:06.803802 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d1d441-664bf44798-dr568" podUID="c3970ed6-ce46-488f-a08c-46a01d41181d" containerName="maas-api" containerID="cri-o://416d6964bcd237c6217dcefefccf5f78fe9dd7f2f52cb9e27c4cb2f9505e8d8d" gracePeriod=30 Oct 31 04:32:06.926025 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:06.925993 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-dw4vw"] Oct 31 04:32:06.929962 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:06.929939 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-dw4vw" Oct 31 04:32:06.957984 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:06.957881 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-dw4vw"] Oct 31 04:32:07.141043 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:07.069651 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74d939ce-44dd-4d3c-b51e-0ba8fa7be377-workload-certs\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-dw4vw\" (UID: \"74d939ce-44dd-4d3c-b51e-0ba8fa7be377\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-dw4vw" Oct 31 04:32:07.141043 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:07.069710 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74d939ce-44dd-4d3c-b51e-0ba8fa7be377-istio-data\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-dw4vw\" (UID: \"74d939ce-44dd-4d3c-b51e-0ba8fa7be377\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-dw4vw" Oct 31 04:32:07.141043 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:07.069809 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74d939ce-44dd-4d3c-b51e-0ba8fa7be377-istiod-ca-cert\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-dw4vw\" (UID: \"74d939ce-44dd-4d3c-b51e-0ba8fa7be377\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-dw4vw" Oct 31 04:32:07.141043 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:07.069857 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74d939ce-44dd-4d3c-b51e-0ba8fa7be377-workload-socket\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-dw4vw\" (UID: \"74d939ce-44dd-4d3c-b51e-0ba8fa7be377\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-dw4vw" Oct 31 04:32:07.141043 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:07.069923 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74d939ce-44dd-4d3c-b51e-0ba8fa7be377-istio-podinfo\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-dw4vw\" (UID: \"74d939ce-44dd-4d3c-b51e-0ba8fa7be377\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-dw4vw" Oct 31 04:32:07.141043 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:07.069976 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74d939ce-44dd-4d3c-b51e-0ba8fa7be377-istio-envoy\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-dw4vw\" (UID: \"74d939ce-44dd-4d3c-b51e-0ba8fa7be377\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-dw4vw" Oct 31 04:32:07.141043 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:07.070006 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74d939ce-44dd-4d3c-b51e-0ba8fa7be377-istio-token\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-dw4vw\" (UID: \"74d939ce-44dd-4d3c-b51e-0ba8fa7be377\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-dw4vw" Oct 31 04:32:07.141043 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:07.070115 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74d939ce-44dd-4d3c-b51e-0ba8fa7be377-credential-socket\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-dw4vw\" (UID: \"74d939ce-44dd-4d3c-b51e-0ba8fa7be377\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-dw4vw" Oct 31 04:32:07.141043 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:07.070175 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-767bs\" (UniqueName: \"kubernetes.io/projected/74d939ce-44dd-4d3c-b51e-0ba8fa7be377-kube-api-access-767bs\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-dw4vw\" (UID: \"74d939ce-44dd-4d3c-b51e-0ba8fa7be377\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-dw4vw" Oct 31 04:32:07.189605 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:07.175234 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-767bs\" (UniqueName: \"kubernetes.io/projected/74d939ce-44dd-4d3c-b51e-0ba8fa7be377-kube-api-access-767bs\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-dw4vw\" (UID: \"74d939ce-44dd-4d3c-b51e-0ba8fa7be377\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-dw4vw" Oct 31 04:32:07.189605 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:07.175306 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74d939ce-44dd-4d3c-b51e-0ba8fa7be377-workload-certs\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-dw4vw\" (UID: \"74d939ce-44dd-4d3c-b51e-0ba8fa7be377\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-dw4vw" Oct 31 04:32:07.189605 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:07.175360 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74d939ce-44dd-4d3c-b51e-0ba8fa7be377-istio-data\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-dw4vw\" (UID: \"74d939ce-44dd-4d3c-b51e-0ba8fa7be377\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-dw4vw" Oct 31 04:32:07.189605 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:07.175477 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74d939ce-44dd-4d3c-b51e-0ba8fa7be377-istiod-ca-cert\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-dw4vw\" (UID: \"74d939ce-44dd-4d3c-b51e-0ba8fa7be377\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-dw4vw" Oct 31 04:32:07.189605 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:07.175531 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74d939ce-44dd-4d3c-b51e-0ba8fa7be377-workload-socket\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-dw4vw\" (UID: \"74d939ce-44dd-4d3c-b51e-0ba8fa7be377\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-dw4vw" Oct 31 04:32:07.189605 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:07.175619 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74d939ce-44dd-4d3c-b51e-0ba8fa7be377-istio-podinfo\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-dw4vw\" (UID: \"74d939ce-44dd-4d3c-b51e-0ba8fa7be377\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-dw4vw" Oct 31 04:32:07.189605 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:07.175693 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74d939ce-44dd-4d3c-b51e-0ba8fa7be377-istio-envoy\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-dw4vw\" (UID: \"74d939ce-44dd-4d3c-b51e-0ba8fa7be377\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-dw4vw" Oct 31 04:32:07.189605 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:07.175725 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74d939ce-44dd-4d3c-b51e-0ba8fa7be377-istio-token\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-dw4vw\" (UID: \"74d939ce-44dd-4d3c-b51e-0ba8fa7be377\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-dw4vw" Oct 31 04:32:07.189605 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:07.175846 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74d939ce-44dd-4d3c-b51e-0ba8fa7be377-credential-socket\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-dw4vw\" (UID: \"74d939ce-44dd-4d3c-b51e-0ba8fa7be377\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-dw4vw" Oct 31 04:32:07.189605 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:07.176345 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74d939ce-44dd-4d3c-b51e-0ba8fa7be377-credential-socket\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-dw4vw\" (UID: \"74d939ce-44dd-4d3c-b51e-0ba8fa7be377\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-dw4vw" Oct 31 04:32:07.189605 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:07.187894 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74d939ce-44dd-4d3c-b51e-0ba8fa7be377-workload-certs\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-dw4vw\" (UID: \"74d939ce-44dd-4d3c-b51e-0ba8fa7be377\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-dw4vw" Oct 31 04:32:07.189605 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:07.188152 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74d939ce-44dd-4d3c-b51e-0ba8fa7be377-istio-data\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-dw4vw\" (UID: \"74d939ce-44dd-4d3c-b51e-0ba8fa7be377\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-dw4vw" Oct 31 04:32:07.189605 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:07.188801 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74d939ce-44dd-4d3c-b51e-0ba8fa7be377-istiod-ca-cert\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-dw4vw\" (UID: \"74d939ce-44dd-4d3c-b51e-0ba8fa7be377\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-dw4vw" Oct 31 04:32:07.189605 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:07.189040 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74d939ce-44dd-4d3c-b51e-0ba8fa7be377-workload-socket\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-dw4vw\" (UID: \"74d939ce-44dd-4d3c-b51e-0ba8fa7be377\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-dw4vw" Oct 31 04:32:07.198122 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:07.198026 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74d939ce-44dd-4d3c-b51e-0ba8fa7be377-istio-podinfo\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-dw4vw\" (UID: \"74d939ce-44dd-4d3c-b51e-0ba8fa7be377\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-dw4vw" Oct 31 04:32:07.200652 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:07.200505 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74d939ce-44dd-4d3c-b51e-0ba8fa7be377-istio-envoy\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-dw4vw\" (UID: \"74d939ce-44dd-4d3c-b51e-0ba8fa7be377\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-dw4vw" Oct 31 04:32:07.216734 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:07.216668 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74d939ce-44dd-4d3c-b51e-0ba8fa7be377-istio-token\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-dw4vw\" (UID: \"74d939ce-44dd-4d3c-b51e-0ba8fa7be377\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-dw4vw" Oct 31 04:32:07.246381 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:07.227370 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-767bs\" (UniqueName: \"kubernetes.io/projected/74d939ce-44dd-4d3c-b51e-0ba8fa7be377-kube-api-access-767bs\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-dw4vw\" (UID: \"74d939ce-44dd-4d3c-b51e-0ba8fa7be377\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-dw4vw" Oct 31 04:32:07.268222 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:07.261212 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-dw4vw" Oct 31 04:32:07.363126 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:07.303468 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-e37d27-75bf88d5c4-mxc78"] Oct 31 04:32:07.363126 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:07.303722 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-e37d27-75bf88d5c4-mxc78" podUID="216d467a-46f9-4fbe-ab89-899e7d9d9f60" containerName="maas-api" containerID="cri-o://a1c7a967629979edfb4ff8b47ac615baf1e0c1b9d763deaf69110bde5ca9994c" gracePeriod=30 Oct 31 04:32:07.405180 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:07.405149 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d1d441-664bf44798-dr568" Oct 31 04:32:07.584067 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:07.584028 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3970ed6-ce46-488f-a08c-46a01d41181d-kube-api-access-4fx8f\" (UniqueName: \"kubernetes.io/projected/c3970ed6-ce46-488f-a08c-46a01d41181d-kube-api-access-4fx8f\") pod \"c3970ed6-ce46-488f-a08c-46a01d41181d\" (UID: \"c3970ed6-ce46-488f-a08c-46a01d41181d\") " Oct 31 04:32:07.584230 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:07.584100 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c3970ed6-ce46-488f-a08c-46a01d41181d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3970ed6-ce46-488f-a08c-46a01d41181d-metrics-certs\") pod \"c3970ed6-ce46-488f-a08c-46a01d41181d\" (UID: \"c3970ed6-ce46-488f-a08c-46a01d41181d\") " Oct 31 04:32:07.584298 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:07.584251 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c3970ed6-ce46-488f-a08c-46a01d41181d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3970ed6-ce46-488f-a08c-46a01d41181d-maas-api-tls\") pod \"c3970ed6-ce46-488f-a08c-46a01d41181d\" (UID: \"c3970ed6-ce46-488f-a08c-46a01d41181d\") " Oct 31 04:32:07.584402 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:07.584314 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3970ed6-ce46-488f-a08c-46a01d41181d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3970ed6-ce46-488f-a08c-46a01d41181d-openshift-service-ca\") pod \"c3970ed6-ce46-488f-a08c-46a01d41181d\" (UID: \"c3970ed6-ce46-488f-a08c-46a01d41181d\") " Oct 31 04:32:07.585407 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:07.585375 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3970ed6-ce46-488f-a08c-46a01d41181d-openshift-service-ca" pod "c3970ed6-ce46-488f-a08c-46a01d41181d" (UID: "c3970ed6-ce46-488f-a08c-46a01d41181d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:32:07.589525 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:07.589493 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3970ed6-ce46-488f-a08c-46a01d41181d-maas-api-tls" pod "c3970ed6-ce46-488f-a08c-46a01d41181d" (UID: "c3970ed6-ce46-488f-a08c-46a01d41181d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:32:07.590618 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:07.590585 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3970ed6-ce46-488f-a08c-46a01d41181d-kube-api-access-4fx8f" pod "c3970ed6-ce46-488f-a08c-46a01d41181d" (UID: "c3970ed6-ce46-488f-a08c-46a01d41181d"). InnerVolumeSpecName "kube-api-access-4fx8f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:32:07.590727 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:07.590665 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3970ed6-ce46-488f-a08c-46a01d41181d-metrics-certs" pod "c3970ed6-ce46-488f-a08c-46a01d41181d" (UID: "c3970ed6-ce46-488f-a08c-46a01d41181d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:32:07.650764 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:07.650738 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-e37d27-75bf88d5c4-mxc78" Oct 31 04:32:07.722408 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:07.720750 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fx8f\" (UniqueName: \"kubernetes.io/projected/c3970ed6-ce46-488f-a08c-46a01d41181d-kube-api-access-4fx8f\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:32:07.722408 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:07.720800 4376 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3970ed6-ce46-488f-a08c-46a01d41181d-metrics-certs\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:32:07.722408 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:07.720817 4376 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3970ed6-ce46-488f-a08c-46a01d41181d-maas-api-tls\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:32:07.722408 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:07.720830 4376 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3970ed6-ce46-488f-a08c-46a01d41181d-openshift-service-ca\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:32:07.830358 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:07.827266 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/216d467a-46f9-4fbe-ab89-899e7d9d9f60-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/216d467a-46f9-4fbe-ab89-899e7d9d9f60-openshift-service-ca\") pod \"216d467a-46f9-4fbe-ab89-899e7d9d9f60\" (UID: \"216d467a-46f9-4fbe-ab89-899e7d9d9f60\") " Oct 31 04:32:07.830358 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:07.827353 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/216d467a-46f9-4fbe-ab89-899e7d9d9f60-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/216d467a-46f9-4fbe-ab89-899e7d9d9f60-maas-api-tls\") pod \"216d467a-46f9-4fbe-ab89-899e7d9d9f60\" (UID: \"216d467a-46f9-4fbe-ab89-899e7d9d9f60\") " Oct 31 04:32:07.830358 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:07.827665 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/216d467a-46f9-4fbe-ab89-899e7d9d9f60-metrics-certs\" (UniqueName: \"kubernetes.io/secret/216d467a-46f9-4fbe-ab89-899e7d9d9f60-metrics-certs\") pod \"216d467a-46f9-4fbe-ab89-899e7d9d9f60\" (UID: \"216d467a-46f9-4fbe-ab89-899e7d9d9f60\") " Oct 31 04:32:07.830358 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:07.827824 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/216d467a-46f9-4fbe-ab89-899e7d9d9f60-kube-api-access-rnx4l\" (UniqueName: \"kubernetes.io/projected/216d467a-46f9-4fbe-ab89-899e7d9d9f60-kube-api-access-rnx4l\") pod \"216d467a-46f9-4fbe-ab89-899e7d9d9f60\" (UID: \"216d467a-46f9-4fbe-ab89-899e7d9d9f60\") " Oct 31 04:32:07.843855 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:07.843215 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/216d467a-46f9-4fbe-ab89-899e7d9d9f60-openshift-service-ca" pod "216d467a-46f9-4fbe-ab89-899e7d9d9f60" (UID: "216d467a-46f9-4fbe-ab89-899e7d9d9f60"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:32:07.921266 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:07.914404 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:32:07.921266 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:07.914497 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:32:07.921266 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:07.914539 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:32:07.921266 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:07.918160 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/216d467a-46f9-4fbe-ab89-899e7d9d9f60-metrics-certs" pod "216d467a-46f9-4fbe-ab89-899e7d9d9f60" (UID: "216d467a-46f9-4fbe-ab89-899e7d9d9f60"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:32:07.921266 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:07.918207 4376 generic.go:356] "Generic (PLEG): container finished" podID="216d467a-46f9-4fbe-ab89-899e7d9d9f60" containerID="a1c7a967629979edfb4ff8b47ac615baf1e0c1b9d763deaf69110bde5ca9994c" exitCode=0 Oct 31 04:32:07.921266 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:07.918270 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-e37d27-75bf88d5c4-mxc78" Oct 31 04:32:07.921266 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:07.918372 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-e37d27-75bf88d5c4-mxc78" event={"ID":"216d467a-46f9-4fbe-ab89-899e7d9d9f60","Type":"ContainerDied","Data":"a1c7a967629979edfb4ff8b47ac615baf1e0c1b9d763deaf69110bde5ca9994c"} Oct 31 04:32:07.921266 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:07.918416 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-e37d27-75bf88d5c4-mxc78" event={"ID":"216d467a-46f9-4fbe-ab89-899e7d9d9f60","Type":"ContainerDied","Data":"918c68930ab1d9c95105b468bd9b63147066557566f97e5115606f37e25bb0f4"} Oct 31 04:32:07.921266 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:07.918448 4376 scope.go:122] "RemoveContainer" containerID="a1c7a967629979edfb4ff8b47ac615baf1e0c1b9d763deaf69110bde5ca9994c" Oct 31 04:32:07.926845 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:07.922457 4376 generic.go:356] "Generic (PLEG): container finished" podID="c3970ed6-ce46-488f-a08c-46a01d41181d" containerID="416d6964bcd237c6217dcefefccf5f78fe9dd7f2f52cb9e27c4cb2f9505e8d8d" exitCode=0 Oct 31 04:32:07.926845 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:07.922527 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d1d441-664bf44798-dr568" event={"ID":"c3970ed6-ce46-488f-a08c-46a01d41181d","Type":"ContainerDied","Data":"416d6964bcd237c6217dcefefccf5f78fe9dd7f2f52cb9e27c4cb2f9505e8d8d"} Oct 31 04:32:07.926845 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:07.922574 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d1d441-664bf44798-dr568" event={"ID":"c3970ed6-ce46-488f-a08c-46a01d41181d","Type":"ContainerDied","Data":"76743cebc8df9704e7dc6f06fccddfe50cedb6d75563c83500c398a29666752b"} Oct 31 04:32:07.926845 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:07.922597 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d1d441-664bf44798-dr568" Oct 31 04:32:07.926845 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:07.925314 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/216d467a-46f9-4fbe-ab89-899e7d9d9f60-kube-api-access-rnx4l" pod "216d467a-46f9-4fbe-ab89-899e7d9d9f60" (UID: "216d467a-46f9-4fbe-ab89-899e7d9d9f60"). InnerVolumeSpecName "kube-api-access-rnx4l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:32:07.927463 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:07.927437 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/216d467a-46f9-4fbe-ab89-899e7d9d9f60-maas-api-tls" pod "216d467a-46f9-4fbe-ab89-899e7d9d9f60" (UID: "216d467a-46f9-4fbe-ab89-899e7d9d9f60"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:32:07.931160 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:07.929905 4376 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/216d467a-46f9-4fbe-ab89-899e7d9d9f60-metrics-certs\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:32:07.931160 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:07.929933 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rnx4l\" (UniqueName: \"kubernetes.io/projected/216d467a-46f9-4fbe-ab89-899e7d9d9f60-kube-api-access-rnx4l\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:32:07.931160 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:07.929948 4376 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/216d467a-46f9-4fbe-ab89-899e7d9d9f60-openshift-service-ca\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:32:07.931160 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:07.929963 4376 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/216d467a-46f9-4fbe-ab89-899e7d9d9f60-maas-api-tls\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:32:07.932593 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:07.932527 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-dw4vw"] Oct 31 04:32:07.961001 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:07.960977 4376 scope.go:122] "RemoveContainer" containerID="a1c7a967629979edfb4ff8b47ac615baf1e0c1b9d763deaf69110bde5ca9994c" Oct 31 04:32:07.962883 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:32:07.962864 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1c7a967629979edfb4ff8b47ac615baf1e0c1b9d763deaf69110bde5ca9994c\": container with ID starting with a1c7a967629979edfb4ff8b47ac615baf1e0c1b9d763deaf69110bde5ca9994c not found: ID does not exist" containerID="a1c7a967629979edfb4ff8b47ac615baf1e0c1b9d763deaf69110bde5ca9994c" Oct 31 04:32:07.963044 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:07.963024 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1c7a967629979edfb4ff8b47ac615baf1e0c1b9d763deaf69110bde5ca9994c"} err="failed to get container status \"a1c7a967629979edfb4ff8b47ac615baf1e0c1b9d763deaf69110bde5ca9994c\": rpc error: code = NotFound desc = could not find container \"a1c7a967629979edfb4ff8b47ac615baf1e0c1b9d763deaf69110bde5ca9994c\": container with ID starting with a1c7a967629979edfb4ff8b47ac615baf1e0c1b9d763deaf69110bde5ca9994c not found: ID does not exist" Oct 31 04:32:07.963120 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:07.963112 4376 scope.go:122] "RemoveContainer" containerID="416d6964bcd237c6217dcefefccf5f78fe9dd7f2f52cb9e27c4cb2f9505e8d8d" Oct 31 04:32:07.979751 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:07.979729 4376 scope.go:122] "RemoveContainer" containerID="416d6964bcd237c6217dcefefccf5f78fe9dd7f2f52cb9e27c4cb2f9505e8d8d" Oct 31 04:32:07.980168 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:32:07.980143 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"416d6964bcd237c6217dcefefccf5f78fe9dd7f2f52cb9e27c4cb2f9505e8d8d\": container with ID starting with 416d6964bcd237c6217dcefefccf5f78fe9dd7f2f52cb9e27c4cb2f9505e8d8d not found: ID does not exist" containerID="416d6964bcd237c6217dcefefccf5f78fe9dd7f2f52cb9e27c4cb2f9505e8d8d" Oct 31 04:32:07.980251 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:07.980180 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"416d6964bcd237c6217dcefefccf5f78fe9dd7f2f52cb9e27c4cb2f9505e8d8d"} err="failed to get container status \"416d6964bcd237c6217dcefefccf5f78fe9dd7f2f52cb9e27c4cb2f9505e8d8d\": rpc error: code = NotFound desc = could not find container \"416d6964bcd237c6217dcefefccf5f78fe9dd7f2f52cb9e27c4cb2f9505e8d8d\": container with ID starting with 416d6964bcd237c6217dcefefccf5f78fe9dd7f2f52cb9e27c4cb2f9505e8d8d not found: ID does not exist" Oct 31 04:32:08.015272 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:08.015242 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-d1d441-664bf44798-dr568"] Oct 31 04:32:08.021817 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:08.021791 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-d1d441-664bf44798-dr568"] Oct 31 04:32:08.268976 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:08.268900 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-e37d27-75bf88d5c4-mxc78"] Oct 31 04:32:08.274036 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:08.274009 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-e37d27-75bf88d5c4-mxc78"] Oct 31 04:32:08.487351 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:08.487073 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="216d467a-46f9-4fbe-ab89-899e7d9d9f60" path="/var/lib/kubelet/pods/216d467a-46f9-4fbe-ab89-899e7d9d9f60/volumes" Oct 31 04:32:08.491647 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:08.487864 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3970ed6-ce46-488f-a08c-46a01d41181d" path="/var/lib/kubelet/pods/c3970ed6-ce46-488f-a08c-46a01d41181d/volumes" Oct 31 04:32:08.968261 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:08.968219 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-dw4vw" event={"ID":"74d939ce-44dd-4d3c-b51e-0ba8fa7be377","Type":"ContainerStarted","Data":"43dda6c731a7c063331c51c61fd7449e6d4a7bbdadee291a0886056d22c384fd"} Oct 31 04:32:08.968521 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:08.968275 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-dw4vw" event={"ID":"74d939ce-44dd-4d3c-b51e-0ba8fa7be377","Type":"ContainerStarted","Data":"b9d0f681eb3e19230ae095ccaea0bf09fe629c41a7d3a77f69e73dd3c854b44c"} Oct 31 04:32:08.982390 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:08.982350 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-c9707f-openshift-default-65b9677b4c-b57kk"] Oct 31 04:32:08.983379 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:08.983350 4376 cpu_manager.go:414] "containerMap: removing container" podUID="216d467a-46f9-4fbe-ab89-899e7d9d9f60" containerName="maas-api" Oct 31 04:32:08.983522 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:08.983509 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="216d467a-46f9-4fbe-ab89-899e7d9d9f60" containerName="maas-api" Oct 31 04:32:08.983603 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:08.983594 4376 cpu_manager.go:414] "containerMap: removing container" podUID="c3970ed6-ce46-488f-a08c-46a01d41181d" containerName="maas-api" Oct 31 04:32:08.983675 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:08.983666 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3970ed6-ce46-488f-a08c-46a01d41181d" containerName="maas-api" Oct 31 04:32:08.983950 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:08.983929 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="216d467a-46f9-4fbe-ab89-899e7d9d9f60" containerName="maas-api" Oct 31 04:32:08.984052 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:08.984042 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3970ed6-ce46-488f-a08c-46a01d41181d" containerName="maas-api" Oct 31 04:32:08.996951 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:08.996695 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-c9707f-openshift-default-65b9677b4c-b57kk" Oct 31 04:32:09.013213 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:09.013162 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-c9707f-openshift-default-dockercfg-ksmf8\"" Oct 31 04:32:09.015050 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:09.014934 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-c9707f-openshift-default-65b9677b4c-b57kk"] Oct 31 04:32:09.071338 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:09.071271 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-dw4vw" podStartSLOduration=3.071250832 podStartE2EDuration="3.071250832s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:32:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:32:09.024664826 +0000 UTC m=+1676.738999443" watchObservedRunningTime="2026-10-31 04:32:09.071250832 +0000 UTC m=+1676.785585450" Oct 31 04:32:09.098803 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:09.098759 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/932966ca-c565-4bab-a007-7f918c7f0c15-istio-podinfo\") pod \"e2e-shared-a-w2-c9707f-openshift-default-65b9677b4c-b57kk\" (UID: \"932966ca-c565-4bab-a007-7f918c7f0c15\") " pod="openshift-ingress/e2e-shared-a-w2-c9707f-openshift-default-65b9677b4c-b57kk" Oct 31 04:32:09.098986 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:09.098890 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kl5lh\" (UniqueName: \"kubernetes.io/projected/932966ca-c565-4bab-a007-7f918c7f0c15-kube-api-access-kl5lh\") pod \"e2e-shared-a-w2-c9707f-openshift-default-65b9677b4c-b57kk\" (UID: \"932966ca-c565-4bab-a007-7f918c7f0c15\") " pod="openshift-ingress/e2e-shared-a-w2-c9707f-openshift-default-65b9677b4c-b57kk" Oct 31 04:32:09.098986 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:09.098922 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/932966ca-c565-4bab-a007-7f918c7f0c15-credential-socket\") pod \"e2e-shared-a-w2-c9707f-openshift-default-65b9677b4c-b57kk\" (UID: \"932966ca-c565-4bab-a007-7f918c7f0c15\") " pod="openshift-ingress/e2e-shared-a-w2-c9707f-openshift-default-65b9677b4c-b57kk" Oct 31 04:32:09.098986 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:09.098978 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/932966ca-c565-4bab-a007-7f918c7f0c15-istio-envoy\") pod \"e2e-shared-a-w2-c9707f-openshift-default-65b9677b4c-b57kk\" (UID: \"932966ca-c565-4bab-a007-7f918c7f0c15\") " pod="openshift-ingress/e2e-shared-a-w2-c9707f-openshift-default-65b9677b4c-b57kk" Oct 31 04:32:09.099134 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:09.099024 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/932966ca-c565-4bab-a007-7f918c7f0c15-workload-certs\") pod \"e2e-shared-a-w2-c9707f-openshift-default-65b9677b4c-b57kk\" (UID: \"932966ca-c565-4bab-a007-7f918c7f0c15\") " pod="openshift-ingress/e2e-shared-a-w2-c9707f-openshift-default-65b9677b4c-b57kk" Oct 31 04:32:09.099134 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:09.099085 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/932966ca-c565-4bab-a007-7f918c7f0c15-istio-data\") pod \"e2e-shared-a-w2-c9707f-openshift-default-65b9677b4c-b57kk\" (UID: \"932966ca-c565-4bab-a007-7f918c7f0c15\") " pod="openshift-ingress/e2e-shared-a-w2-c9707f-openshift-default-65b9677b4c-b57kk" Oct 31 04:32:09.099134 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:09.099117 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/932966ca-c565-4bab-a007-7f918c7f0c15-istiod-ca-cert\") pod \"e2e-shared-a-w2-c9707f-openshift-default-65b9677b4c-b57kk\" (UID: \"932966ca-c565-4bab-a007-7f918c7f0c15\") " pod="openshift-ingress/e2e-shared-a-w2-c9707f-openshift-default-65b9677b4c-b57kk" Oct 31 04:32:09.099251 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:09.099149 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/932966ca-c565-4bab-a007-7f918c7f0c15-istio-token\") pod \"e2e-shared-a-w2-c9707f-openshift-default-65b9677b4c-b57kk\" (UID: \"932966ca-c565-4bab-a007-7f918c7f0c15\") " pod="openshift-ingress/e2e-shared-a-w2-c9707f-openshift-default-65b9677b4c-b57kk" Oct 31 04:32:09.099289 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:09.099263 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/932966ca-c565-4bab-a007-7f918c7f0c15-workload-socket\") pod \"e2e-shared-a-w2-c9707f-openshift-default-65b9677b4c-b57kk\" (UID: \"932966ca-c565-4bab-a007-7f918c7f0c15\") " pod="openshift-ingress/e2e-shared-a-w2-c9707f-openshift-default-65b9677b4c-b57kk" Oct 31 04:32:09.214871 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:09.214780 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kl5lh\" (UniqueName: \"kubernetes.io/projected/932966ca-c565-4bab-a007-7f918c7f0c15-kube-api-access-kl5lh\") pod \"e2e-shared-a-w2-c9707f-openshift-default-65b9677b4c-b57kk\" (UID: \"932966ca-c565-4bab-a007-7f918c7f0c15\") " pod="openshift-ingress/e2e-shared-a-w2-c9707f-openshift-default-65b9677b4c-b57kk" Oct 31 04:32:09.215472 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:09.215250 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/932966ca-c565-4bab-a007-7f918c7f0c15-credential-socket\") pod \"e2e-shared-a-w2-c9707f-openshift-default-65b9677b4c-b57kk\" (UID: \"932966ca-c565-4bab-a007-7f918c7f0c15\") " pod="openshift-ingress/e2e-shared-a-w2-c9707f-openshift-default-65b9677b4c-b57kk" Oct 31 04:32:09.215795 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:09.215775 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/932966ca-c565-4bab-a007-7f918c7f0c15-istio-envoy\") pod \"e2e-shared-a-w2-c9707f-openshift-default-65b9677b4c-b57kk\" (UID: \"932966ca-c565-4bab-a007-7f918c7f0c15\") " pod="openshift-ingress/e2e-shared-a-w2-c9707f-openshift-default-65b9677b4c-b57kk" Oct 31 04:32:09.215932 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:09.215908 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/932966ca-c565-4bab-a007-7f918c7f0c15-credential-socket\") pod \"e2e-shared-a-w2-c9707f-openshift-default-65b9677b4c-b57kk\" (UID: \"932966ca-c565-4bab-a007-7f918c7f0c15\") " pod="openshift-ingress/e2e-shared-a-w2-c9707f-openshift-default-65b9677b4c-b57kk" Oct 31 04:32:09.216043 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:09.216027 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/932966ca-c565-4bab-a007-7f918c7f0c15-workload-certs\") pod \"e2e-shared-a-w2-c9707f-openshift-default-65b9677b4c-b57kk\" (UID: \"932966ca-c565-4bab-a007-7f918c7f0c15\") " pod="openshift-ingress/e2e-shared-a-w2-c9707f-openshift-default-65b9677b4c-b57kk" Oct 31 04:32:09.216178 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:09.216163 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/932966ca-c565-4bab-a007-7f918c7f0c15-istio-data\") pod \"e2e-shared-a-w2-c9707f-openshift-default-65b9677b4c-b57kk\" (UID: \"932966ca-c565-4bab-a007-7f918c7f0c15\") " pod="openshift-ingress/e2e-shared-a-w2-c9707f-openshift-default-65b9677b4c-b57kk" Oct 31 04:32:09.216338 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:09.216310 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/932966ca-c565-4bab-a007-7f918c7f0c15-istiod-ca-cert\") pod \"e2e-shared-a-w2-c9707f-openshift-default-65b9677b4c-b57kk\" (UID: \"932966ca-c565-4bab-a007-7f918c7f0c15\") " pod="openshift-ingress/e2e-shared-a-w2-c9707f-openshift-default-65b9677b4c-b57kk" Oct 31 04:32:09.216492 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:09.216464 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/932966ca-c565-4bab-a007-7f918c7f0c15-istio-token\") pod \"e2e-shared-a-w2-c9707f-openshift-default-65b9677b4c-b57kk\" (UID: \"932966ca-c565-4bab-a007-7f918c7f0c15\") " pod="openshift-ingress/e2e-shared-a-w2-c9707f-openshift-default-65b9677b4c-b57kk" Oct 31 04:32:09.216719 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:09.216704 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/932966ca-c565-4bab-a007-7f918c7f0c15-workload-socket\") pod \"e2e-shared-a-w2-c9707f-openshift-default-65b9677b4c-b57kk\" (UID: \"932966ca-c565-4bab-a007-7f918c7f0c15\") " pod="openshift-ingress/e2e-shared-a-w2-c9707f-openshift-default-65b9677b4c-b57kk" Oct 31 04:32:09.216937 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:09.216920 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/932966ca-c565-4bab-a007-7f918c7f0c15-istio-podinfo\") pod \"e2e-shared-a-w2-c9707f-openshift-default-65b9677b4c-b57kk\" (UID: \"932966ca-c565-4bab-a007-7f918c7f0c15\") " pod="openshift-ingress/e2e-shared-a-w2-c9707f-openshift-default-65b9677b4c-b57kk" Oct 31 04:32:09.217264 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:09.216537 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/932966ca-c565-4bab-a007-7f918c7f0c15-workload-certs\") pod \"e2e-shared-a-w2-c9707f-openshift-default-65b9677b4c-b57kk\" (UID: \"932966ca-c565-4bab-a007-7f918c7f0c15\") " pod="openshift-ingress/e2e-shared-a-w2-c9707f-openshift-default-65b9677b4c-b57kk" Oct 31 04:32:09.217435 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:09.217411 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/932966ca-c565-4bab-a007-7f918c7f0c15-istio-data\") pod \"e2e-shared-a-w2-c9707f-openshift-default-65b9677b4c-b57kk\" (UID: \"932966ca-c565-4bab-a007-7f918c7f0c15\") " pod="openshift-ingress/e2e-shared-a-w2-c9707f-openshift-default-65b9677b4c-b57kk" Oct 31 04:32:09.218406 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:09.217733 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/932966ca-c565-4bab-a007-7f918c7f0c15-workload-socket\") pod \"e2e-shared-a-w2-c9707f-openshift-default-65b9677b4c-b57kk\" (UID: \"932966ca-c565-4bab-a007-7f918c7f0c15\") " pod="openshift-ingress/e2e-shared-a-w2-c9707f-openshift-default-65b9677b4c-b57kk" Oct 31 04:32:09.218406 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:09.217841 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/932966ca-c565-4bab-a007-7f918c7f0c15-istiod-ca-cert\") pod \"e2e-shared-a-w2-c9707f-openshift-default-65b9677b4c-b57kk\" (UID: \"932966ca-c565-4bab-a007-7f918c7f0c15\") " pod="openshift-ingress/e2e-shared-a-w2-c9707f-openshift-default-65b9677b4c-b57kk" Oct 31 04:32:09.247788 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:09.247755 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/932966ca-c565-4bab-a007-7f918c7f0c15-istio-podinfo\") pod \"e2e-shared-a-w2-c9707f-openshift-default-65b9677b4c-b57kk\" (UID: \"932966ca-c565-4bab-a007-7f918c7f0c15\") " pod="openshift-ingress/e2e-shared-a-w2-c9707f-openshift-default-65b9677b4c-b57kk" Oct 31 04:32:09.248816 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:09.248791 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/932966ca-c565-4bab-a007-7f918c7f0c15-istio-envoy\") pod \"e2e-shared-a-w2-c9707f-openshift-default-65b9677b4c-b57kk\" (UID: \"932966ca-c565-4bab-a007-7f918c7f0c15\") " pod="openshift-ingress/e2e-shared-a-w2-c9707f-openshift-default-65b9677b4c-b57kk" Oct 31 04:32:09.255380 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:09.255284 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/932966ca-c565-4bab-a007-7f918c7f0c15-istio-token\") pod \"e2e-shared-a-w2-c9707f-openshift-default-65b9677b4c-b57kk\" (UID: \"932966ca-c565-4bab-a007-7f918c7f0c15\") " pod="openshift-ingress/e2e-shared-a-w2-c9707f-openshift-default-65b9677b4c-b57kk" Oct 31 04:32:09.259625 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:09.259571 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kl5lh\" (UniqueName: \"kubernetes.io/projected/932966ca-c565-4bab-a007-7f918c7f0c15-kube-api-access-kl5lh\") pod \"e2e-shared-a-w2-c9707f-openshift-default-65b9677b4c-b57kk\" (UID: \"932966ca-c565-4bab-a007-7f918c7f0c15\") " pod="openshift-ingress/e2e-shared-a-w2-c9707f-openshift-default-65b9677b4c-b57kk" Oct 31 04:32:09.280734 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:09.278717 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-dw4vw" Oct 31 04:32:09.281649 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:09.281320 4376 patch_prober.go:29] interesting pod/e2e-worker-w2-09a20a-openshift-default-55d8647ff-dw4vw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body= Oct 31 04:32:09.281649 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:09.281391 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-dw4vw" podUID="74d939ce-44dd-4d3c-b51e-0ba8fa7be377" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" Oct 31 04:32:09.339164 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:09.336764 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-c9707f-openshift-default-65b9677b4c-b57kk" Oct 31 04:32:09.382905 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:09.382825 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-d1d441-openshift-default-c79997b97-84mqf"] Oct 31 04:32:09.383265 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:09.383156 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-d1d441-openshift-default-c79997b97-84mqf" podUID="c9240e5d-789e-4ebf-8cd3-e7807036e8a7" containerName="istio-proxy" containerID="cri-o://921c2507634db695656d081bd28dd097aa6799a82fd730ce384a56c618ea72e8" gracePeriod=30 Oct 31 04:32:09.967617 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:32:09.967575 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod932966ca_c565_4bab_a007_7f918c7f0c15.slice/crio-97d19c8b418af56540f26c71d79c70cf2e7ed7a9feed3e8c30aa4d5385f38c08 WatchSource:0}: Error finding container 97d19c8b418af56540f26c71d79c70cf2e7ed7a9feed3e8c30aa4d5385f38c08: Status 404 returned error can't find the container with id 97d19c8b418af56540f26c71d79c70cf2e7ed7a9feed3e8c30aa4d5385f38c08 Oct 31 04:32:09.967879 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:09.967855 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-c9707f-openshift-default-65b9677b4c-b57kk"] Oct 31 04:32:09.970913 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:09.970875 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:32:09.971010 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:09.970959 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:32:09.971050 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:09.971004 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:32:09.981915 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:09.981888 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-c9707f-openshift-default-65b9677b4c-b57kk" event={"ID":"932966ca-c565-4bab-a007-7f918c7f0c15","Type":"ContainerStarted","Data":"97d19c8b418af56540f26c71d79c70cf2e7ed7a9feed3e8c30aa4d5385f38c08"} Oct 31 04:32:10.269305 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:10.269213 4376 patch_prober.go:29] interesting pod/e2e-worker-w2-09a20a-openshift-default-55d8647ff-dw4vw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body= Oct 31 04:32:10.269305 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:10.269273 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-dw4vw" podUID="74d939ce-44dd-4d3c-b51e-0ba8fa7be377" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" Oct 31 04:32:10.990504 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:10.990469 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-c9707f-openshift-default-65b9677b4c-b57kk" event={"ID":"932966ca-c565-4bab-a007-7f918c7f0c15","Type":"ContainerStarted","Data":"0f66c5f0d835c6a13b765cb395beaa14cefce2a0b4d49bf6e756db60d2b08bfb"} Oct 31 04:32:11.002415 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:32:11.002383 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 04:32:11.063366 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:11.058296 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-c9707f-openshift-default-65b9677b4c-b57kk" podStartSLOduration=3.05827686 podStartE2EDuration="3.05827686s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:32:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:32:11.057174107 +0000 UTC m=+1678.771508742" watchObservedRunningTime="2026-10-31 04:32:11.05827686 +0000 UTC m=+1678.772611477" Oct 31 04:32:11.270407 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:11.268018 4376 patch_prober.go:29] interesting pod/e2e-worker-w2-09a20a-openshift-default-55d8647ff-dw4vw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body= Oct 31 04:32:11.270407 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:11.268090 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-dw4vw" podUID="74d939ce-44dd-4d3c-b51e-0ba8fa7be377" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" Oct 31 04:32:11.337149 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:11.337117 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-c9707f-openshift-default-65b9677b4c-b57kk" Oct 31 04:32:11.365091 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:11.365052 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-c9707f-openshift-default-65b9677b4c-b57kk" Oct 31 04:32:11.997389 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:11.997359 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-c9707f-openshift-default-65b9677b4c-b57kk" Oct 31 04:32:11.998470 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:11.998449 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-c9707f-openshift-default-65b9677b4c-b57kk" Oct 31 04:32:12.267989 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:12.267896 4376 patch_prober.go:29] interesting pod/e2e-worker-w2-09a20a-openshift-default-55d8647ff-dw4vw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body= Oct 31 04:32:12.267989 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:12.267962 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-dw4vw" podUID="74d939ce-44dd-4d3c-b51e-0ba8fa7be377" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" Oct 31 04:32:13.267896 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:13.267855 4376 patch_prober.go:29] interesting pod/e2e-worker-w2-09a20a-openshift-default-55d8647ff-dw4vw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body= Oct 31 04:32:13.268407 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:13.267931 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-dw4vw" podUID="74d939ce-44dd-4d3c-b51e-0ba8fa7be377" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" Oct 31 04:32:14.270354 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:14.268732 4376 patch_prober.go:29] interesting pod/e2e-worker-w2-09a20a-openshift-default-55d8647ff-dw4vw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body= Oct 31 04:32:14.270354 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:14.268806 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-dw4vw" podUID="74d939ce-44dd-4d3c-b51e-0ba8fa7be377" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" Oct 31 04:32:14.674216 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:14.674183 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-d1d441-openshift-default-c79997b97-84mqf" Oct 31 04:32:14.833830 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:14.833740 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c9240e5d-789e-4ebf-8cd3-e7807036e8a7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9240e5d-789e-4ebf-8cd3-e7807036e8a7-istio-podinfo\") pod \"c9240e5d-789e-4ebf-8cd3-e7807036e8a7\" (UID: \"c9240e5d-789e-4ebf-8cd3-e7807036e8a7\") " Oct 31 04:32:14.833830 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:14.833793 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9240e5d-789e-4ebf-8cd3-e7807036e8a7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9240e5d-789e-4ebf-8cd3-e7807036e8a7-istiod-ca-cert\") pod \"c9240e5d-789e-4ebf-8cd3-e7807036e8a7\" (UID: \"c9240e5d-789e-4ebf-8cd3-e7807036e8a7\") " Oct 31 04:32:14.834021 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:14.833938 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9240e5d-789e-4ebf-8cd3-e7807036e8a7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9240e5d-789e-4ebf-8cd3-e7807036e8a7-workload-socket\") pod \"c9240e5d-789e-4ebf-8cd3-e7807036e8a7\" (UID: \"c9240e5d-789e-4ebf-8cd3-e7807036e8a7\") " Oct 31 04:32:14.834021 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:14.833986 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9240e5d-789e-4ebf-8cd3-e7807036e8a7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9240e5d-789e-4ebf-8cd3-e7807036e8a7-istio-data\") pod \"c9240e5d-789e-4ebf-8cd3-e7807036e8a7\" (UID: \"c9240e5d-789e-4ebf-8cd3-e7807036e8a7\") " Oct 31 04:32:14.834140 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:14.834022 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9240e5d-789e-4ebf-8cd3-e7807036e8a7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9240e5d-789e-4ebf-8cd3-e7807036e8a7-workload-certs\") pod \"c9240e5d-789e-4ebf-8cd3-e7807036e8a7\" (UID: \"c9240e5d-789e-4ebf-8cd3-e7807036e8a7\") " Oct 31 04:32:14.834140 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:14.834076 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9240e5d-789e-4ebf-8cd3-e7807036e8a7-kube-api-access-kx2qd\" (UniqueName: \"kubernetes.io/projected/c9240e5d-789e-4ebf-8cd3-e7807036e8a7-kube-api-access-kx2qd\") pod \"c9240e5d-789e-4ebf-8cd3-e7807036e8a7\" (UID: \"c9240e5d-789e-4ebf-8cd3-e7807036e8a7\") " Oct 31 04:32:14.834249 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:14.834187 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9240e5d-789e-4ebf-8cd3-e7807036e8a7-istio-token\" (UniqueName: \"kubernetes.io/projected/c9240e5d-789e-4ebf-8cd3-e7807036e8a7-istio-token\") pod \"c9240e5d-789e-4ebf-8cd3-e7807036e8a7\" (UID: \"c9240e5d-789e-4ebf-8cd3-e7807036e8a7\") " Oct 31 04:32:14.834249 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:14.834197 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9240e5d-789e-4ebf-8cd3-e7807036e8a7-istiod-ca-cert" pod "c9240e5d-789e-4ebf-8cd3-e7807036e8a7" (UID: "c9240e5d-789e-4ebf-8cd3-e7807036e8a7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:32:14.834373 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:14.834243 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9240e5d-789e-4ebf-8cd3-e7807036e8a7-workload-socket" pod "c9240e5d-789e-4ebf-8cd3-e7807036e8a7" (UID: "c9240e5d-789e-4ebf-8cd3-e7807036e8a7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:32:14.834373 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:14.834309 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9240e5d-789e-4ebf-8cd3-e7807036e8a7-istio-data" pod "c9240e5d-789e-4ebf-8cd3-e7807036e8a7" (UID: "c9240e5d-789e-4ebf-8cd3-e7807036e8a7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:32:14.834373 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:14.834354 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9240e5d-789e-4ebf-8cd3-e7807036e8a7-workload-certs" pod "c9240e5d-789e-4ebf-8cd3-e7807036e8a7" (UID: "c9240e5d-789e-4ebf-8cd3-e7807036e8a7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:32:14.834373 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:14.834355 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9240e5d-789e-4ebf-8cd3-e7807036e8a7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9240e5d-789e-4ebf-8cd3-e7807036e8a7-istio-envoy\") pod \"c9240e5d-789e-4ebf-8cd3-e7807036e8a7\" (UID: \"c9240e5d-789e-4ebf-8cd3-e7807036e8a7\") " Oct 31 04:32:14.834611 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:14.834472 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9240e5d-789e-4ebf-8cd3-e7807036e8a7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9240e5d-789e-4ebf-8cd3-e7807036e8a7-credential-socket\") pod \"c9240e5d-789e-4ebf-8cd3-e7807036e8a7\" (UID: \"c9240e5d-789e-4ebf-8cd3-e7807036e8a7\") " Oct 31 04:32:14.834822 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:14.834793 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9240e5d-789e-4ebf-8cd3-e7807036e8a7-credential-socket" pod "c9240e5d-789e-4ebf-8cd3-e7807036e8a7" (UID: "c9240e5d-789e-4ebf-8cd3-e7807036e8a7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:32:14.835248 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:14.835226 4376 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9240e5d-789e-4ebf-8cd3-e7807036e8a7-workload-socket\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:32:14.835374 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:14.835253 4376 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9240e5d-789e-4ebf-8cd3-e7807036e8a7-istio-data\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:32:14.835374 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:14.835271 4376 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9240e5d-789e-4ebf-8cd3-e7807036e8a7-workload-certs\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:32:14.835374 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:14.835286 4376 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9240e5d-789e-4ebf-8cd3-e7807036e8a7-credential-socket\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:32:14.835374 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:14.835302 4376 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9240e5d-789e-4ebf-8cd3-e7807036e8a7-istiod-ca-cert\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:32:14.836546 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:14.836518 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c9240e5d-789e-4ebf-8cd3-e7807036e8a7-istio-podinfo" pod "c9240e5d-789e-4ebf-8cd3-e7807036e8a7" (UID: "c9240e5d-789e-4ebf-8cd3-e7807036e8a7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 04:32:14.836546 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:14.836536 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9240e5d-789e-4ebf-8cd3-e7807036e8a7-kube-api-access-kx2qd" pod "c9240e5d-789e-4ebf-8cd3-e7807036e8a7" (UID: "c9240e5d-789e-4ebf-8cd3-e7807036e8a7"). InnerVolumeSpecName "kube-api-access-kx2qd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:32:14.836546 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:14.836531 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9240e5d-789e-4ebf-8cd3-e7807036e8a7-istio-envoy" pod "c9240e5d-789e-4ebf-8cd3-e7807036e8a7" (UID: "c9240e5d-789e-4ebf-8cd3-e7807036e8a7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:32:14.836894 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:14.836869 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9240e5d-789e-4ebf-8cd3-e7807036e8a7-istio-token" pod "c9240e5d-789e-4ebf-8cd3-e7807036e8a7" (UID: "c9240e5d-789e-4ebf-8cd3-e7807036e8a7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:32:14.937191 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:14.937144 4376 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9240e5d-789e-4ebf-8cd3-e7807036e8a7-istio-envoy\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:32:14.937191 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:14.937186 4376 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9240e5d-789e-4ebf-8cd3-e7807036e8a7-istio-podinfo\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:32:14.937191 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:14.937197 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kx2qd\" (UniqueName: \"kubernetes.io/projected/c9240e5d-789e-4ebf-8cd3-e7807036e8a7-kube-api-access-kx2qd\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:32:14.937473 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:14.937206 4376 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9240e5d-789e-4ebf-8cd3-e7807036e8a7-istio-token\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:32:15.106151 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:15.106063 4376 generic.go:356] "Generic (PLEG): container finished" podID="c9240e5d-789e-4ebf-8cd3-e7807036e8a7" containerID="921c2507634db695656d081bd28dd097aa6799a82fd730ce384a56c618ea72e8" exitCode=0 Oct 31 04:32:15.106378 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:15.106179 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-d1d441-openshift-default-c79997b97-84mqf" event={"ID":"c9240e5d-789e-4ebf-8cd3-e7807036e8a7","Type":"ContainerDied","Data":"921c2507634db695656d081bd28dd097aa6799a82fd730ce384a56c618ea72e8"} Oct 31 04:32:15.106378 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:15.106297 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-d1d441-openshift-default-c79997b97-84mqf" event={"ID":"c9240e5d-789e-4ebf-8cd3-e7807036e8a7","Type":"ContainerDied","Data":"700364b4a230d6083e8da32331a7c22031ed8fbda65d9e399d713529212bfbaf"} Oct 31 04:32:15.106378 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:15.106344 4376 scope.go:122] "RemoveContainer" containerID="921c2507634db695656d081bd28dd097aa6799a82fd730ce384a56c618ea72e8" Oct 31 04:32:15.106564 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:15.106525 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-d1d441-openshift-default-c79997b97-84mqf" Oct 31 04:32:15.124685 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:15.124664 4376 scope.go:122] "RemoveContainer" containerID="256c492ec07aeb1473d776dac5053a9be1813e76d7b35ba430f04da83e72bb7b" Oct 31 04:32:15.134209 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:15.134187 4376 scope.go:122] "RemoveContainer" containerID="921c2507634db695656d081bd28dd097aa6799a82fd730ce384a56c618ea72e8" Oct 31 04:32:15.134578 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:32:15.134546 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"921c2507634db695656d081bd28dd097aa6799a82fd730ce384a56c618ea72e8\": container with ID starting with 921c2507634db695656d081bd28dd097aa6799a82fd730ce384a56c618ea72e8 not found: ID does not exist" containerID="921c2507634db695656d081bd28dd097aa6799a82fd730ce384a56c618ea72e8" Oct 31 04:32:15.134700 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:15.134586 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"921c2507634db695656d081bd28dd097aa6799a82fd730ce384a56c618ea72e8"} err="failed to get container status \"921c2507634db695656d081bd28dd097aa6799a82fd730ce384a56c618ea72e8\": rpc error: code = NotFound desc = could not find container \"921c2507634db695656d081bd28dd097aa6799a82fd730ce384a56c618ea72e8\": container with ID starting with 921c2507634db695656d081bd28dd097aa6799a82fd730ce384a56c618ea72e8 not found: ID does not exist" Oct 31 04:32:15.134700 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:15.134608 4376 scope.go:122] "RemoveContainer" containerID="256c492ec07aeb1473d776dac5053a9be1813e76d7b35ba430f04da83e72bb7b" Oct 31 04:32:15.134905 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:32:15.134884 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"256c492ec07aeb1473d776dac5053a9be1813e76d7b35ba430f04da83e72bb7b\": container with ID starting with 256c492ec07aeb1473d776dac5053a9be1813e76d7b35ba430f04da83e72bb7b not found: ID does not exist" containerID="256c492ec07aeb1473d776dac5053a9be1813e76d7b35ba430f04da83e72bb7b" Oct 31 04:32:15.134957 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:15.134912 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"256c492ec07aeb1473d776dac5053a9be1813e76d7b35ba430f04da83e72bb7b"} err="failed to get container status \"256c492ec07aeb1473d776dac5053a9be1813e76d7b35ba430f04da83e72bb7b\": rpc error: code = NotFound desc = could not find container \"256c492ec07aeb1473d776dac5053a9be1813e76d7b35ba430f04da83e72bb7b\": container with ID starting with 256c492ec07aeb1473d776dac5053a9be1813e76d7b35ba430f04da83e72bb7b not found: ID does not exist" Oct 31 04:32:15.163806 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:15.163564 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-d1d441-openshift-default-c79997b97-84mqf"] Oct 31 04:32:15.167673 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:15.167646 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-d1d441-openshift-default-c79997b97-84mqf"] Oct 31 04:32:15.935692 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:15.935659 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-dw4vw" Oct 31 04:32:15.936185 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:15.936123 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-dw4vw" Oct 31 04:32:15.937064 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:15.937042 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-dw4vw" Oct 31 04:32:16.026307 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:16.026270 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c9707f-8598fffd47-fcvbn"] Oct 31 04:32:16.026872 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:16.026858 4376 cpu_manager.go:414] "containerMap: removing container" podUID="c9240e5d-789e-4ebf-8cd3-e7807036e8a7" containerName="istio-proxy" Oct 31 04:32:16.026931 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:16.026874 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9240e5d-789e-4ebf-8cd3-e7807036e8a7" containerName="istio-proxy" Oct 31 04:32:16.026931 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:16.026889 4376 cpu_manager.go:414] "containerMap: removing container" podUID="c9240e5d-789e-4ebf-8cd3-e7807036e8a7" containerName="istio-proxy" Oct 31 04:32:16.026931 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:16.026896 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9240e5d-789e-4ebf-8cd3-e7807036e8a7" containerName="istio-proxy" Oct 31 04:32:16.027026 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:16.026986 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9240e5d-789e-4ebf-8cd3-e7807036e8a7" containerName="istio-proxy" Oct 31 04:32:16.027026 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:16.026994 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9240e5d-789e-4ebf-8cd3-e7807036e8a7" containerName="istio-proxy" Oct 31 04:32:16.031517 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:16.031502 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c9707f-8598fffd47-fcvbn" Oct 31 04:32:16.078354 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:16.076513 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-c9707f\"" Oct 31 04:32:16.078354 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:16.076861 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-c9707f-dockercfg-6g442\"" Oct 31 04:32:16.118353 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:16.104705 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f31be489-e267-4087-9b94-86d7d622c841-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-c9707f-8598fffd47-fcvbn\" (UID: \"f31be489-e267-4087-9b94-86d7d622c841\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c9707f-8598fffd47-fcvbn" Oct 31 04:32:16.118353 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:16.104830 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fl7gh\" (UniqueName: \"kubernetes.io/projected/f31be489-e267-4087-9b94-86d7d622c841-kube-api-access-fl7gh\") pod \"payload-pre-processing-e2e-shared-a-w2-c9707f-8598fffd47-fcvbn\" (UID: \"f31be489-e267-4087-9b94-86d7d622c841\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c9707f-8598fffd47-fcvbn" Oct 31 04:32:16.167358 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:16.165375 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-c9707f-5d5859459c-tv5n9"] Oct 31 04:32:16.285684 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:16.284951 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f31be489-e267-4087-9b94-86d7d622c841-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-c9707f-8598fffd47-fcvbn\" (UID: \"f31be489-e267-4087-9b94-86d7d622c841\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c9707f-8598fffd47-fcvbn" Oct 31 04:32:16.304308 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:16.303645 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c9707f-5d5859459c-tv5n9" Oct 31 04:32:16.306380 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:16.306346 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fl7gh\" (UniqueName: \"kubernetes.io/projected/f31be489-e267-4087-9b94-86d7d622c841-kube-api-access-fl7gh\") pod \"payload-pre-processing-e2e-shared-a-w2-c9707f-8598fffd47-fcvbn\" (UID: \"f31be489-e267-4087-9b94-86d7d622c841\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c9707f-8598fffd47-fcvbn" Oct 31 04:32:16.313268 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:16.313075 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f31be489-e267-4087-9b94-86d7d622c841-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-c9707f-8598fffd47-fcvbn\" (UID: \"f31be489-e267-4087-9b94-86d7d622c841\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c9707f-8598fffd47-fcvbn" Oct 31 04:32:16.327845 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:16.327453 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c9707f-8598fffd47-fcvbn"] Oct 31 04:32:16.335403 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:16.335209 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fl7gh\" (UniqueName: \"kubernetes.io/projected/f31be489-e267-4087-9b94-86d7d622c841-kube-api-access-fl7gh\") pod \"payload-pre-processing-e2e-shared-a-w2-c9707f-8598fffd47-fcvbn\" (UID: \"f31be489-e267-4087-9b94-86d7d622c841\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c9707f-8598fffd47-fcvbn" Oct 31 04:32:16.339289 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:16.339260 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-c9707f-5d5859459c-tv5n9"] Oct 31 04:32:16.348810 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:16.347817 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c9707f-8598fffd47-fcvbn" Oct 31 04:32:16.408901 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:16.408808 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe905f47-822c-4bbb-b562-8bd04f2b35a1-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-c9707f-5d5859459c-tv5n9\" (UID: \"fe905f47-822c-4bbb-b562-8bd04f2b35a1\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c9707f-5d5859459c-tv5n9" Oct 31 04:32:16.411418 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:16.411386 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpwrb\" (UniqueName: \"kubernetes.io/projected/fe905f47-822c-4bbb-b562-8bd04f2b35a1-kube-api-access-jpwrb\") pod \"payload-processing-e2e-shared-a-w2-c9707f-5d5859459c-tv5n9\" (UID: \"fe905f47-822c-4bbb-b562-8bd04f2b35a1\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c9707f-5d5859459c-tv5n9" Oct 31 04:32:16.446102 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:16.446069 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9240e5d-789e-4ebf-8cd3-e7807036e8a7" path="/var/lib/kubelet/pods/c9240e5d-789e-4ebf-8cd3-e7807036e8a7/volumes" Oct 31 04:32:16.512521 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:16.512487 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpwrb\" (UniqueName: \"kubernetes.io/projected/fe905f47-822c-4bbb-b562-8bd04f2b35a1-kube-api-access-jpwrb\") pod \"payload-processing-e2e-shared-a-w2-c9707f-5d5859459c-tv5n9\" (UID: \"fe905f47-822c-4bbb-b562-8bd04f2b35a1\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c9707f-5d5859459c-tv5n9" Oct 31 04:32:16.512745 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:16.512723 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe905f47-822c-4bbb-b562-8bd04f2b35a1-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-c9707f-5d5859459c-tv5n9\" (UID: \"fe905f47-822c-4bbb-b562-8bd04f2b35a1\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c9707f-5d5859459c-tv5n9" Oct 31 04:32:16.513476 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:16.513459 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe905f47-822c-4bbb-b562-8bd04f2b35a1-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-c9707f-5d5859459c-tv5n9\" (UID: \"fe905f47-822c-4bbb-b562-8bd04f2b35a1\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c9707f-5d5859459c-tv5n9" Oct 31 04:32:16.537153 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:16.537081 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpwrb\" (UniqueName: \"kubernetes.io/projected/fe905f47-822c-4bbb-b562-8bd04f2b35a1-kube-api-access-jpwrb\") pod \"payload-processing-e2e-shared-a-w2-c9707f-5d5859459c-tv5n9\" (UID: \"fe905f47-822c-4bbb-b562-8bd04f2b35a1\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c9707f-5d5859459c-tv5n9" Oct 31 04:32:16.630643 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:16.630604 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c9707f-5d5859459c-tv5n9" Oct 31 04:32:16.660142 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:16.660115 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c9707f-8598fffd47-fcvbn"] Oct 31 04:32:16.938644 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:32:16.938606 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe905f47_822c_4bbb_b562_8bd04f2b35a1.slice/crio-ff28eab2535966dced62a2831668816b2517e7f5fd1a7f02f4f8f6e0e7e2f2eb WatchSource:0}: Error finding container ff28eab2535966dced62a2831668816b2517e7f5fd1a7f02f4f8f6e0e7e2f2eb: Status 404 returned error can't find the container with id ff28eab2535966dced62a2831668816b2517e7f5fd1a7f02f4f8f6e0e7e2f2eb Oct 31 04:32:16.939934 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:16.939910 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-c9707f-5d5859459c-tv5n9"] Oct 31 04:32:17.335285 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:17.335190 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c9707f-5d5859459c-tv5n9" event={"ID":"fe905f47-822c-4bbb-b562-8bd04f2b35a1","Type":"ContainerStarted","Data":"de1f895f9dd3e6d776886171b1813cd34b3987f58de5bbd7ffcfc00dc5bedc84"} Oct 31 04:32:17.335285 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:17.335262 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c9707f-5d5859459c-tv5n9" Oct 31 04:32:17.335531 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:17.335289 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c9707f-5d5859459c-tv5n9" event={"ID":"fe905f47-822c-4bbb-b562-8bd04f2b35a1","Type":"ContainerStarted","Data":"ff28eab2535966dced62a2831668816b2517e7f5fd1a7f02f4f8f6e0e7e2f2eb"} Oct 31 04:32:17.337192 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:17.337165 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c9707f-8598fffd47-fcvbn" event={"ID":"f31be489-e267-4087-9b94-86d7d622c841","Type":"ContainerStarted","Data":"245441bd30c6b7857276850e7acae00ac12f1bd56db09eb5521ead8cfc94b753"} Oct 31 04:32:17.337393 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:17.337359 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c9707f-8598fffd47-fcvbn" Oct 31 04:32:17.337555 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:17.337405 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c9707f-8598fffd47-fcvbn" event={"ID":"f31be489-e267-4087-9b94-86d7d622c841","Type":"ContainerStarted","Data":"b1ef1d5aaf7e15bdd66ee3f73bb8c1ad3a6fd1f4ad79c4d16c34d98f949b13b3"} Oct 31 04:32:17.377642 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:17.377584 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c9707f-5d5859459c-tv5n9" podStartSLOduration=1.377571733 podStartE2EDuration="1.377571733s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:32:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:32:17.376895539 +0000 UTC m=+1685.091230157" watchObservedRunningTime="2026-10-31 04:32:17.377571733 +0000 UTC m=+1685.091906349" Oct 31 04:32:17.421138 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:17.421082 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c9707f-8598fffd47-fcvbn" podStartSLOduration=2.421065969 podStartE2EDuration="2.421065969s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:32:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:32:17.418227553 +0000 UTC m=+1685.132562171" watchObservedRunningTime="2026-10-31 04:32:17.421065969 +0000 UTC m=+1685.135400586" Oct 31 04:32:20.386543 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:20.386514 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-81c810-5749ff85ff-5fr52"] Oct 31 04:32:21.878626 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:21.878589 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 04:32:22.864859 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:22.864825 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-e6a0de-5b64759d6c-qqgln"] Oct 31 04:32:22.906180 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:22.906152 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-e6a0de-5987ffc46c-4mxw7"] Oct 31 04:32:25.388382 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:25.388289 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-81c810-5749ff85ff-5fr52" podUID="c2e98cfe-b0c2-420e-8b92-540de74cd07f" containerName="payload-processing" containerID="cri-o://ba3dcb8b2618c605e83785f85594c60ad78cb2266701d357c65940ce4110001d" gracePeriod=25 Oct 31 04:32:25.892537 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:25.892509 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-81c810-5749ff85ff-5fr52" Oct 31 04:32:25.980206 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:25.980135 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2e98cfe-b0c2-420e-8b92-540de74cd07f-kube-api-access-pl7z7\" (UniqueName: \"kubernetes.io/projected/c2e98cfe-b0c2-420e-8b92-540de74cd07f-kube-api-access-pl7z7\") pod \"c2e98cfe-b0c2-420e-8b92-540de74cd07f\" (UID: \"c2e98cfe-b0c2-420e-8b92-540de74cd07f\") " Oct 31 04:32:25.980452 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:25.980429 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2e98cfe-b0c2-420e-8b92-540de74cd07f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2e98cfe-b0c2-420e-8b92-540de74cd07f-plugins-config-volume\") pod \"c2e98cfe-b0c2-420e-8b92-540de74cd07f\" (UID: \"c2e98cfe-b0c2-420e-8b92-540de74cd07f\") " Oct 31 04:32:25.981636 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:25.981611 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2e98cfe-b0c2-420e-8b92-540de74cd07f-plugins-config-volume" pod "c2e98cfe-b0c2-420e-8b92-540de74cd07f" (UID: "c2e98cfe-b0c2-420e-8b92-540de74cd07f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:32:25.983737 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:25.983704 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2e98cfe-b0c2-420e-8b92-540de74cd07f-kube-api-access-pl7z7" pod "c2e98cfe-b0c2-420e-8b92-540de74cd07f" (UID: "c2e98cfe-b0c2-420e-8b92-540de74cd07f"). InnerVolumeSpecName "kube-api-access-pl7z7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:32:26.082935 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:26.082889 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pl7z7\" (UniqueName: \"kubernetes.io/projected/c2e98cfe-b0c2-420e-8b92-540de74cd07f-kube-api-access-pl7z7\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:32:26.082935 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:26.082924 4376 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2e98cfe-b0c2-420e-8b92-540de74cd07f-plugins-config-volume\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:32:26.571826 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:26.571780 4376 generic.go:356] "Generic (PLEG): container finished" podID="c2e98cfe-b0c2-420e-8b92-540de74cd07f" containerID="ba3dcb8b2618c605e83785f85594c60ad78cb2266701d357c65940ce4110001d" exitCode=0 Oct 31 04:32:26.571826 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:26.571829 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-81c810-5749ff85ff-5fr52" Oct 31 04:32:26.572380 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:26.571863 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-81c810-5749ff85ff-5fr52" event={"ID":"c2e98cfe-b0c2-420e-8b92-540de74cd07f","Type":"ContainerDied","Data":"ba3dcb8b2618c605e83785f85594c60ad78cb2266701d357c65940ce4110001d"} Oct 31 04:32:26.572380 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:26.571923 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-81c810-5749ff85ff-5fr52" event={"ID":"c2e98cfe-b0c2-420e-8b92-540de74cd07f","Type":"ContainerDied","Data":"3e812823b3379a76df8cac91a73162bdbfe7ffdcdf8f2dba0990c2008276c978"} Oct 31 04:32:26.572380 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:26.571952 4376 scope.go:122] "RemoveContainer" containerID="ba3dcb8b2618c605e83785f85594c60ad78cb2266701d357c65940ce4110001d" Oct 31 04:32:26.581686 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:26.581665 4376 scope.go:122] "RemoveContainer" containerID="ba3dcb8b2618c605e83785f85594c60ad78cb2266701d357c65940ce4110001d" Oct 31 04:32:26.581966 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:32:26.581943 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba3dcb8b2618c605e83785f85594c60ad78cb2266701d357c65940ce4110001d\": container with ID starting with ba3dcb8b2618c605e83785f85594c60ad78cb2266701d357c65940ce4110001d not found: ID does not exist" containerID="ba3dcb8b2618c605e83785f85594c60ad78cb2266701d357c65940ce4110001d" Oct 31 04:32:26.582027 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:26.581978 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba3dcb8b2618c605e83785f85594c60ad78cb2266701d357c65940ce4110001d"} err="failed to get container status \"ba3dcb8b2618c605e83785f85594c60ad78cb2266701d357c65940ce4110001d\": rpc error: code = NotFound desc = could not find container \"ba3dcb8b2618c605e83785f85594c60ad78cb2266701d357c65940ce4110001d\": container with ID starting with ba3dcb8b2618c605e83785f85594c60ad78cb2266701d357c65940ce4110001d not found: ID does not exist" Oct 31 04:32:26.626015 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:26.625975 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-81c810-5749ff85ff-5fr52"] Oct 31 04:32:26.638681 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:26.638651 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-81c810-5749ff85ff-5fr52"] Oct 31 04:32:26.832006 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:26.831923 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wmkf5"] Oct 31 04:32:26.832175 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:26.832135 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wmkf5" podUID="d135a318-172d-4408-8e38-ca0e26946381" containerName="istio-proxy" containerID="cri-o://74946a5a9a9631dd0fafccc2fb98e2ade83a6269ccfcbd538e533d04741f061b" gracePeriod=30 Oct 31 04:32:27.865426 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:27.865379 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-e6a0de-5b64759d6c-qqgln" podUID="dfd97100-7d93-4575-bb69-43e6f8f462d9" containerName="payload-processing" containerID="cri-o://2803e9d05fc94dbd96f6099e2e8708abfc95a19535bdf425791ad77c9cb565fc" gracePeriod=25 Oct 31 04:32:27.906503 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:27.906440 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e6a0de-5987ffc46c-4mxw7" podUID="7a3b2925-c12d-4014-9781-067ed8da9b1e" containerName="payload-pre-processing" containerID="cri-o://0dc2d5283b97c25821054f79c4b38d4620e0db700ce79ba4c6e6b427037aeff7" gracePeriod=25 Oct 31 04:32:28.206463 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:28.206435 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-e6a0de-5b64759d6c-qqgln" Oct 31 04:32:28.242390 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:28.242360 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e6a0de-5987ffc46c-4mxw7" Oct 31 04:32:28.315142 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:28.315102 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a3b2925-c12d-4014-9781-067ed8da9b1e-kube-api-access-svd9w\" (UniqueName: \"kubernetes.io/projected/7a3b2925-c12d-4014-9781-067ed8da9b1e-kube-api-access-svd9w\") pod \"7a3b2925-c12d-4014-9781-067ed8da9b1e\" (UID: \"7a3b2925-c12d-4014-9781-067ed8da9b1e\") " Oct 31 04:32:28.315411 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:28.315171 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfd97100-7d93-4575-bb69-43e6f8f462d9-kube-api-access-km5f6\" (UniqueName: \"kubernetes.io/projected/dfd97100-7d93-4575-bb69-43e6f8f462d9-kube-api-access-km5f6\") pod \"dfd97100-7d93-4575-bb69-43e6f8f462d9\" (UID: \"dfd97100-7d93-4575-bb69-43e6f8f462d9\") " Oct 31 04:32:28.315411 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:28.315208 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a3b2925-c12d-4014-9781-067ed8da9b1e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a3b2925-c12d-4014-9781-067ed8da9b1e-plugins-config-volume\") pod \"7a3b2925-c12d-4014-9781-067ed8da9b1e\" (UID: \"7a3b2925-c12d-4014-9781-067ed8da9b1e\") " Oct 31 04:32:28.315411 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:28.315386 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dfd97100-7d93-4575-bb69-43e6f8f462d9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfd97100-7d93-4575-bb69-43e6f8f462d9-plugins-config-volume\") pod \"dfd97100-7d93-4575-bb69-43e6f8f462d9\" (UID: \"dfd97100-7d93-4575-bb69-43e6f8f462d9\") " Oct 31 04:32:28.316001 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:28.315968 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a3b2925-c12d-4014-9781-067ed8da9b1e-plugins-config-volume" pod "7a3b2925-c12d-4014-9781-067ed8da9b1e" (UID: "7a3b2925-c12d-4014-9781-067ed8da9b1e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:32:28.316001 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:28.315983 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfd97100-7d93-4575-bb69-43e6f8f462d9-plugins-config-volume" pod "dfd97100-7d93-4575-bb69-43e6f8f462d9" (UID: "dfd97100-7d93-4575-bb69-43e6f8f462d9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:32:28.316302 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:28.316279 4376 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfd97100-7d93-4575-bb69-43e6f8f462d9-plugins-config-volume\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:32:28.316408 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:28.316308 4376 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a3b2925-c12d-4014-9781-067ed8da9b1e-plugins-config-volume\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:32:28.317616 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:28.317595 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a3b2925-c12d-4014-9781-067ed8da9b1e-kube-api-access-svd9w" pod "7a3b2925-c12d-4014-9781-067ed8da9b1e" (UID: "7a3b2925-c12d-4014-9781-067ed8da9b1e"). InnerVolumeSpecName "kube-api-access-svd9w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:32:28.318490 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:28.318466 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfd97100-7d93-4575-bb69-43e6f8f462d9-kube-api-access-km5f6" pod "dfd97100-7d93-4575-bb69-43e6f8f462d9" (UID: "dfd97100-7d93-4575-bb69-43e6f8f462d9"). InnerVolumeSpecName "kube-api-access-km5f6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:32:28.347202 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:28.347180 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c9707f-8598fffd47-fcvbn" Oct 31 04:32:28.347303 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:28.347241 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c9707f-5d5859459c-tv5n9" Oct 31 04:32:28.417876 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:28.417730 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-svd9w\" (UniqueName: \"kubernetes.io/projected/7a3b2925-c12d-4014-9781-067ed8da9b1e-kube-api-access-svd9w\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:32:28.417876 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:28.417762 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-km5f6\" (UniqueName: \"kubernetes.io/projected/dfd97100-7d93-4575-bb69-43e6f8f462d9-kube-api-access-km5f6\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:32:28.455400 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:28.455364 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2e98cfe-b0c2-420e-8b92-540de74cd07f" path="/var/lib/kubelet/pods/c2e98cfe-b0c2-420e-8b92-540de74cd07f/volumes" Oct 31 04:32:28.587550 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:28.587514 4376 generic.go:356] "Generic (PLEG): container finished" podID="7a3b2925-c12d-4014-9781-067ed8da9b1e" containerID="0dc2d5283b97c25821054f79c4b38d4620e0db700ce79ba4c6e6b427037aeff7" exitCode=0 Oct 31 04:32:28.587723 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:28.587575 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e6a0de-5987ffc46c-4mxw7" Oct 31 04:32:28.587723 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:28.587602 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e6a0de-5987ffc46c-4mxw7" event={"ID":"7a3b2925-c12d-4014-9781-067ed8da9b1e","Type":"ContainerDied","Data":"0dc2d5283b97c25821054f79c4b38d4620e0db700ce79ba4c6e6b427037aeff7"} Oct 31 04:32:28.587723 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:28.587661 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e6a0de-5987ffc46c-4mxw7" event={"ID":"7a3b2925-c12d-4014-9781-067ed8da9b1e","Type":"ContainerDied","Data":"b1d60d5364e2ed367892fb8e31aab83de42946734733a4d923ab9fdd775d8e1c"} Oct 31 04:32:28.587723 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:28.587701 4376 scope.go:122] "RemoveContainer" containerID="0dc2d5283b97c25821054f79c4b38d4620e0db700ce79ba4c6e6b427037aeff7" Oct 31 04:32:28.589168 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:28.589147 4376 generic.go:356] "Generic (PLEG): container finished" podID="dfd97100-7d93-4575-bb69-43e6f8f462d9" containerID="2803e9d05fc94dbd96f6099e2e8708abfc95a19535bdf425791ad77c9cb565fc" exitCode=0 Oct 31 04:32:28.589280 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:28.589189 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-e6a0de-5b64759d6c-qqgln" event={"ID":"dfd97100-7d93-4575-bb69-43e6f8f462d9","Type":"ContainerDied","Data":"2803e9d05fc94dbd96f6099e2e8708abfc95a19535bdf425791ad77c9cb565fc"} Oct 31 04:32:28.589280 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:28.589227 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-e6a0de-5b64759d6c-qqgln" Oct 31 04:32:28.589400 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:28.589232 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-e6a0de-5b64759d6c-qqgln" event={"ID":"dfd97100-7d93-4575-bb69-43e6f8f462d9","Type":"ContainerDied","Data":"c107a3087bb1266bce0eb83826fb841b1f5bb8caf73f03533aab30c3dc50525e"} Oct 31 04:32:28.597681 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:28.597662 4376 scope.go:122] "RemoveContainer" containerID="0dc2d5283b97c25821054f79c4b38d4620e0db700ce79ba4c6e6b427037aeff7" Oct 31 04:32:28.597961 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:32:28.597939 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0dc2d5283b97c25821054f79c4b38d4620e0db700ce79ba4c6e6b427037aeff7\": container with ID starting with 0dc2d5283b97c25821054f79c4b38d4620e0db700ce79ba4c6e6b427037aeff7 not found: ID does not exist" containerID="0dc2d5283b97c25821054f79c4b38d4620e0db700ce79ba4c6e6b427037aeff7" Oct 31 04:32:28.598043 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:28.597967 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0dc2d5283b97c25821054f79c4b38d4620e0db700ce79ba4c6e6b427037aeff7"} err="failed to get container status \"0dc2d5283b97c25821054f79c4b38d4620e0db700ce79ba4c6e6b427037aeff7\": rpc error: code = NotFound desc = could not find container \"0dc2d5283b97c25821054f79c4b38d4620e0db700ce79ba4c6e6b427037aeff7\": container with ID starting with 0dc2d5283b97c25821054f79c4b38d4620e0db700ce79ba4c6e6b427037aeff7 not found: ID does not exist" Oct 31 04:32:28.598043 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:28.597986 4376 scope.go:122] "RemoveContainer" containerID="2803e9d05fc94dbd96f6099e2e8708abfc95a19535bdf425791ad77c9cb565fc" Oct 31 04:32:28.606385 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:28.606362 4376 scope.go:122] "RemoveContainer" containerID="2803e9d05fc94dbd96f6099e2e8708abfc95a19535bdf425791ad77c9cb565fc" Oct 31 04:32:28.606636 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:32:28.606620 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2803e9d05fc94dbd96f6099e2e8708abfc95a19535bdf425791ad77c9cb565fc\": container with ID starting with 2803e9d05fc94dbd96f6099e2e8708abfc95a19535bdf425791ad77c9cb565fc not found: ID does not exist" containerID="2803e9d05fc94dbd96f6099e2e8708abfc95a19535bdf425791ad77c9cb565fc" Oct 31 04:32:28.606701 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:28.606643 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2803e9d05fc94dbd96f6099e2e8708abfc95a19535bdf425791ad77c9cb565fc"} err="failed to get container status \"2803e9d05fc94dbd96f6099e2e8708abfc95a19535bdf425791ad77c9cb565fc\": rpc error: code = NotFound desc = could not find container \"2803e9d05fc94dbd96f6099e2e8708abfc95a19535bdf425791ad77c9cb565fc\": container with ID starting with 2803e9d05fc94dbd96f6099e2e8708abfc95a19535bdf425791ad77c9cb565fc not found: ID does not exist" Oct 31 04:32:28.642184 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:28.642158 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-e6a0de-5b64759d6c-qqgln"] Oct 31 04:32:28.649039 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:28.649016 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-e6a0de-5b64759d6c-qqgln"] Oct 31 04:32:28.683722 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:28.683665 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-e6a0de-5987ffc46c-4mxw7"] Oct 31 04:32:28.687277 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:28.687256 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-e6a0de-5987ffc46c-4mxw7"] Oct 31 04:32:29.611886 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:29.611855 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-57dc6d-66998bfb9f-tdltw"] Oct 31 04:32:29.612449 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:29.612434 4376 cpu_manager.go:414] "containerMap: removing container" podUID="dfd97100-7d93-4575-bb69-43e6f8f462d9" containerName="payload-processing" Oct 31 04:32:29.612527 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:29.612450 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfd97100-7d93-4575-bb69-43e6f8f462d9" containerName="payload-processing" Oct 31 04:32:29.612527 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:29.612473 4376 cpu_manager.go:414] "containerMap: removing container" podUID="7a3b2925-c12d-4014-9781-067ed8da9b1e" containerName="payload-pre-processing" Oct 31 04:32:29.612527 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:29.612478 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a3b2925-c12d-4014-9781-067ed8da9b1e" containerName="payload-pre-processing" Oct 31 04:32:29.612527 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:29.612497 4376 cpu_manager.go:414] "containerMap: removing container" podUID="c2e98cfe-b0c2-420e-8b92-540de74cd07f" containerName="payload-processing" Oct 31 04:32:29.612527 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:29.612502 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2e98cfe-b0c2-420e-8b92-540de74cd07f" containerName="payload-processing" Oct 31 04:32:29.612687 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:29.612608 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfd97100-7d93-4575-bb69-43e6f8f462d9" containerName="payload-processing" Oct 31 04:32:29.612687 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:29.612618 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a3b2925-c12d-4014-9781-067ed8da9b1e" containerName="payload-pre-processing" Oct 31 04:32:29.612687 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:29.612627 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2e98cfe-b0c2-420e-8b92-540de74cd07f" containerName="payload-processing" Oct 31 04:32:29.617190 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:29.617164 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-57dc6d-66998bfb9f-tdltw" Oct 31 04:32:29.633982 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:29.633955 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-57dc6d-66998bfb9f-tdltw"] Oct 31 04:32:29.634116 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:29.634013 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-57dc6d\"" Oct 31 04:32:29.634198 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:29.634129 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w4-57dc6d-dockercfg-lr92d\"" Oct 31 04:32:29.778416 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:29.778384 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-57dc6d-5675c77f6c-8t8zt"] Oct 31 04:32:29.892163 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:29.886594 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-57dc6d-5675c77f6c-8t8zt"] Oct 31 04:32:29.892163 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:29.886719 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-57dc6d-5675c77f6c-8t8zt" Oct 31 04:32:29.918514 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:29.918473 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4beb0fb-a6b6-4864-a65a-38e9e3a9c05f-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-57dc6d-5675c77f6c-8t8zt\" (UID: \"a4beb0fb-a6b6-4864-a65a-38e9e3a9c05f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-57dc6d-5675c77f6c-8t8zt" Oct 31 04:32:29.918737 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:29.918719 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwc4j\" (UniqueName: \"kubernetes.io/projected/a4beb0fb-a6b6-4864-a65a-38e9e3a9c05f-kube-api-access-jwc4j\") pod \"payload-processing-e2e-shared-a-w4-57dc6d-5675c77f6c-8t8zt\" (UID: \"a4beb0fb-a6b6-4864-a65a-38e9e3a9c05f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-57dc6d-5675c77f6c-8t8zt" Oct 31 04:32:29.918845 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:29.918831 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmkgj\" (UniqueName: \"kubernetes.io/projected/07764cf3-3c73-4f06-82c6-7df4d5e9955a-kube-api-access-qmkgj\") pod \"payload-pre-processing-e2e-shared-a-w4-57dc6d-66998bfb9f-tdltw\" (UID: \"07764cf3-3c73-4f06-82c6-7df4d5e9955a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-57dc6d-66998bfb9f-tdltw" Oct 31 04:32:29.919221 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:29.919186 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07764cf3-3c73-4f06-82c6-7df4d5e9955a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-57dc6d-66998bfb9f-tdltw\" (UID: \"07764cf3-3c73-4f06-82c6-7df4d5e9955a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-57dc6d-66998bfb9f-tdltw" Oct 31 04:32:30.021011 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:30.020980 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4beb0fb-a6b6-4864-a65a-38e9e3a9c05f-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-57dc6d-5675c77f6c-8t8zt\" (UID: \"a4beb0fb-a6b6-4864-a65a-38e9e3a9c05f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-57dc6d-5675c77f6c-8t8zt" Oct 31 04:32:30.021219 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:30.021020 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwc4j\" (UniqueName: \"kubernetes.io/projected/a4beb0fb-a6b6-4864-a65a-38e9e3a9c05f-kube-api-access-jwc4j\") pod \"payload-processing-e2e-shared-a-w4-57dc6d-5675c77f6c-8t8zt\" (UID: \"a4beb0fb-a6b6-4864-a65a-38e9e3a9c05f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-57dc6d-5675c77f6c-8t8zt" Oct 31 04:32:30.021219 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:30.021144 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmkgj\" (UniqueName: \"kubernetes.io/projected/07764cf3-3c73-4f06-82c6-7df4d5e9955a-kube-api-access-qmkgj\") pod \"payload-pre-processing-e2e-shared-a-w4-57dc6d-66998bfb9f-tdltw\" (UID: \"07764cf3-3c73-4f06-82c6-7df4d5e9955a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-57dc6d-66998bfb9f-tdltw" Oct 31 04:32:30.021436 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:30.021415 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07764cf3-3c73-4f06-82c6-7df4d5e9955a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-57dc6d-66998bfb9f-tdltw\" (UID: \"07764cf3-3c73-4f06-82c6-7df4d5e9955a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-57dc6d-66998bfb9f-tdltw" Oct 31 04:32:30.021838 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:30.021817 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4beb0fb-a6b6-4864-a65a-38e9e3a9c05f-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-57dc6d-5675c77f6c-8t8zt\" (UID: \"a4beb0fb-a6b6-4864-a65a-38e9e3a9c05f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-57dc6d-5675c77f6c-8t8zt" Oct 31 04:32:30.022119 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:30.022098 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07764cf3-3c73-4f06-82c6-7df4d5e9955a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-57dc6d-66998bfb9f-tdltw\" (UID: \"07764cf3-3c73-4f06-82c6-7df4d5e9955a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-57dc6d-66998bfb9f-tdltw" Oct 31 04:32:30.044927 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:30.044906 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmkgj\" (UniqueName: \"kubernetes.io/projected/07764cf3-3c73-4f06-82c6-7df4d5e9955a-kube-api-access-qmkgj\") pod \"payload-pre-processing-e2e-shared-a-w4-57dc6d-66998bfb9f-tdltw\" (UID: \"07764cf3-3c73-4f06-82c6-7df4d5e9955a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-57dc6d-66998bfb9f-tdltw" Oct 31 04:32:30.045038 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:30.044949 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwc4j\" (UniqueName: \"kubernetes.io/projected/a4beb0fb-a6b6-4864-a65a-38e9e3a9c05f-kube-api-access-jwc4j\") pod \"payload-processing-e2e-shared-a-w4-57dc6d-5675c77f6c-8t8zt\" (UID: \"a4beb0fb-a6b6-4864-a65a-38e9e3a9c05f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-57dc6d-5675c77f6c-8t8zt" Oct 31 04:32:30.208525 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:30.208451 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-57dc6d-5675c77f6c-8t8zt" Oct 31 04:32:30.233152 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:30.233123 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-57dc6d-66998bfb9f-tdltw" Oct 31 04:32:30.444179 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:30.444135 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a3b2925-c12d-4014-9781-067ed8da9b1e" path="/var/lib/kubelet/pods/7a3b2925-c12d-4014-9781-067ed8da9b1e/volumes" Oct 31 04:32:30.444609 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:30.444594 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dfd97100-7d93-4575-bb69-43e6f8f462d9" path="/var/lib/kubelet/pods/dfd97100-7d93-4575-bb69-43e6f8f462d9/volumes" Oct 31 04:32:30.511560 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:32:30.511528 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4beb0fb_a6b6_4864_a65a_38e9e3a9c05f.slice/crio-9473677f0de2b838df8df6eb63c2a39699da8e392e3d2187e181c50ba12d951e WatchSource:0}: Error finding container 9473677f0de2b838df8df6eb63c2a39699da8e392e3d2187e181c50ba12d951e: Status 404 returned error can't find the container with id 9473677f0de2b838df8df6eb63c2a39699da8e392e3d2187e181c50ba12d951e Oct 31 04:32:30.513407 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:30.513289 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-57dc6d-5675c77f6c-8t8zt"] Oct 31 04:32:30.544819 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:32:30.544621 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07764cf3_3c73_4f06_82c6_7df4d5e9955a.slice/crio-dee0eecf1c5f019fe0614dac843a0a3db9a11555266b5fd0971c61d1cd535571 WatchSource:0}: Error finding container dee0eecf1c5f019fe0614dac843a0a3db9a11555266b5fd0971c61d1cd535571: Status 404 returned error can't find the container with id dee0eecf1c5f019fe0614dac843a0a3db9a11555266b5fd0971c61d1cd535571 Oct 31 04:32:30.545020 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:30.544990 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-57dc6d-66998bfb9f-tdltw"] Oct 31 04:32:30.605552 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:30.605525 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-57dc6d-5675c77f6c-8t8zt" event={"ID":"a4beb0fb-a6b6-4864-a65a-38e9e3a9c05f","Type":"ContainerStarted","Data":"1758d081570b8116946a169c4a5db9527b35b5cecea86452c37517f52ca46260"} Oct 31 04:32:30.605684 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:30.605584 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-57dc6d-5675c77f6c-8t8zt" event={"ID":"a4beb0fb-a6b6-4864-a65a-38e9e3a9c05f","Type":"ContainerStarted","Data":"9473677f0de2b838df8df6eb63c2a39699da8e392e3d2187e181c50ba12d951e"} Oct 31 04:32:30.605750 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:30.605721 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-57dc6d-5675c77f6c-8t8zt" Oct 31 04:32:30.606800 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:30.606773 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-57dc6d-66998bfb9f-tdltw" event={"ID":"07764cf3-3c73-4f06-82c6-7df4d5e9955a","Type":"ContainerStarted","Data":"dee0eecf1c5f019fe0614dac843a0a3db9a11555266b5fd0971c61d1cd535571"} Oct 31 04:32:30.972679 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:30.972627 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-57dc6d-5675c77f6c-8t8zt" podStartSLOduration=1.9726100130000002 podStartE2EDuration="1.972610013s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:32:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:32:30.660603721 +0000 UTC m=+1698.374938342" watchObservedRunningTime="2026-10-31 04:32:30.972610013 +0000 UTC m=+1698.686944627" Oct 31 04:32:30.976007 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:30.975984 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c9707f-d9cc59bdd-xz2sb"] Oct 31 04:32:30.980901 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:30.980878 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c9707f-d9cc59bdd-xz2sb" Oct 31 04:32:30.996024 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:30.995984 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-c9707f-serving-cert\"" Oct 31 04:32:30.998250 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:30.998222 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c9707f-d9cc59bdd-xz2sb"] Oct 31 04:32:31.033744 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:31.033705 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/65edcc7f-0600-4161-9ba1-89e26f064dd4-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-c9707f-d9cc59bdd-xz2sb\" (UID: \"65edcc7f-0600-4161-9ba1-89e26f064dd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c9707f-d9cc59bdd-xz2sb" Oct 31 04:32:31.034041 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:31.033950 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtzq6\" (UniqueName: \"kubernetes.io/projected/65edcc7f-0600-4161-9ba1-89e26f064dd4-kube-api-access-jtzq6\") pod \"maas-api-e2e-shared-a-w2-c9707f-d9cc59bdd-xz2sb\" (UID: \"65edcc7f-0600-4161-9ba1-89e26f064dd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c9707f-d9cc59bdd-xz2sb" Oct 31 04:32:31.034113 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:31.034099 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/65edcc7f-0600-4161-9ba1-89e26f064dd4-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-c9707f-d9cc59bdd-xz2sb\" (UID: \"65edcc7f-0600-4161-9ba1-89e26f064dd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c9707f-d9cc59bdd-xz2sb" Oct 31 04:32:31.034229 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:31.034187 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/65edcc7f-0600-4161-9ba1-89e26f064dd4-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-c9707f-d9cc59bdd-xz2sb\" (UID: \"65edcc7f-0600-4161-9ba1-89e26f064dd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c9707f-d9cc59bdd-xz2sb" Oct 31 04:32:31.135461 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:31.135428 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/65edcc7f-0600-4161-9ba1-89e26f064dd4-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-c9707f-d9cc59bdd-xz2sb\" (UID: \"65edcc7f-0600-4161-9ba1-89e26f064dd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c9707f-d9cc59bdd-xz2sb" Oct 31 04:32:31.135461 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:31.135465 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/65edcc7f-0600-4161-9ba1-89e26f064dd4-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-c9707f-d9cc59bdd-xz2sb\" (UID: \"65edcc7f-0600-4161-9ba1-89e26f064dd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c9707f-d9cc59bdd-xz2sb" Oct 31 04:32:31.135679 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:31.135527 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/65edcc7f-0600-4161-9ba1-89e26f064dd4-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-c9707f-d9cc59bdd-xz2sb\" (UID: \"65edcc7f-0600-4161-9ba1-89e26f064dd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c9707f-d9cc59bdd-xz2sb" Oct 31 04:32:31.135720 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:31.135679 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtzq6\" (UniqueName: \"kubernetes.io/projected/65edcc7f-0600-4161-9ba1-89e26f064dd4-kube-api-access-jtzq6\") pod \"maas-api-e2e-shared-a-w2-c9707f-d9cc59bdd-xz2sb\" (UID: \"65edcc7f-0600-4161-9ba1-89e26f064dd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c9707f-d9cc59bdd-xz2sb" Oct 31 04:32:31.136218 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:31.136171 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/65edcc7f-0600-4161-9ba1-89e26f064dd4-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-c9707f-d9cc59bdd-xz2sb\" (UID: \"65edcc7f-0600-4161-9ba1-89e26f064dd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c9707f-d9cc59bdd-xz2sb" Oct 31 04:32:31.137844 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:31.137824 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/65edcc7f-0600-4161-9ba1-89e26f064dd4-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-c9707f-d9cc59bdd-xz2sb\" (UID: \"65edcc7f-0600-4161-9ba1-89e26f064dd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c9707f-d9cc59bdd-xz2sb" Oct 31 04:32:31.137940 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:31.137899 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/65edcc7f-0600-4161-9ba1-89e26f064dd4-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-c9707f-d9cc59bdd-xz2sb\" (UID: \"65edcc7f-0600-4161-9ba1-89e26f064dd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c9707f-d9cc59bdd-xz2sb" Oct 31 04:32:31.158999 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:31.158976 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtzq6\" (UniqueName: \"kubernetes.io/projected/65edcc7f-0600-4161-9ba1-89e26f064dd4-kube-api-access-jtzq6\") pod \"maas-api-e2e-shared-a-w2-c9707f-d9cc59bdd-xz2sb\" (UID: \"65edcc7f-0600-4161-9ba1-89e26f064dd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c9707f-d9cc59bdd-xz2sb" Oct 31 04:32:31.295939 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:31.295606 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c9707f-d9cc59bdd-xz2sb" Oct 31 04:32:31.616818 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:31.616773 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-57dc6d-66998bfb9f-tdltw" event={"ID":"07764cf3-3c73-4f06-82c6-7df4d5e9955a","Type":"ContainerStarted","Data":"cf5938a4da3a2d3c07bd214995e57d887657c35f23802f7eeaa0d0cd98b8be1c"} Oct 31 04:32:31.616988 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:31.616844 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-57dc6d-66998bfb9f-tdltw" Oct 31 04:32:31.628418 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:32:31.628392 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65edcc7f_0600_4161_9ba1_89e26f064dd4.slice/crio-921ef8723867ec83929d8aa33f13c013e8f526bed60f1bda406e7ba575ea1711 WatchSource:0}: Error finding container 921ef8723867ec83929d8aa33f13c013e8f526bed60f1bda406e7ba575ea1711: Status 404 returned error can't find the container with id 921ef8723867ec83929d8aa33f13c013e8f526bed60f1bda406e7ba575ea1711 Oct 31 04:32:31.629474 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:31.629450 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c9707f-d9cc59bdd-xz2sb"] Oct 31 04:32:31.629747 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:31.629731 4376 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 04:32:31.665042 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:31.664997 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-57dc6d-66998bfb9f-tdltw" podStartSLOduration=2.664984156 podStartE2EDuration="2.664984156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:32:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:32:31.664246471 +0000 UTC m=+1699.378581113" watchObservedRunningTime="2026-10-31 04:32:31.664984156 +0000 UTC m=+1699.379318772" Oct 31 04:32:32.168606 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:32.168576 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wmkf5" Oct 31 04:32:32.270301 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:32.270269 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d135a318-172d-4408-8e38-ca0e26946381-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d135a318-172d-4408-8e38-ca0e26946381-istiod-ca-cert\") pod \"d135a318-172d-4408-8e38-ca0e26946381\" (UID: \"d135a318-172d-4408-8e38-ca0e26946381\") " Oct 31 04:32:32.270498 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:32.270305 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d135a318-172d-4408-8e38-ca0e26946381-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d135a318-172d-4408-8e38-ca0e26946381-credential-socket\") pod \"d135a318-172d-4408-8e38-ca0e26946381\" (UID: \"d135a318-172d-4408-8e38-ca0e26946381\") " Oct 31 04:32:32.270498 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:32.270380 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d135a318-172d-4408-8e38-ca0e26946381-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d135a318-172d-4408-8e38-ca0e26946381-workload-socket\") pod \"d135a318-172d-4408-8e38-ca0e26946381\" (UID: \"d135a318-172d-4408-8e38-ca0e26946381\") " Oct 31 04:32:32.270498 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:32.270400 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d135a318-172d-4408-8e38-ca0e26946381-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d135a318-172d-4408-8e38-ca0e26946381-istio-podinfo\") pod \"d135a318-172d-4408-8e38-ca0e26946381\" (UID: \"d135a318-172d-4408-8e38-ca0e26946381\") " Oct 31 04:32:32.270498 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:32.270443 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d135a318-172d-4408-8e38-ca0e26946381-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d135a318-172d-4408-8e38-ca0e26946381-workload-certs\") pod \"d135a318-172d-4408-8e38-ca0e26946381\" (UID: \"d135a318-172d-4408-8e38-ca0e26946381\") " Oct 31 04:32:32.270713 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:32.270500 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d135a318-172d-4408-8e38-ca0e26946381-kube-api-access-hmdx4\" (UniqueName: \"kubernetes.io/projected/d135a318-172d-4408-8e38-ca0e26946381-kube-api-access-hmdx4\") pod \"d135a318-172d-4408-8e38-ca0e26946381\" (UID: \"d135a318-172d-4408-8e38-ca0e26946381\") " Oct 31 04:32:32.270713 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:32.270579 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d135a318-172d-4408-8e38-ca0e26946381-istio-token\" (UniqueName: \"kubernetes.io/projected/d135a318-172d-4408-8e38-ca0e26946381-istio-token\") pod \"d135a318-172d-4408-8e38-ca0e26946381\" (UID: \"d135a318-172d-4408-8e38-ca0e26946381\") " Oct 31 04:32:32.270900 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:32.270719 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d135a318-172d-4408-8e38-ca0e26946381-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d135a318-172d-4408-8e38-ca0e26946381-istio-envoy\") pod \"d135a318-172d-4408-8e38-ca0e26946381\" (UID: \"d135a318-172d-4408-8e38-ca0e26946381\") " Oct 31 04:32:32.270900 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:32.270736 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d135a318-172d-4408-8e38-ca0e26946381-credential-socket" pod "d135a318-172d-4408-8e38-ca0e26946381" (UID: "d135a318-172d-4408-8e38-ca0e26946381"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:32:32.270900 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:32.270755 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d135a318-172d-4408-8e38-ca0e26946381-workload-socket" pod "d135a318-172d-4408-8e38-ca0e26946381" (UID: "d135a318-172d-4408-8e38-ca0e26946381"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:32:32.270900 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:32.270764 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d135a318-172d-4408-8e38-ca0e26946381-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d135a318-172d-4408-8e38-ca0e26946381-istio-data\") pod \"d135a318-172d-4408-8e38-ca0e26946381\" (UID: \"d135a318-172d-4408-8e38-ca0e26946381\") " Oct 31 04:32:32.270900 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:32.270718 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d135a318-172d-4408-8e38-ca0e26946381-istiod-ca-cert" pod "d135a318-172d-4408-8e38-ca0e26946381" (UID: "d135a318-172d-4408-8e38-ca0e26946381"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:32:32.271152 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:32.271120 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d135a318-172d-4408-8e38-ca0e26946381-istio-data" pod "d135a318-172d-4408-8e38-ca0e26946381" (UID: "d135a318-172d-4408-8e38-ca0e26946381"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:32:32.271152 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:32.271115 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d135a318-172d-4408-8e38-ca0e26946381-workload-certs" pod "d135a318-172d-4408-8e38-ca0e26946381" (UID: "d135a318-172d-4408-8e38-ca0e26946381"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:32:32.271406 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:32.271383 4376 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d135a318-172d-4408-8e38-ca0e26946381-istio-data\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:32:32.271508 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:32.271409 4376 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d135a318-172d-4408-8e38-ca0e26946381-istiod-ca-cert\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:32:32.271508 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:32.271426 4376 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d135a318-172d-4408-8e38-ca0e26946381-credential-socket\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:32:32.271508 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:32.271440 4376 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d135a318-172d-4408-8e38-ca0e26946381-workload-socket\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:32:32.271508 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:32.271452 4376 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d135a318-172d-4408-8e38-ca0e26946381-workload-certs\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:32:32.272976 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:32.272953 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d135a318-172d-4408-8e38-ca0e26946381-istio-podinfo" pod "d135a318-172d-4408-8e38-ca0e26946381" (UID: "d135a318-172d-4408-8e38-ca0e26946381"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 04:32:32.273283 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:32.273268 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d135a318-172d-4408-8e38-ca0e26946381-istio-envoy" pod "d135a318-172d-4408-8e38-ca0e26946381" (UID: "d135a318-172d-4408-8e38-ca0e26946381"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:32:32.273477 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:32.273461 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d135a318-172d-4408-8e38-ca0e26946381-istio-token" pod "d135a318-172d-4408-8e38-ca0e26946381" (UID: "d135a318-172d-4408-8e38-ca0e26946381"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:32:32.273554 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:32.273528 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d135a318-172d-4408-8e38-ca0e26946381-kube-api-access-hmdx4" pod "d135a318-172d-4408-8e38-ca0e26946381" (UID: "d135a318-172d-4408-8e38-ca0e26946381"). InnerVolumeSpecName "kube-api-access-hmdx4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:32:32.388864 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:32.375302 4376 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d135a318-172d-4408-8e38-ca0e26946381-istio-envoy\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:32:32.388864 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:32.375391 4376 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d135a318-172d-4408-8e38-ca0e26946381-istio-podinfo\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:32:32.388864 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:32.375408 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmdx4\" (UniqueName: \"kubernetes.io/projected/d135a318-172d-4408-8e38-ca0e26946381-kube-api-access-hmdx4\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:32:32.388864 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:32.375437 4376 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d135a318-172d-4408-8e38-ca0e26946381-istio-token\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:32:32.624813 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:32.624770 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c9707f-d9cc59bdd-xz2sb" event={"ID":"65edcc7f-0600-4161-9ba1-89e26f064dd4","Type":"ContainerStarted","Data":"5d47d193f75c01dcae11a35926621b1a0b5ec6d6e740a33beb6610b14d5dff9f"} Oct 31 04:32:32.624990 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:32.624844 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c9707f-d9cc59bdd-xz2sb" event={"ID":"65edcc7f-0600-4161-9ba1-89e26f064dd4","Type":"ContainerStarted","Data":"921ef8723867ec83929d8aa33f13c013e8f526bed60f1bda406e7ba575ea1711"} Oct 31 04:32:32.625070 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:32.624996 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c9707f-d9cc59bdd-xz2sb" Oct 31 04:32:32.626436 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:32.626408 4376 generic.go:356] "Generic (PLEG): container finished" podID="d135a318-172d-4408-8e38-ca0e26946381" containerID="74946a5a9a9631dd0fafccc2fb98e2ade83a6269ccfcbd538e533d04741f061b" exitCode=0 Oct 31 04:32:32.626542 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:32.626448 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wmkf5" Oct 31 04:32:32.626542 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:32.626486 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wmkf5" event={"ID":"d135a318-172d-4408-8e38-ca0e26946381","Type":"ContainerDied","Data":"74946a5a9a9631dd0fafccc2fb98e2ade83a6269ccfcbd538e533d04741f061b"} Oct 31 04:32:32.626542 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:32.626537 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wmkf5" event={"ID":"d135a318-172d-4408-8e38-ca0e26946381","Type":"ContainerDied","Data":"55f4f034380f1d2019eb2d1a4c1d75db3aec6a69bbf3c7093c8498e0c71bf8c1"} Oct 31 04:32:32.626542 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:32.626560 4376 scope.go:122] "RemoveContainer" containerID="74946a5a9a9631dd0fafccc2fb98e2ade83a6269ccfcbd538e533d04741f061b" Oct 31 04:32:32.637249 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:32.637230 4376 scope.go:122] "RemoveContainer" containerID="74946a5a9a9631dd0fafccc2fb98e2ade83a6269ccfcbd538e533d04741f061b" Oct 31 04:32:32.637581 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:32:32.637563 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74946a5a9a9631dd0fafccc2fb98e2ade83a6269ccfcbd538e533d04741f061b\": container with ID starting with 74946a5a9a9631dd0fafccc2fb98e2ade83a6269ccfcbd538e533d04741f061b not found: ID does not exist" containerID="74946a5a9a9631dd0fafccc2fb98e2ade83a6269ccfcbd538e533d04741f061b" Oct 31 04:32:32.637636 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:32.637590 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74946a5a9a9631dd0fafccc2fb98e2ade83a6269ccfcbd538e533d04741f061b"} err="failed to get container status \"74946a5a9a9631dd0fafccc2fb98e2ade83a6269ccfcbd538e533d04741f061b\": rpc error: code = NotFound desc = could not find container \"74946a5a9a9631dd0fafccc2fb98e2ade83a6269ccfcbd538e533d04741f061b\": container with ID starting with 74946a5a9a9631dd0fafccc2fb98e2ade83a6269ccfcbd538e533d04741f061b not found: ID does not exist" Oct 31 04:32:32.669264 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:32.669213 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c9707f-d9cc59bdd-xz2sb" podStartSLOduration=2.098699568 podStartE2EDuration="2.669200356s" totalImagesPullingTime="570.500788ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:32:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:32:32.667007648 +0000 UTC m=+1700.381342292" watchObservedRunningTime="2026-10-31 04:32:32.669200356 +0000 UTC m=+1700.383535020" Oct 31 04:32:32.706087 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:32.706048 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wmkf5"] Oct 31 04:32:32.709916 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:32.709889 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wmkf5"] Oct 31 04:32:34.445150 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:34.445120 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d135a318-172d-4408-8e38-ca0e26946381" path="/var/lib/kubelet/pods/d135a318-172d-4408-8e38-ca0e26946381/volumes" Oct 31 04:32:37.371393 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:37.365365 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-81c810-openshift-default-5fbd5b64c7-kbvbk"] Oct 31 04:32:37.371393 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:37.365654 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-81c810-openshift-default-5fbd5b64c7-kbvbk" podUID="3c34e357-743e-442a-957d-788f1618aaf3" containerName="istio-proxy" containerID="cri-o://a7bb272b3c6d152f9c092163497389233315768c200b9bf8eb4f1e4699cdbd59" gracePeriod=30 Oct 31 04:32:37.503391 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:37.500384 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-338f2e-849686b74-9bcnc"] Oct 31 04:32:37.503391 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:37.501435 4376 cpu_manager.go:414] "containerMap: removing container" podUID="d135a318-172d-4408-8e38-ca0e26946381" containerName="istio-proxy" Oct 31 04:32:37.503391 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:37.501457 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d135a318-172d-4408-8e38-ca0e26946381" containerName="istio-proxy" Oct 31 04:32:37.503391 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:37.501610 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="d135a318-172d-4408-8e38-ca0e26946381" containerName="istio-proxy" Oct 31 04:32:37.526718 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:37.526684 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-338f2e-849686b74-9bcnc" Oct 31 04:32:37.573532 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:37.566273 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-338f2e-serving-cert\"" Oct 31 04:32:37.577380 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:37.574918 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-338f2e-849686b74-9bcnc"] Oct 31 04:32:37.701360 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:37.689593 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7cffa2e2-93ef-4f2b-a512-107f5d14b698-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-338f2e-849686b74-9bcnc\" (UID: \"7cffa2e2-93ef-4f2b-a512-107f5d14b698\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-338f2e-849686b74-9bcnc" Oct 31 04:32:37.701360 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:37.689829 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7cffa2e2-93ef-4f2b-a512-107f5d14b698-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-338f2e-849686b74-9bcnc\" (UID: \"7cffa2e2-93ef-4f2b-a512-107f5d14b698\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-338f2e-849686b74-9bcnc" Oct 31 04:32:37.701360 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:37.689925 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7cffa2e2-93ef-4f2b-a512-107f5d14b698-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-338f2e-849686b74-9bcnc\" (UID: \"7cffa2e2-93ef-4f2b-a512-107f5d14b698\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-338f2e-849686b74-9bcnc" Oct 31 04:32:37.701360 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:37.689979 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twk6m\" (UniqueName: \"kubernetes.io/projected/7cffa2e2-93ef-4f2b-a512-107f5d14b698-kube-api-access-twk6m\") pod \"maas-api-e2e-shared-a-w3-338f2e-849686b74-9bcnc\" (UID: \"7cffa2e2-93ef-4f2b-a512-107f5d14b698\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-338f2e-849686b74-9bcnc" Oct 31 04:32:37.795242 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:37.793506 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7cffa2e2-93ef-4f2b-a512-107f5d14b698-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-338f2e-849686b74-9bcnc\" (UID: \"7cffa2e2-93ef-4f2b-a512-107f5d14b698\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-338f2e-849686b74-9bcnc" Oct 31 04:32:37.796170 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:37.796135 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7cffa2e2-93ef-4f2b-a512-107f5d14b698-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-338f2e-849686b74-9bcnc\" (UID: \"7cffa2e2-93ef-4f2b-a512-107f5d14b698\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-338f2e-849686b74-9bcnc" Oct 31 04:32:37.796451 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:37.796247 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7cffa2e2-93ef-4f2b-a512-107f5d14b698-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-338f2e-849686b74-9bcnc\" (UID: \"7cffa2e2-93ef-4f2b-a512-107f5d14b698\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-338f2e-849686b74-9bcnc" Oct 31 04:32:37.796451 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:37.796307 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twk6m\" (UniqueName: \"kubernetes.io/projected/7cffa2e2-93ef-4f2b-a512-107f5d14b698-kube-api-access-twk6m\") pod \"maas-api-e2e-shared-a-w3-338f2e-849686b74-9bcnc\" (UID: \"7cffa2e2-93ef-4f2b-a512-107f5d14b698\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-338f2e-849686b74-9bcnc" Oct 31 04:32:37.805833 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:37.805809 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7cffa2e2-93ef-4f2b-a512-107f5d14b698-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-338f2e-849686b74-9bcnc\" (UID: \"7cffa2e2-93ef-4f2b-a512-107f5d14b698\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-338f2e-849686b74-9bcnc" Oct 31 04:32:37.807730 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:37.807708 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7cffa2e2-93ef-4f2b-a512-107f5d14b698-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-338f2e-849686b74-9bcnc\" (UID: \"7cffa2e2-93ef-4f2b-a512-107f5d14b698\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-338f2e-849686b74-9bcnc" Oct 31 04:32:37.808228 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:37.808208 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7cffa2e2-93ef-4f2b-a512-107f5d14b698-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-338f2e-849686b74-9bcnc\" (UID: \"7cffa2e2-93ef-4f2b-a512-107f5d14b698\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-338f2e-849686b74-9bcnc" Oct 31 04:32:37.823832 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:37.823801 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twk6m\" (UniqueName: \"kubernetes.io/projected/7cffa2e2-93ef-4f2b-a512-107f5d14b698-kube-api-access-twk6m\") pod \"maas-api-e2e-shared-a-w3-338f2e-849686b74-9bcnc\" (UID: \"7cffa2e2-93ef-4f2b-a512-107f5d14b698\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-338f2e-849686b74-9bcnc" Oct 31 04:32:38.029042 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:38.028959 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-338f2e-849686b74-9bcnc" Oct 31 04:32:38.328417 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:32:38.328377 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cffa2e2_93ef_4f2b_a512_107f5d14b698.slice/crio-b0b05d723d91511c08d3f6e3bd06a0dd6725fce484636ebeeae101dc6f0a972c WatchSource:0}: Error finding container b0b05d723d91511c08d3f6e3bd06a0dd6725fce484636ebeeae101dc6f0a972c: Status 404 returned error can't find the container with id b0b05d723d91511c08d3f6e3bd06a0dd6725fce484636ebeeae101dc6f0a972c Oct 31 04:32:38.329830 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:38.329804 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-338f2e-849686b74-9bcnc"] Oct 31 04:32:38.639808 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:38.639781 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c9707f-d9cc59bdd-xz2sb" Oct 31 04:32:38.741143 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:38.741115 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-338f2e-849686b74-9bcnc" event={"ID":"7cffa2e2-93ef-4f2b-a512-107f5d14b698","Type":"ContainerStarted","Data":"b0b05d723d91511c08d3f6e3bd06a0dd6725fce484636ebeeae101dc6f0a972c"} Oct 31 04:32:39.749509 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:39.749469 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-338f2e-849686b74-9bcnc" event={"ID":"7cffa2e2-93ef-4f2b-a512-107f5d14b698","Type":"ContainerStarted","Data":"df8432fa1463660d33bd46848e0ec0659352c10e8b8f09528915d8808c9f3138"} Oct 31 04:32:39.749908 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:39.749540 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-338f2e-849686b74-9bcnc" Oct 31 04:32:39.801353 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:39.796509 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-338f2e-849686b74-9bcnc" podStartSLOduration=2.452898169 podStartE2EDuration="2.796488618s" totalImagesPullingTime="343.590449ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:32:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:32:39.793822474 +0000 UTC m=+1707.508157114" watchObservedRunningTime="2026-10-31 04:32:39.796488618 +0000 UTC m=+1707.510823238" Oct 31 04:32:41.001821 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:32:41.001786 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 04:32:41.620773 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:41.620742 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-57dc6d-5675c77f6c-8t8zt" Oct 31 04:32:42.631595 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:42.630045 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-57dc6d-66998bfb9f-tdltw" Oct 31 04:32:42.661960 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:42.661938 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-81c810-openshift-default-5fbd5b64c7-kbvbk" Oct 31 04:32:42.772503 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:42.772462 4376 generic.go:356] "Generic (PLEG): container finished" podID="3c34e357-743e-442a-957d-788f1618aaf3" containerID="a7bb272b3c6d152f9c092163497389233315768c200b9bf8eb4f1e4699cdbd59" exitCode=0 Oct 31 04:32:42.772677 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:42.772511 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-81c810-openshift-default-5fbd5b64c7-kbvbk" Oct 31 04:32:42.772677 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:42.772540 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-81c810-openshift-default-5fbd5b64c7-kbvbk" event={"ID":"3c34e357-743e-442a-957d-788f1618aaf3","Type":"ContainerDied","Data":"a7bb272b3c6d152f9c092163497389233315768c200b9bf8eb4f1e4699cdbd59"} Oct 31 04:32:42.772677 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:42.772600 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-81c810-openshift-default-5fbd5b64c7-kbvbk" event={"ID":"3c34e357-743e-442a-957d-788f1618aaf3","Type":"ContainerDied","Data":"e7eb828a5b246059d3c1eff7c4d034e07c711d818749a272acc813193df06fe3"} Oct 31 04:32:42.772677 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:42.772632 4376 scope.go:122] "RemoveContainer" containerID="a7bb272b3c6d152f9c092163497389233315768c200b9bf8eb4f1e4699cdbd59" Oct 31 04:32:42.790185 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:42.790154 4376 scope.go:122] "RemoveContainer" containerID="a0c0db5b3fe8488903d4b2f1b246295390bce9fbf6fe570b7d7cd55ebe738ca1" Oct 31 04:32:42.800947 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:42.800927 4376 scope.go:122] "RemoveContainer" containerID="a7bb272b3c6d152f9c092163497389233315768c200b9bf8eb4f1e4699cdbd59" Oct 31 04:32:42.801249 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:32:42.801229 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7bb272b3c6d152f9c092163497389233315768c200b9bf8eb4f1e4699cdbd59\": container with ID starting with a7bb272b3c6d152f9c092163497389233315768c200b9bf8eb4f1e4699cdbd59 not found: ID does not exist" containerID="a7bb272b3c6d152f9c092163497389233315768c200b9bf8eb4f1e4699cdbd59" Oct 31 04:32:42.801318 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:42.801261 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7bb272b3c6d152f9c092163497389233315768c200b9bf8eb4f1e4699cdbd59"} err="failed to get container status \"a7bb272b3c6d152f9c092163497389233315768c200b9bf8eb4f1e4699cdbd59\": rpc error: code = NotFound desc = could not find container \"a7bb272b3c6d152f9c092163497389233315768c200b9bf8eb4f1e4699cdbd59\": container with ID starting with a7bb272b3c6d152f9c092163497389233315768c200b9bf8eb4f1e4699cdbd59 not found: ID does not exist" Oct 31 04:32:42.801318 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:42.801281 4376 scope.go:122] "RemoveContainer" containerID="a0c0db5b3fe8488903d4b2f1b246295390bce9fbf6fe570b7d7cd55ebe738ca1" Oct 31 04:32:42.801625 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:32:42.801610 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0c0db5b3fe8488903d4b2f1b246295390bce9fbf6fe570b7d7cd55ebe738ca1\": container with ID starting with a0c0db5b3fe8488903d4b2f1b246295390bce9fbf6fe570b7d7cd55ebe738ca1 not found: ID does not exist" containerID="a0c0db5b3fe8488903d4b2f1b246295390bce9fbf6fe570b7d7cd55ebe738ca1" Oct 31 04:32:42.801673 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:42.801628 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0c0db5b3fe8488903d4b2f1b246295390bce9fbf6fe570b7d7cd55ebe738ca1"} err="failed to get container status \"a0c0db5b3fe8488903d4b2f1b246295390bce9fbf6fe570b7d7cd55ebe738ca1\": rpc error: code = NotFound desc = could not find container \"a0c0db5b3fe8488903d4b2f1b246295390bce9fbf6fe570b7d7cd55ebe738ca1\": container with ID starting with a0c0db5b3fe8488903d4b2f1b246295390bce9fbf6fe570b7d7cd55ebe738ca1 not found: ID does not exist" Oct 31 04:32:42.825386 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:42.825363 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c34e357-743e-442a-957d-788f1618aaf3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c34e357-743e-442a-957d-788f1618aaf3-istio-envoy\") pod \"3c34e357-743e-442a-957d-788f1618aaf3\" (UID: \"3c34e357-743e-442a-957d-788f1618aaf3\") " Oct 31 04:32:42.825505 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:42.825449 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c34e357-743e-442a-957d-788f1618aaf3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c34e357-743e-442a-957d-788f1618aaf3-istio-data\") pod \"3c34e357-743e-442a-957d-788f1618aaf3\" (UID: \"3c34e357-743e-442a-957d-788f1618aaf3\") " Oct 31 04:32:42.825505 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:42.825489 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c34e357-743e-442a-957d-788f1618aaf3-kube-api-access-k7mbq\" (UniqueName: \"kubernetes.io/projected/3c34e357-743e-442a-957d-788f1618aaf3-kube-api-access-k7mbq\") pod \"3c34e357-743e-442a-957d-788f1618aaf3\" (UID: \"3c34e357-743e-442a-957d-788f1618aaf3\") " Oct 31 04:32:42.825626 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:42.825575 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c34e357-743e-442a-957d-788f1618aaf3-istio-token\" (UniqueName: \"kubernetes.io/projected/3c34e357-743e-442a-957d-788f1618aaf3-istio-token\") pod \"3c34e357-743e-442a-957d-788f1618aaf3\" (UID: \"3c34e357-743e-442a-957d-788f1618aaf3\") " Oct 31 04:32:42.825626 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:42.825609 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c34e357-743e-442a-957d-788f1618aaf3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c34e357-743e-442a-957d-788f1618aaf3-workload-certs\") pod \"3c34e357-743e-442a-957d-788f1618aaf3\" (UID: \"3c34e357-743e-442a-957d-788f1618aaf3\") " Oct 31 04:32:42.825739 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:42.825628 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c34e357-743e-442a-957d-788f1618aaf3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c34e357-743e-442a-957d-788f1618aaf3-istiod-ca-cert\") pod \"3c34e357-743e-442a-957d-788f1618aaf3\" (UID: \"3c34e357-743e-442a-957d-788f1618aaf3\") " Oct 31 04:32:42.825739 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:42.825693 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3c34e357-743e-442a-957d-788f1618aaf3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c34e357-743e-442a-957d-788f1618aaf3-istio-podinfo\") pod \"3c34e357-743e-442a-957d-788f1618aaf3\" (UID: \"3c34e357-743e-442a-957d-788f1618aaf3\") " Oct 31 04:32:42.825739 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:42.825720 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c34e357-743e-442a-957d-788f1618aaf3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c34e357-743e-442a-957d-788f1618aaf3-workload-socket\") pod \"3c34e357-743e-442a-957d-788f1618aaf3\" (UID: \"3c34e357-743e-442a-957d-788f1618aaf3\") " Oct 31 04:32:42.825739 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:42.825726 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c34e357-743e-442a-957d-788f1618aaf3-istio-data" pod "3c34e357-743e-442a-957d-788f1618aaf3" (UID: "3c34e357-743e-442a-957d-788f1618aaf3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:32:42.825943 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:42.825802 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c34e357-743e-442a-957d-788f1618aaf3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c34e357-743e-442a-957d-788f1618aaf3-credential-socket\") pod \"3c34e357-743e-442a-957d-788f1618aaf3\" (UID: \"3c34e357-743e-442a-957d-788f1618aaf3\") " Oct 31 04:32:42.825943 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:42.825911 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c34e357-743e-442a-957d-788f1618aaf3-workload-certs" pod "3c34e357-743e-442a-957d-788f1618aaf3" (UID: "3c34e357-743e-442a-957d-788f1618aaf3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:32:42.826218 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:42.826138 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c34e357-743e-442a-957d-788f1618aaf3-workload-socket" pod "3c34e357-743e-442a-957d-788f1618aaf3" (UID: "3c34e357-743e-442a-957d-788f1618aaf3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:32:42.826218 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:42.826167 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c34e357-743e-442a-957d-788f1618aaf3-credential-socket" pod "3c34e357-743e-442a-957d-788f1618aaf3" (UID: "3c34e357-743e-442a-957d-788f1618aaf3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:32:42.826436 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:42.826315 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c34e357-743e-442a-957d-788f1618aaf3-istiod-ca-cert" pod "3c34e357-743e-442a-957d-788f1618aaf3" (UID: "3c34e357-743e-442a-957d-788f1618aaf3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:32:42.826516 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:42.826490 4376 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c34e357-743e-442a-957d-788f1618aaf3-workload-socket\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:32:42.826624 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:42.826519 4376 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c34e357-743e-442a-957d-788f1618aaf3-credential-socket\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:32:42.826624 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:42.826534 4376 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c34e357-743e-442a-957d-788f1618aaf3-istio-data\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:32:42.826624 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:42.826543 4376 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c34e357-743e-442a-957d-788f1618aaf3-workload-certs\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:32:42.826624 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:42.826551 4376 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c34e357-743e-442a-957d-788f1618aaf3-istiod-ca-cert\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:32:42.827757 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:42.827728 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c34e357-743e-442a-957d-788f1618aaf3-kube-api-access-k7mbq" pod "3c34e357-743e-442a-957d-788f1618aaf3" (UID: "3c34e357-743e-442a-957d-788f1618aaf3"). InnerVolumeSpecName "kube-api-access-k7mbq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:32:42.827841 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:42.827787 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3c34e357-743e-442a-957d-788f1618aaf3-istio-podinfo" pod "3c34e357-743e-442a-957d-788f1618aaf3" (UID: "3c34e357-743e-442a-957d-788f1618aaf3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 04:32:42.827930 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:42.827914 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c34e357-743e-442a-957d-788f1618aaf3-istio-envoy" pod "3c34e357-743e-442a-957d-788f1618aaf3" (UID: "3c34e357-743e-442a-957d-788f1618aaf3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:32:42.827978 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:42.827936 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c34e357-743e-442a-957d-788f1618aaf3-istio-token" pod "3c34e357-743e-442a-957d-788f1618aaf3" (UID: "3c34e357-743e-442a-957d-788f1618aaf3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:32:42.928098 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:42.928023 4376 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c34e357-743e-442a-957d-788f1618aaf3-istio-podinfo\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:32:42.928098 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:42.928056 4376 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c34e357-743e-442a-957d-788f1618aaf3-istio-envoy\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:32:42.928098 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:42.928071 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7mbq\" (UniqueName: \"kubernetes.io/projected/3c34e357-743e-442a-957d-788f1618aaf3-kube-api-access-k7mbq\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:32:42.928098 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:42.928085 4376 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c34e357-743e-442a-957d-788f1618aaf3-istio-token\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:32:43.134246 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:43.134202 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-81c810-openshift-default-5fbd5b64c7-kbvbk"] Oct 31 04:32:43.141443 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:43.141418 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-81c810-openshift-default-5fbd5b64c7-kbvbk"] Oct 31 04:32:44.448182 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:44.448140 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c34e357-743e-442a-957d-788f1618aaf3" path="/var/lib/kubelet/pods/3c34e357-743e-442a-957d-788f1618aaf3/volumes" Oct 31 04:32:45.778008 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:45.777960 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-338f2e-849686b74-9bcnc" Oct 31 04:32:54.863723 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:54.863678 4376 generic.go:356] "Generic (PLEG): container finished" podID="932966ca-c565-4bab-a007-7f918c7f0c15" containerID="0f66c5f0d835c6a13b765cb395beaa14cefce2a0b4d49bf6e756db60d2b08bfb" exitCode=0 Oct 31 04:32:54.864290 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:54.863835 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-c9707f-openshift-default-65b9677b4c-b57kk" event={"ID":"932966ca-c565-4bab-a007-7f918c7f0c15","Type":"ContainerDied","Data":"0f66c5f0d835c6a13b765cb395beaa14cefce2a0b4d49bf6e756db60d2b08bfb"} Oct 31 04:32:54.864418 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:54.864397 4376 scope.go:122] "RemoveContainer" containerID="0f66c5f0d835c6a13b765cb395beaa14cefce2a0b4d49bf6e756db60d2b08bfb" Oct 31 04:32:54.867290 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:54.867241 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:32:54.867441 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:54.867415 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:32:54.867537 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:54.867515 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:32:55.882965 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:55.882923 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-c9707f-openshift-default-65b9677b4c-b57kk" event={"ID":"932966ca-c565-4bab-a007-7f918c7f0c15","Type":"ContainerStarted","Data":"d19db211d103111c049291a91f9841f5f2e0da1f67602e3285ce2101fed0d941"} Oct 31 04:32:56.337218 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:56.337118 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-c9707f-openshift-default-65b9677b4c-b57kk" Oct 31 04:32:56.345353 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:56.344456 4376 patch_prober.go:29] interesting pod/e2e-shared-a-w2-c9707f-openshift-default-65b9677b4c-b57kk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body= Oct 31 04:32:56.345353 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:56.344528 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-c9707f-openshift-default-65b9677b4c-b57kk" podUID="932966ca-c565-4bab-a007-7f918c7f0c15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" Oct 31 04:32:56.998054 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:56.998021 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-c9707f-openshift-default-65b9677b4c-b57kk" Oct 31 04:32:57.346548 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:57.346452 4376 patch_prober.go:29] interesting pod/e2e-shared-a-w2-c9707f-openshift-default-65b9677b4c-b57kk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body= Oct 31 04:32:57.346899 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:57.346865 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-c9707f-openshift-default-65b9677b4c-b57kk" podUID="932966ca-c565-4bab-a007-7f918c7f0c15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" Oct 31 04:32:58.340154 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:58.339699 4376 patch_prober.go:29] interesting pod/e2e-shared-a-w2-c9707f-openshift-default-65b9677b4c-b57kk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body= Oct 31 04:32:58.340154 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:58.339762 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-c9707f-openshift-default-65b9677b4c-b57kk" podUID="932966ca-c565-4bab-a007-7f918c7f0c15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" Oct 31 04:32:59.356352 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:59.344669 4376 patch_prober.go:29] interesting pod/e2e-shared-a-w2-c9707f-openshift-default-65b9677b4c-b57kk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body= Oct 31 04:32:59.356352 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:32:59.344771 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-c9707f-openshift-default-65b9677b4c-b57kk" podUID="932966ca-c565-4bab-a007-7f918c7f0c15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" Oct 31 04:33:00.338785 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:00.338694 4376 patch_prober.go:29] interesting pod/e2e-shared-a-w2-c9707f-openshift-default-65b9677b4c-b57kk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body= Oct 31 04:33:00.339601 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:00.339054 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-c9707f-openshift-default-65b9677b4c-b57kk" podUID="932966ca-c565-4bab-a007-7f918c7f0c15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" Oct 31 04:33:01.338214 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:01.338183 4376 patch_prober.go:29] interesting pod/e2e-shared-a-w2-c9707f-openshift-default-65b9677b4c-b57kk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body= Oct 31 04:33:01.338642 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:01.338235 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-c9707f-openshift-default-65b9677b4c-b57kk" podUID="932966ca-c565-4bab-a007-7f918c7f0c15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" Oct 31 04:33:02.341824 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:02.341797 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-c9707f-openshift-default-65b9677b4c-b57kk" Oct 31 04:33:02.342978 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:02.342958 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-c9707f-openshift-default-65b9677b4c-b57kk" Oct 31 04:33:06.224120 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:06.224028 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 04:33:08.680373 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:08.678236 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b2a372-84cc9765b7-8q729"] Oct 31 04:33:08.680373 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:08.679191 4376 cpu_manager.go:414] "containerMap: removing container" podUID="3c34e357-743e-442a-957d-788f1618aaf3" containerName="istio-proxy" Oct 31 04:33:08.680373 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:08.679214 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c34e357-743e-442a-957d-788f1618aaf3" containerName="istio-proxy" Oct 31 04:33:08.680373 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:08.679517 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c34e357-743e-442a-957d-788f1618aaf3" containerName="istio-proxy" Oct 31 04:33:08.680373 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:08.679539 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c34e357-743e-442a-957d-788f1618aaf3" containerName="istio-proxy" Oct 31 04:33:08.698346 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:08.686767 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b2a372-84cc9765b7-8q729"] Oct 31 04:33:08.708773 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:08.706313 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b2a372-84cc9765b7-8q729" Oct 31 04:33:08.727875 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:08.724061 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-b2a372\"" Oct 31 04:33:08.736407 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:08.736381 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-b2a372-dockercfg-p26ld\"" Oct 31 04:33:08.766476 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:08.766443 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8414bff7-0d22-4ccd-8c0d-e6c2dd294022-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-b2a372-84cc9765b7-8q729\" (UID: \"8414bff7-0d22-4ccd-8c0d-e6c2dd294022\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b2a372-84cc9765b7-8q729" Oct 31 04:33:08.766758 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:08.766727 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5pmw\" (UniqueName: \"kubernetes.io/projected/8414bff7-0d22-4ccd-8c0d-e6c2dd294022-kube-api-access-j5pmw\") pod \"payload-pre-processing-e2e-shared-b-w3-b2a372-84cc9765b7-8q729\" (UID: \"8414bff7-0d22-4ccd-8c0d-e6c2dd294022\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b2a372-84cc9765b7-8q729" Oct 31 04:33:08.868559 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:08.868521 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8414bff7-0d22-4ccd-8c0d-e6c2dd294022-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-b2a372-84cc9765b7-8q729\" (UID: \"8414bff7-0d22-4ccd-8c0d-e6c2dd294022\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b2a372-84cc9765b7-8q729" Oct 31 04:33:08.868868 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:08.868846 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5pmw\" (UniqueName: \"kubernetes.io/projected/8414bff7-0d22-4ccd-8c0d-e6c2dd294022-kube-api-access-j5pmw\") pod \"payload-pre-processing-e2e-shared-b-w3-b2a372-84cc9765b7-8q729\" (UID: \"8414bff7-0d22-4ccd-8c0d-e6c2dd294022\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b2a372-84cc9765b7-8q729" Oct 31 04:33:08.873710 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:08.873685 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8414bff7-0d22-4ccd-8c0d-e6c2dd294022-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-b2a372-84cc9765b7-8q729\" (UID: \"8414bff7-0d22-4ccd-8c0d-e6c2dd294022\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b2a372-84cc9765b7-8q729" Oct 31 04:33:08.907655 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:08.907590 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5pmw\" (UniqueName: \"kubernetes.io/projected/8414bff7-0d22-4ccd-8c0d-e6c2dd294022-kube-api-access-j5pmw\") pod \"payload-pre-processing-e2e-shared-b-w3-b2a372-84cc9765b7-8q729\" (UID: \"8414bff7-0d22-4ccd-8c0d-e6c2dd294022\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b2a372-84cc9765b7-8q729" Oct 31 04:33:09.032232 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:09.032144 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b2a372-84cc9765b7-8q729" Oct 31 04:33:09.435877 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:09.433626 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b2a372-84cc9765b7-8q729"] Oct 31 04:33:10.126830 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:10.126795 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b2a372-84cc9765b7-8q729" event={"ID":"8414bff7-0d22-4ccd-8c0d-e6c2dd294022","Type":"ContainerStarted","Data":"49a566f597506c107acb92205ef5700da38eadccc16245a23304d33651a6be56"} Oct 31 04:33:10.127408 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:10.126854 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b2a372-84cc9765b7-8q729" event={"ID":"8414bff7-0d22-4ccd-8c0d-e6c2dd294022","Type":"ContainerStarted","Data":"016939757b72b4cfae531417bc6b1b7d2fea8a18b83ab819a61b0d2b1c430485"} Oct 31 04:33:10.127408 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:10.127033 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b2a372-84cc9765b7-8q729" Oct 31 04:33:10.168700 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:10.168649 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b2a372-84cc9765b7-8q729" podStartSLOduration=2.168635315 podStartE2EDuration="2.168635315s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:33:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:33:10.168262008 +0000 UTC m=+1737.882596625" watchObservedRunningTime="2026-10-31 04:33:10.168635315 +0000 UTC m=+1737.882969931" Oct 31 04:33:11.008669 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:33:11.008635 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 04:33:15.281296 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:15.281261 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-9633f0-6b8776566f-bbqd7"] Oct 31 04:33:15.281904 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:15.281889 4376 cpu_manager.go:414] "containerMap: removing container" podUID="3c34e357-743e-442a-957d-788f1618aaf3" containerName="istio-proxy" Oct 31 04:33:15.281949 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:15.281905 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c34e357-743e-442a-957d-788f1618aaf3" containerName="istio-proxy" Oct 31 04:33:15.285687 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:15.285665 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-9633f0-6b8776566f-bbqd7" Oct 31 04:33:15.303655 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:15.303627 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-9633f0-6b8776566f-bbqd7"] Oct 31 04:33:15.351716 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:15.351679 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-9633f0\"" Oct 31 04:33:15.384813 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:15.384765 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86377cdd-c75b-496b-8fea-1d8bcd541be8-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-9633f0-6b8776566f-bbqd7\" (UID: \"86377cdd-c75b-496b-8fea-1d8bcd541be8\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-9633f0-6b8776566f-bbqd7" Oct 31 04:33:15.385007 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:15.384890 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mhhn\" (UniqueName: \"kubernetes.io/projected/86377cdd-c75b-496b-8fea-1d8bcd541be8-kube-api-access-9mhhn\") pod \"payload-processing-e2e-shared-b-w4-9633f0-6b8776566f-bbqd7\" (UID: \"86377cdd-c75b-496b-8fea-1d8bcd541be8\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-9633f0-6b8776566f-bbqd7" Oct 31 04:33:15.510384 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:15.504738 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86377cdd-c75b-496b-8fea-1d8bcd541be8-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-9633f0-6b8776566f-bbqd7\" (UID: \"86377cdd-c75b-496b-8fea-1d8bcd541be8\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-9633f0-6b8776566f-bbqd7" Oct 31 04:33:15.510384 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:15.504843 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mhhn\" (UniqueName: \"kubernetes.io/projected/86377cdd-c75b-496b-8fea-1d8bcd541be8-kube-api-access-9mhhn\") pod \"payload-processing-e2e-shared-b-w4-9633f0-6b8776566f-bbqd7\" (UID: \"86377cdd-c75b-496b-8fea-1d8bcd541be8\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-9633f0-6b8776566f-bbqd7" Oct 31 04:33:15.530352 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:15.525410 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86377cdd-c75b-496b-8fea-1d8bcd541be8-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-9633f0-6b8776566f-bbqd7\" (UID: \"86377cdd-c75b-496b-8fea-1d8bcd541be8\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-9633f0-6b8776566f-bbqd7" Oct 31 04:33:15.577188 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:15.577110 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mhhn\" (UniqueName: \"kubernetes.io/projected/86377cdd-c75b-496b-8fea-1d8bcd541be8-kube-api-access-9mhhn\") pod \"payload-processing-e2e-shared-b-w4-9633f0-6b8776566f-bbqd7\" (UID: \"86377cdd-c75b-496b-8fea-1d8bcd541be8\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-9633f0-6b8776566f-bbqd7" Oct 31 04:33:15.610946 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:15.600693 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-9633f0-6b8776566f-bbqd7" Oct 31 04:33:15.927934 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:33:15.927895 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86377cdd_c75b_496b_8fea_1d8bcd541be8.slice/crio-0bef063e1d9fe6367ee8f908929e78106f3549afcbd8960d6a1f58929b9c1370 WatchSource:0}: Error finding container 0bef063e1d9fe6367ee8f908929e78106f3549afcbd8960d6a1f58929b9c1370: Status 404 returned error can't find the container with id 0bef063e1d9fe6367ee8f908929e78106f3549afcbd8960d6a1f58929b9c1370 Oct 31 04:33:15.929397 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:15.929371 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-9633f0-6b8776566f-bbqd7"] Oct 31 04:33:15.988921 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:15.988895 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-7229f2-openshift-default-7b5c69df47-7tnqr"] Oct 31 04:33:16.013365 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:16.012039 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-7229f2-openshift-default-7b5c69df47-7tnqr" Oct 31 04:33:16.021775 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:16.021745 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-7229f2-openshift-default-7b5c69df47-7tnqr"] Oct 31 04:33:16.028493 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:16.028150 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-7229f2-openshift-default-dockercfg-htdwk\"" Oct 31 04:33:16.155411 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:16.133411 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e23ed56-ce2f-47b9-bd20-d22f8aa8a3ae-istio-podinfo\") pod \"e2e-autores-7229f2-openshift-default-7b5c69df47-7tnqr\" (UID: \"9e23ed56-ce2f-47b9-bd20-d22f8aa8a3ae\") " pod="openshift-ingress/e2e-autores-7229f2-openshift-default-7b5c69df47-7tnqr" Oct 31 04:33:16.155812 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:16.155777 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e23ed56-ce2f-47b9-bd20-d22f8aa8a3ae-workload-certs\") pod \"e2e-autores-7229f2-openshift-default-7b5c69df47-7tnqr\" (UID: \"9e23ed56-ce2f-47b9-bd20-d22f8aa8a3ae\") " pod="openshift-ingress/e2e-autores-7229f2-openshift-default-7b5c69df47-7tnqr" Oct 31 04:33:16.155972 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:16.155954 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e23ed56-ce2f-47b9-bd20-d22f8aa8a3ae-credential-socket\") pod \"e2e-autores-7229f2-openshift-default-7b5c69df47-7tnqr\" (UID: \"9e23ed56-ce2f-47b9-bd20-d22f8aa8a3ae\") " pod="openshift-ingress/e2e-autores-7229f2-openshift-default-7b5c69df47-7tnqr" Oct 31 04:33:16.156104 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:16.156088 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e23ed56-ce2f-47b9-bd20-d22f8aa8a3ae-istio-envoy\") pod \"e2e-autores-7229f2-openshift-default-7b5c69df47-7tnqr\" (UID: \"9e23ed56-ce2f-47b9-bd20-d22f8aa8a3ae\") " pod="openshift-ingress/e2e-autores-7229f2-openshift-default-7b5c69df47-7tnqr" Oct 31 04:33:16.156260 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:16.156243 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xztnh\" (UniqueName: \"kubernetes.io/projected/9e23ed56-ce2f-47b9-bd20-d22f8aa8a3ae-kube-api-access-xztnh\") pod \"e2e-autores-7229f2-openshift-default-7b5c69df47-7tnqr\" (UID: \"9e23ed56-ce2f-47b9-bd20-d22f8aa8a3ae\") " pod="openshift-ingress/e2e-autores-7229f2-openshift-default-7b5c69df47-7tnqr" Oct 31 04:33:16.156405 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:16.156389 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e23ed56-ce2f-47b9-bd20-d22f8aa8a3ae-istio-data\") pod \"e2e-autores-7229f2-openshift-default-7b5c69df47-7tnqr\" (UID: \"9e23ed56-ce2f-47b9-bd20-d22f8aa8a3ae\") " pod="openshift-ingress/e2e-autores-7229f2-openshift-default-7b5c69df47-7tnqr" Oct 31 04:33:16.156738 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:16.156718 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e23ed56-ce2f-47b9-bd20-d22f8aa8a3ae-istiod-ca-cert\") pod \"e2e-autores-7229f2-openshift-default-7b5c69df47-7tnqr\" (UID: \"9e23ed56-ce2f-47b9-bd20-d22f8aa8a3ae\") " pod="openshift-ingress/e2e-autores-7229f2-openshift-default-7b5c69df47-7tnqr" Oct 31 04:33:16.156938 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:16.156918 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e23ed56-ce2f-47b9-bd20-d22f8aa8a3ae-workload-socket\") pod \"e2e-autores-7229f2-openshift-default-7b5c69df47-7tnqr\" (UID: \"9e23ed56-ce2f-47b9-bd20-d22f8aa8a3ae\") " pod="openshift-ingress/e2e-autores-7229f2-openshift-default-7b5c69df47-7tnqr" Oct 31 04:33:16.178546 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:16.178457 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e23ed56-ce2f-47b9-bd20-d22f8aa8a3ae-istio-token\") pod \"e2e-autores-7229f2-openshift-default-7b5c69df47-7tnqr\" (UID: \"9e23ed56-ce2f-47b9-bd20-d22f8aa8a3ae\") " pod="openshift-ingress/e2e-autores-7229f2-openshift-default-7b5c69df47-7tnqr" Oct 31 04:33:16.245352 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:16.240388 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-9633f0-6b8776566f-bbqd7" event={"ID":"86377cdd-c75b-496b-8fea-1d8bcd541be8","Type":"ContainerStarted","Data":"165729339cf75e2852dfb4e74a25f79dc0456a64c9af3d7428d67cfea8f3f735"} Oct 31 04:33:16.245352 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:16.240460 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-9633f0-6b8776566f-bbqd7" event={"ID":"86377cdd-c75b-496b-8fea-1d8bcd541be8","Type":"ContainerStarted","Data":"0bef063e1d9fe6367ee8f908929e78106f3549afcbd8960d6a1f58929b9c1370"} Oct 31 04:33:16.245352 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:16.241612 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-9633f0-6b8776566f-bbqd7" Oct 31 04:33:16.283351 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:16.281303 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e23ed56-ce2f-47b9-bd20-d22f8aa8a3ae-istio-podinfo\") pod \"e2e-autores-7229f2-openshift-default-7b5c69df47-7tnqr\" (UID: \"9e23ed56-ce2f-47b9-bd20-d22f8aa8a3ae\") " pod="openshift-ingress/e2e-autores-7229f2-openshift-default-7b5c69df47-7tnqr" Oct 31 04:33:16.283351 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:16.281394 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e23ed56-ce2f-47b9-bd20-d22f8aa8a3ae-workload-certs\") pod \"e2e-autores-7229f2-openshift-default-7b5c69df47-7tnqr\" (UID: \"9e23ed56-ce2f-47b9-bd20-d22f8aa8a3ae\") " pod="openshift-ingress/e2e-autores-7229f2-openshift-default-7b5c69df47-7tnqr" Oct 31 04:33:16.283351 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:16.281423 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e23ed56-ce2f-47b9-bd20-d22f8aa8a3ae-credential-socket\") pod \"e2e-autores-7229f2-openshift-default-7b5c69df47-7tnqr\" (UID: \"9e23ed56-ce2f-47b9-bd20-d22f8aa8a3ae\") " pod="openshift-ingress/e2e-autores-7229f2-openshift-default-7b5c69df47-7tnqr" Oct 31 04:33:16.283351 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:16.281461 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e23ed56-ce2f-47b9-bd20-d22f8aa8a3ae-istio-envoy\") pod \"e2e-autores-7229f2-openshift-default-7b5c69df47-7tnqr\" (UID: \"9e23ed56-ce2f-47b9-bd20-d22f8aa8a3ae\") " pod="openshift-ingress/e2e-autores-7229f2-openshift-default-7b5c69df47-7tnqr" Oct 31 04:33:16.283351 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:16.281543 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xztnh\" (UniqueName: \"kubernetes.io/projected/9e23ed56-ce2f-47b9-bd20-d22f8aa8a3ae-kube-api-access-xztnh\") pod \"e2e-autores-7229f2-openshift-default-7b5c69df47-7tnqr\" (UID: \"9e23ed56-ce2f-47b9-bd20-d22f8aa8a3ae\") " pod="openshift-ingress/e2e-autores-7229f2-openshift-default-7b5c69df47-7tnqr" Oct 31 04:33:16.283351 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:16.281589 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e23ed56-ce2f-47b9-bd20-d22f8aa8a3ae-istio-data\") pod \"e2e-autores-7229f2-openshift-default-7b5c69df47-7tnqr\" (UID: \"9e23ed56-ce2f-47b9-bd20-d22f8aa8a3ae\") " pod="openshift-ingress/e2e-autores-7229f2-openshift-default-7b5c69df47-7tnqr" Oct 31 04:33:16.283351 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:16.281729 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e23ed56-ce2f-47b9-bd20-d22f8aa8a3ae-istiod-ca-cert\") pod \"e2e-autores-7229f2-openshift-default-7b5c69df47-7tnqr\" (UID: \"9e23ed56-ce2f-47b9-bd20-d22f8aa8a3ae\") " pod="openshift-ingress/e2e-autores-7229f2-openshift-default-7b5c69df47-7tnqr" Oct 31 04:33:16.283351 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:16.281791 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e23ed56-ce2f-47b9-bd20-d22f8aa8a3ae-workload-socket\") pod \"e2e-autores-7229f2-openshift-default-7b5c69df47-7tnqr\" (UID: \"9e23ed56-ce2f-47b9-bd20-d22f8aa8a3ae\") " pod="openshift-ingress/e2e-autores-7229f2-openshift-default-7b5c69df47-7tnqr" Oct 31 04:33:16.283351 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:16.281836 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e23ed56-ce2f-47b9-bd20-d22f8aa8a3ae-istio-token\") pod \"e2e-autores-7229f2-openshift-default-7b5c69df47-7tnqr\" (UID: \"9e23ed56-ce2f-47b9-bd20-d22f8aa8a3ae\") " pod="openshift-ingress/e2e-autores-7229f2-openshift-default-7b5c69df47-7tnqr" Oct 31 04:33:16.284244 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:16.284033 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e23ed56-ce2f-47b9-bd20-d22f8aa8a3ae-istio-data\") pod \"e2e-autores-7229f2-openshift-default-7b5c69df47-7tnqr\" (UID: \"9e23ed56-ce2f-47b9-bd20-d22f8aa8a3ae\") " pod="openshift-ingress/e2e-autores-7229f2-openshift-default-7b5c69df47-7tnqr" Oct 31 04:33:16.287384 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:16.286465 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e23ed56-ce2f-47b9-bd20-d22f8aa8a3ae-credential-socket\") pod \"e2e-autores-7229f2-openshift-default-7b5c69df47-7tnqr\" (UID: \"9e23ed56-ce2f-47b9-bd20-d22f8aa8a3ae\") " pod="openshift-ingress/e2e-autores-7229f2-openshift-default-7b5c69df47-7tnqr" Oct 31 04:33:16.287384 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:16.287071 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e23ed56-ce2f-47b9-bd20-d22f8aa8a3ae-istiod-ca-cert\") pod \"e2e-autores-7229f2-openshift-default-7b5c69df47-7tnqr\" (UID: \"9e23ed56-ce2f-47b9-bd20-d22f8aa8a3ae\") " pod="openshift-ingress/e2e-autores-7229f2-openshift-default-7b5c69df47-7tnqr" Oct 31 04:33:16.287384 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:16.287299 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e23ed56-ce2f-47b9-bd20-d22f8aa8a3ae-workload-socket\") pod \"e2e-autores-7229f2-openshift-default-7b5c69df47-7tnqr\" (UID: \"9e23ed56-ce2f-47b9-bd20-d22f8aa8a3ae\") " pod="openshift-ingress/e2e-autores-7229f2-openshift-default-7b5c69df47-7tnqr" Oct 31 04:33:16.287971 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:16.287950 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e23ed56-ce2f-47b9-bd20-d22f8aa8a3ae-workload-certs\") pod \"e2e-autores-7229f2-openshift-default-7b5c69df47-7tnqr\" (UID: \"9e23ed56-ce2f-47b9-bd20-d22f8aa8a3ae\") " pod="openshift-ingress/e2e-autores-7229f2-openshift-default-7b5c69df47-7tnqr" Oct 31 04:33:16.288670 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:16.288644 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e23ed56-ce2f-47b9-bd20-d22f8aa8a3ae-istio-podinfo\") pod \"e2e-autores-7229f2-openshift-default-7b5c69df47-7tnqr\" (UID: \"9e23ed56-ce2f-47b9-bd20-d22f8aa8a3ae\") " pod="openshift-ingress/e2e-autores-7229f2-openshift-default-7b5c69df47-7tnqr" Oct 31 04:33:16.289156 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:16.289097 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-9633f0-6b8776566f-bbqd7" podStartSLOduration=1.289081553 podStartE2EDuration="1.289081553s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:33:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:33:16.288692255 +0000 UTC m=+1744.003026899" watchObservedRunningTime="2026-10-31 04:33:16.289081553 +0000 UTC m=+1744.003416169" Oct 31 04:33:16.304884 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:16.297102 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e23ed56-ce2f-47b9-bd20-d22f8aa8a3ae-istio-envoy\") pod \"e2e-autores-7229f2-openshift-default-7b5c69df47-7tnqr\" (UID: \"9e23ed56-ce2f-47b9-bd20-d22f8aa8a3ae\") " pod="openshift-ingress/e2e-autores-7229f2-openshift-default-7b5c69df47-7tnqr" Oct 31 04:33:16.312674 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:16.312642 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xztnh\" (UniqueName: \"kubernetes.io/projected/9e23ed56-ce2f-47b9-bd20-d22f8aa8a3ae-kube-api-access-xztnh\") pod \"e2e-autores-7229f2-openshift-default-7b5c69df47-7tnqr\" (UID: \"9e23ed56-ce2f-47b9-bd20-d22f8aa8a3ae\") " pod="openshift-ingress/e2e-autores-7229f2-openshift-default-7b5c69df47-7tnqr" Oct 31 04:33:16.313016 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:16.312996 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e23ed56-ce2f-47b9-bd20-d22f8aa8a3ae-istio-token\") pod \"e2e-autores-7229f2-openshift-default-7b5c69df47-7tnqr\" (UID: \"9e23ed56-ce2f-47b9-bd20-d22f8aa8a3ae\") " pod="openshift-ingress/e2e-autores-7229f2-openshift-default-7b5c69df47-7tnqr" Oct 31 04:33:16.335056 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:16.335020 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-7229f2-openshift-default-7b5c69df47-7tnqr" Oct 31 04:33:16.660596 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:33:16.660566 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e23ed56_ce2f_47b9_bd20_d22f8aa8a3ae.slice/crio-88d17da629a8a0b15d5bdd45ee6fb8d65c363f1742837db4be149c460ba2b2ae WatchSource:0}: Error finding container 88d17da629a8a0b15d5bdd45ee6fb8d65c363f1742837db4be149c460ba2b2ae: Status 404 returned error can't find the container with id 88d17da629a8a0b15d5bdd45ee6fb8d65c363f1742837db4be149c460ba2b2ae Oct 31 04:33:16.663801 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:16.663777 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-7229f2-openshift-default-7b5c69df47-7tnqr"] Oct 31 04:33:16.663801 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:16.663783 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:33:16.663976 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:16.663857 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:33:16.663976 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:16.663901 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:33:17.264091 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:17.264053 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-7229f2-openshift-default-7b5c69df47-7tnqr" event={"ID":"9e23ed56-ce2f-47b9-bd20-d22f8aa8a3ae","Type":"ContainerStarted","Data":"49d599099893cf360611bee983fceaf09d3b863db4c87b8c129e8a2c6a2b3058"} Oct 31 04:33:17.264697 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:17.264671 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-7229f2-openshift-default-7b5c69df47-7tnqr" event={"ID":"9e23ed56-ce2f-47b9-bd20-d22f8aa8a3ae","Type":"ContainerStarted","Data":"88d17da629a8a0b15d5bdd45ee6fb8d65c363f1742837db4be149c460ba2b2ae"} Oct 31 04:33:17.312645 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:17.312593 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-7229f2-openshift-default-7b5c69df47-7tnqr" podStartSLOduration=2.312577624 podStartE2EDuration="2.312577624s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:33:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:33:17.309633652 +0000 UTC m=+1745.023968288" watchObservedRunningTime="2026-10-31 04:33:17.312577624 +0000 UTC m=+1745.026912241" Oct 31 04:33:17.336009 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:17.335984 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-7229f2-openshift-default-7b5c69df47-7tnqr" Oct 31 04:33:17.341784 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:17.341763 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-7229f2-openshift-default-7b5c69df47-7tnqr" Oct 31 04:33:18.268611 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:18.268580 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-7229f2-openshift-default-7b5c69df47-7tnqr" Oct 31 04:33:18.269427 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:18.269407 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-7229f2-openshift-default-7b5c69df47-7tnqr" Oct 31 04:33:21.137722 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:21.137693 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b2a372-84cc9765b7-8q729" Oct 31 04:33:22.183383 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:22.181381 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-7229f2-7b7cfc846f-4h24n"] Oct 31 04:33:22.209694 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:22.209663 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-7229f2-7b7cfc846f-4h24n"] Oct 31 04:33:22.210017 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:22.210000 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-7229f2-7b7cfc846f-4h24n" Oct 31 04:33:22.243670 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:22.243643 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-7229f2\"" Oct 31 04:33:22.245322 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:22.245261 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-7229f2-dockercfg-dj94c\"" Oct 31 04:33:22.305382 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:22.305345 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6cff2bce-3e01-4f33-a36d-eda47fbeae3d-plugins-config-volume\") pod \"payload-processing-e2e-autores-7229f2-7b7cfc846f-4h24n\" (UID: \"6cff2bce-3e01-4f33-a36d-eda47fbeae3d\") " pod="openshift-ingress/payload-processing-e2e-autores-7229f2-7b7cfc846f-4h24n" Oct 31 04:33:22.305719 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:22.305690 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thgnp\" (UniqueName: \"kubernetes.io/projected/6cff2bce-3e01-4f33-a36d-eda47fbeae3d-kube-api-access-thgnp\") pod \"payload-processing-e2e-autores-7229f2-7b7cfc846f-4h24n\" (UID: \"6cff2bce-3e01-4f33-a36d-eda47fbeae3d\") " pod="openshift-ingress/payload-processing-e2e-autores-7229f2-7b7cfc846f-4h24n" Oct 31 04:33:22.407044 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:22.407008 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thgnp\" (UniqueName: \"kubernetes.io/projected/6cff2bce-3e01-4f33-a36d-eda47fbeae3d-kube-api-access-thgnp\") pod \"payload-processing-e2e-autores-7229f2-7b7cfc846f-4h24n\" (UID: \"6cff2bce-3e01-4f33-a36d-eda47fbeae3d\") " pod="openshift-ingress/payload-processing-e2e-autores-7229f2-7b7cfc846f-4h24n" Oct 31 04:33:22.407225 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:22.407064 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6cff2bce-3e01-4f33-a36d-eda47fbeae3d-plugins-config-volume\") pod \"payload-processing-e2e-autores-7229f2-7b7cfc846f-4h24n\" (UID: \"6cff2bce-3e01-4f33-a36d-eda47fbeae3d\") " pod="openshift-ingress/payload-processing-e2e-autores-7229f2-7b7cfc846f-4h24n" Oct 31 04:33:22.407849 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:22.407825 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6cff2bce-3e01-4f33-a36d-eda47fbeae3d-plugins-config-volume\") pod \"payload-processing-e2e-autores-7229f2-7b7cfc846f-4h24n\" (UID: \"6cff2bce-3e01-4f33-a36d-eda47fbeae3d\") " pod="openshift-ingress/payload-processing-e2e-autores-7229f2-7b7cfc846f-4h24n" Oct 31 04:33:22.431118 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:22.431089 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-thgnp\" (UniqueName: \"kubernetes.io/projected/6cff2bce-3e01-4f33-a36d-eda47fbeae3d-kube-api-access-thgnp\") pod \"payload-processing-e2e-autores-7229f2-7b7cfc846f-4h24n\" (UID: \"6cff2bce-3e01-4f33-a36d-eda47fbeae3d\") " pod="openshift-ingress/payload-processing-e2e-autores-7229f2-7b7cfc846f-4h24n" Oct 31 04:33:22.548544 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:22.548464 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-7229f2-7b7cfc846f-4h24n" Oct 31 04:33:22.846569 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:33:22.846532 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cff2bce_3e01_4f33_a36d_eda47fbeae3d.slice/crio-01e39bdf2d84cfa698efa48ee744332a662b6d4da62822788b09b5ed662ec19b WatchSource:0}: Error finding container 01e39bdf2d84cfa698efa48ee744332a662b6d4da62822788b09b5ed662ec19b: Status 404 returned error can't find the container with id 01e39bdf2d84cfa698efa48ee744332a662b6d4da62822788b09b5ed662ec19b Oct 31 04:33:22.848502 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:22.848481 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-7229f2-7b7cfc846f-4h24n"] Oct 31 04:33:23.351687 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:23.351651 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-7229f2-7b7cfc846f-4h24n" event={"ID":"6cff2bce-3e01-4f33-a36d-eda47fbeae3d","Type":"ContainerStarted","Data":"9d9524a952d6f91d31d8e450b527dc50509105d26de44f19986807644d4adbdf"} Oct 31 04:33:23.352128 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:23.351736 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-7229f2-7b7cfc846f-4h24n" event={"ID":"6cff2bce-3e01-4f33-a36d-eda47fbeae3d","Type":"ContainerStarted","Data":"01e39bdf2d84cfa698efa48ee744332a662b6d4da62822788b09b5ed662ec19b"} Oct 31 04:33:23.352128 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:23.351756 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-7229f2-7b7cfc846f-4h24n" Oct 31 04:33:23.394643 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:23.394591 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-7229f2-7b7cfc846f-4h24n" podStartSLOduration=1.394578166 podStartE2EDuration="1.394578166s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:33:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:33:23.39246838 +0000 UTC m=+1751.106802997" watchObservedRunningTime="2026-10-31 04:33:23.394578166 +0000 UTC m=+1751.108912783" Oct 31 04:33:28.296146 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:28.296045 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-9633f0-6b8776566f-bbqd7" Oct 31 04:33:31.056849 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:31.056762 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-895sj"] Oct 31 04:33:31.062295 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:31.062272 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-895sj" Oct 31 04:33:31.085191 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:31.085159 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-895sj"] Oct 31 04:33:31.161091 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:31.161058 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nxp4\" (UniqueName: \"kubernetes.io/projected/bd748a1b-27f9-4e1e-9398-e3bff9c73ae7-kube-api-access-6nxp4\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-895sj\" (UID: \"bd748a1b-27f9-4e1e-9398-e3bff9c73ae7\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-895sj" Oct 31 04:33:31.161252 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:31.161105 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd748a1b-27f9-4e1e-9398-e3bff9c73ae7-credential-socket\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-895sj\" (UID: \"bd748a1b-27f9-4e1e-9398-e3bff9c73ae7\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-895sj" Oct 31 04:33:31.161252 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:31.161222 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd748a1b-27f9-4e1e-9398-e3bff9c73ae7-istiod-ca-cert\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-895sj\" (UID: \"bd748a1b-27f9-4e1e-9398-e3bff9c73ae7\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-895sj" Oct 31 04:33:31.161369 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:31.161253 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd748a1b-27f9-4e1e-9398-e3bff9c73ae7-workload-certs\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-895sj\" (UID: \"bd748a1b-27f9-4e1e-9398-e3bff9c73ae7\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-895sj" Oct 31 04:33:31.161369 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:31.161323 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd748a1b-27f9-4e1e-9398-e3bff9c73ae7-istio-envoy\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-895sj\" (UID: \"bd748a1b-27f9-4e1e-9398-e3bff9c73ae7\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-895sj" Oct 31 04:33:31.161513 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:31.161492 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd748a1b-27f9-4e1e-9398-e3bff9c73ae7-istio-data\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-895sj\" (UID: \"bd748a1b-27f9-4e1e-9398-e3bff9c73ae7\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-895sj" Oct 31 04:33:31.161569 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:31.161528 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd748a1b-27f9-4e1e-9398-e3bff9c73ae7-istio-podinfo\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-895sj\" (UID: \"bd748a1b-27f9-4e1e-9398-e3bff9c73ae7\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-895sj" Oct 31 04:33:31.161638 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:31.161622 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd748a1b-27f9-4e1e-9398-e3bff9c73ae7-workload-socket\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-895sj\" (UID: \"bd748a1b-27f9-4e1e-9398-e3bff9c73ae7\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-895sj" Oct 31 04:33:31.161693 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:31.161670 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd748a1b-27f9-4e1e-9398-e3bff9c73ae7-istio-token\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-895sj\" (UID: \"bd748a1b-27f9-4e1e-9398-e3bff9c73ae7\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-895sj" Oct 31 04:33:31.265413 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:31.265372 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nxp4\" (UniqueName: \"kubernetes.io/projected/bd748a1b-27f9-4e1e-9398-e3bff9c73ae7-kube-api-access-6nxp4\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-895sj\" (UID: \"bd748a1b-27f9-4e1e-9398-e3bff9c73ae7\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-895sj" Oct 31 04:33:31.281356 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:31.265660 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd748a1b-27f9-4e1e-9398-e3bff9c73ae7-credential-socket\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-895sj\" (UID: \"bd748a1b-27f9-4e1e-9398-e3bff9c73ae7\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-895sj" Oct 31 04:33:31.281356 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:31.265770 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd748a1b-27f9-4e1e-9398-e3bff9c73ae7-istiod-ca-cert\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-895sj\" (UID: \"bd748a1b-27f9-4e1e-9398-e3bff9c73ae7\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-895sj" Oct 31 04:33:31.281356 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:31.265810 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd748a1b-27f9-4e1e-9398-e3bff9c73ae7-workload-certs\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-895sj\" (UID: \"bd748a1b-27f9-4e1e-9398-e3bff9c73ae7\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-895sj" Oct 31 04:33:31.281356 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:31.265872 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd748a1b-27f9-4e1e-9398-e3bff9c73ae7-istio-envoy\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-895sj\" (UID: \"bd748a1b-27f9-4e1e-9398-e3bff9c73ae7\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-895sj" Oct 31 04:33:31.281356 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:31.265930 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd748a1b-27f9-4e1e-9398-e3bff9c73ae7-istio-data\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-895sj\" (UID: \"bd748a1b-27f9-4e1e-9398-e3bff9c73ae7\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-895sj" Oct 31 04:33:31.281356 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:31.265954 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd748a1b-27f9-4e1e-9398-e3bff9c73ae7-istio-podinfo\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-895sj\" (UID: \"bd748a1b-27f9-4e1e-9398-e3bff9c73ae7\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-895sj" Oct 31 04:33:31.281356 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:31.266014 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd748a1b-27f9-4e1e-9398-e3bff9c73ae7-workload-socket\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-895sj\" (UID: \"bd748a1b-27f9-4e1e-9398-e3bff9c73ae7\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-895sj" Oct 31 04:33:31.281356 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:31.266053 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd748a1b-27f9-4e1e-9398-e3bff9c73ae7-istio-token\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-895sj\" (UID: \"bd748a1b-27f9-4e1e-9398-e3bff9c73ae7\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-895sj" Oct 31 04:33:31.281356 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:31.267181 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd748a1b-27f9-4e1e-9398-e3bff9c73ae7-credential-socket\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-895sj\" (UID: \"bd748a1b-27f9-4e1e-9398-e3bff9c73ae7\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-895sj" Oct 31 04:33:31.281356 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:31.267828 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd748a1b-27f9-4e1e-9398-e3bff9c73ae7-istiod-ca-cert\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-895sj\" (UID: \"bd748a1b-27f9-4e1e-9398-e3bff9c73ae7\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-895sj" Oct 31 04:33:31.281356 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:31.268068 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd748a1b-27f9-4e1e-9398-e3bff9c73ae7-workload-certs\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-895sj\" (UID: \"bd748a1b-27f9-4e1e-9398-e3bff9c73ae7\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-895sj" Oct 31 04:33:31.282441 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:31.282414 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd748a1b-27f9-4e1e-9398-e3bff9c73ae7-workload-socket\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-895sj\" (UID: \"bd748a1b-27f9-4e1e-9398-e3bff9c73ae7\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-895sj" Oct 31 04:33:31.283692 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:31.283223 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd748a1b-27f9-4e1e-9398-e3bff9c73ae7-istio-data\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-895sj\" (UID: \"bd748a1b-27f9-4e1e-9398-e3bff9c73ae7\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-895sj" Oct 31 04:33:31.289653 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:31.289609 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd748a1b-27f9-4e1e-9398-e3bff9c73ae7-istio-envoy\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-895sj\" (UID: \"bd748a1b-27f9-4e1e-9398-e3bff9c73ae7\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-895sj" Oct 31 04:33:31.316204 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:31.316091 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd748a1b-27f9-4e1e-9398-e3bff9c73ae7-istio-token\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-895sj\" (UID: \"bd748a1b-27f9-4e1e-9398-e3bff9c73ae7\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-895sj" Oct 31 04:33:31.323232 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:31.323174 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nxp4\" (UniqueName: \"kubernetes.io/projected/bd748a1b-27f9-4e1e-9398-e3bff9c73ae7-kube-api-access-6nxp4\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-895sj\" (UID: \"bd748a1b-27f9-4e1e-9398-e3bff9c73ae7\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-895sj" Oct 31 04:33:31.328184 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:31.328154 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd748a1b-27f9-4e1e-9398-e3bff9c73ae7-istio-podinfo\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-895sj\" (UID: \"bd748a1b-27f9-4e1e-9398-e3bff9c73ae7\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-895sj" Oct 31 04:33:31.404856 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:31.404816 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-895sj" Oct 31 04:33:31.773990 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:33:31.773769 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd748a1b_27f9_4e1e_9398_e3bff9c73ae7.slice/crio-f95863a5c065618edeaae98286ed0a0438cea0909b8853b5d708935b7642b682 WatchSource:0}: Error finding container f95863a5c065618edeaae98286ed0a0438cea0909b8853b5d708935b7642b682: Status 404 returned error can't find the container with id f95863a5c065618edeaae98286ed0a0438cea0909b8853b5d708935b7642b682 Oct 31 04:33:31.775268 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:31.775242 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-895sj"] Oct 31 04:33:31.776540 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:31.776509 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:33:31.776625 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:31.776570 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:33:31.776625 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:31.776600 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:33:32.470031 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:32.469992 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-895sj" event={"ID":"bd748a1b-27f9-4e1e-9398-e3bff9c73ae7","Type":"ContainerStarted","Data":"5ccc783fb255eae038c19b151fd480cb4d0a34175fff95d454b54ee0030712e6"} Oct 31 04:33:32.470435 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:32.470063 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-895sj" event={"ID":"bd748a1b-27f9-4e1e-9398-e3bff9c73ae7","Type":"ContainerStarted","Data":"f95863a5c065618edeaae98286ed0a0438cea0909b8853b5d708935b7642b682"} Oct 31 04:33:32.528687 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:32.528632 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-895sj" podStartSLOduration=1.528618987 podStartE2EDuration="1.528618987s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:33:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:33:32.527869678 +0000 UTC m=+1760.242204293" watchObservedRunningTime="2026-10-31 04:33:32.528618987 +0000 UTC m=+1760.242953603" Oct 31 04:33:33.407293 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:33.407248 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-895sj" Oct 31 04:33:33.409514 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:33.409485 4376 patch_prober.go:29] interesting pod/e2e-worker-w0-965f61-openshift-default-55854964c-895sj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body= Oct 31 04:33:33.409674 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:33.409534 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-895sj" podUID="bd748a1b-27f9-4e1e-9398-e3bff9c73ae7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" Oct 31 04:33:34.375865 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:34.375836 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-7229f2-7b7cfc846f-4h24n" Oct 31 04:33:34.407153 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:34.407122 4376 patch_prober.go:29] interesting pod/e2e-worker-w0-965f61-openshift-default-55854964c-895sj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body= Oct 31 04:33:34.407354 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:34.407179 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-895sj" podUID="bd748a1b-27f9-4e1e-9398-e3bff9c73ae7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" Oct 31 04:33:35.407458 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:35.407416 4376 patch_prober.go:29] interesting pod/e2e-worker-w0-965f61-openshift-default-55854964c-895sj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body= Oct 31 04:33:35.407854 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:35.407491 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-895sj" podUID="bd748a1b-27f9-4e1e-9398-e3bff9c73ae7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" Oct 31 04:33:36.407037 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:36.407006 4376 patch_prober.go:29] interesting pod/e2e-worker-w0-965f61-openshift-default-55854964c-895sj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body= Oct 31 04:33:36.407222 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:36.407068 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-895sj" podUID="bd748a1b-27f9-4e1e-9398-e3bff9c73ae7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" Oct 31 04:33:37.407600 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:37.407553 4376 patch_prober.go:29] interesting pod/e2e-worker-w0-965f61-openshift-default-55854964c-895sj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body= Oct 31 04:33:37.408007 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:37.407620 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-895sj" podUID="bd748a1b-27f9-4e1e-9398-e3bff9c73ae7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" Oct 31 04:33:37.685545 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:37.685453 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-9633f0-7c5f7dbc86-prfm5"] Oct 31 04:33:37.689795 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:37.689769 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-9633f0-7c5f7dbc86-prfm5" Oct 31 04:33:37.708116 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:37.708084 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-9633f0-7c5f7dbc86-prfm5"] Oct 31 04:33:37.715627 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:37.715564 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/775b25ed-d591-4102-98b8-59ce485f9f52-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-9633f0-7c5f7dbc86-prfm5\" (UID: \"775b25ed-d591-4102-98b8-59ce485f9f52\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-9633f0-7c5f7dbc86-prfm5" Oct 31 04:33:37.715814 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:37.715655 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/775b25ed-d591-4102-98b8-59ce485f9f52-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-9633f0-7c5f7dbc86-prfm5\" (UID: \"775b25ed-d591-4102-98b8-59ce485f9f52\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-9633f0-7c5f7dbc86-prfm5" Oct 31 04:33:37.715814 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:37.715698 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/775b25ed-d591-4102-98b8-59ce485f9f52-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-9633f0-7c5f7dbc86-prfm5\" (UID: \"775b25ed-d591-4102-98b8-59ce485f9f52\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-9633f0-7c5f7dbc86-prfm5" Oct 31 04:33:37.715814 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:37.715742 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xktf8\" (UniqueName: \"kubernetes.io/projected/775b25ed-d591-4102-98b8-59ce485f9f52-kube-api-access-xktf8\") pod \"maas-api-e2e-shared-b-w4-9633f0-7c5f7dbc86-prfm5\" (UID: \"775b25ed-d591-4102-98b8-59ce485f9f52\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-9633f0-7c5f7dbc86-prfm5" Oct 31 04:33:37.717875 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:37.717842 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w4-9633f0-serving-cert\"" Oct 31 04:33:37.817397 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:37.817355 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/775b25ed-d591-4102-98b8-59ce485f9f52-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-9633f0-7c5f7dbc86-prfm5\" (UID: \"775b25ed-d591-4102-98b8-59ce485f9f52\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-9633f0-7c5f7dbc86-prfm5" Oct 31 04:33:37.817590 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:37.817439 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/775b25ed-d591-4102-98b8-59ce485f9f52-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-9633f0-7c5f7dbc86-prfm5\" (UID: \"775b25ed-d591-4102-98b8-59ce485f9f52\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-9633f0-7c5f7dbc86-prfm5" Oct 31 04:33:37.817590 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:37.817470 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/775b25ed-d591-4102-98b8-59ce485f9f52-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-9633f0-7c5f7dbc86-prfm5\" (UID: \"775b25ed-d591-4102-98b8-59ce485f9f52\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-9633f0-7c5f7dbc86-prfm5" Oct 31 04:33:37.817590 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:37.817495 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xktf8\" (UniqueName: \"kubernetes.io/projected/775b25ed-d591-4102-98b8-59ce485f9f52-kube-api-access-xktf8\") pod \"maas-api-e2e-shared-b-w4-9633f0-7c5f7dbc86-prfm5\" (UID: \"775b25ed-d591-4102-98b8-59ce485f9f52\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-9633f0-7c5f7dbc86-prfm5" Oct 31 04:33:37.817590 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:33:37.817526 4376 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-9633f0-serving-cert: secret "maas-api-e2e-shared-b-w4-9633f0-serving-cert" not found Oct 31 04:33:37.817756 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:33:37.817602 4376 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/775b25ed-d591-4102-98b8-59ce485f9f52-maas-api-tls podName:775b25ed-d591-4102-98b8-59ce485f9f52 nodeName:}" failed. No retries permitted until 2026-10-31 04:33:38.317580136 +0000 UTC m=+1766.031914747 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/775b25ed-d591-4102-98b8-59ce485f9f52-maas-api-tls") pod "maas-api-e2e-shared-b-w4-9633f0-7c5f7dbc86-prfm5" (UID: "775b25ed-d591-4102-98b8-59ce485f9f52") : secret "maas-api-e2e-shared-b-w4-9633f0-serving-cert" not found Oct 31 04:33:37.818183 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:37.818159 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/775b25ed-d591-4102-98b8-59ce485f9f52-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-9633f0-7c5f7dbc86-prfm5\" (UID: \"775b25ed-d591-4102-98b8-59ce485f9f52\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-9633f0-7c5f7dbc86-prfm5" Oct 31 04:33:37.820087 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:37.820067 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/775b25ed-d591-4102-98b8-59ce485f9f52-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-9633f0-7c5f7dbc86-prfm5\" (UID: \"775b25ed-d591-4102-98b8-59ce485f9f52\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-9633f0-7c5f7dbc86-prfm5" Oct 31 04:33:37.841401 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:37.841375 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xktf8\" (UniqueName: \"kubernetes.io/projected/775b25ed-d591-4102-98b8-59ce485f9f52-kube-api-access-xktf8\") pod \"maas-api-e2e-shared-b-w4-9633f0-7c5f7dbc86-prfm5\" (UID: \"775b25ed-d591-4102-98b8-59ce485f9f52\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-9633f0-7c5f7dbc86-prfm5" Oct 31 04:33:38.328778 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:38.328735 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/775b25ed-d591-4102-98b8-59ce485f9f52-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-9633f0-7c5f7dbc86-prfm5\" (UID: \"775b25ed-d591-4102-98b8-59ce485f9f52\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-9633f0-7c5f7dbc86-prfm5" Oct 31 04:33:38.331282 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:38.331261 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/775b25ed-d591-4102-98b8-59ce485f9f52-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-9633f0-7c5f7dbc86-prfm5\" (UID: \"775b25ed-d591-4102-98b8-59ce485f9f52\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-9633f0-7c5f7dbc86-prfm5" Oct 31 04:33:38.406919 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:38.406879 4376 patch_prober.go:29] interesting pod/e2e-worker-w0-965f61-openshift-default-55854964c-895sj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body= Oct 31 04:33:38.407122 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:38.406948 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-895sj" podUID="bd748a1b-27f9-4e1e-9398-e3bff9c73ae7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" Oct 31 04:33:38.604671 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:38.604574 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-9633f0-7c5f7dbc86-prfm5" Oct 31 04:33:38.921854 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:38.921824 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-9633f0-7c5f7dbc86-prfm5"] Oct 31 04:33:39.412064 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:39.412034 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-895sj" Oct 31 04:33:39.412534 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:39.412518 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-895sj" Oct 31 04:33:39.413254 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:39.413236 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-895sj" Oct 31 04:33:39.552848 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:39.552757 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-9633f0-7c5f7dbc86-prfm5" event={"ID":"775b25ed-d591-4102-98b8-59ce485f9f52","Type":"ContainerStarted","Data":"409bb134f43af78336e192b2b1f9238e6d60ef9df19f2b69e4faea4c96991a2b"} Oct 31 04:33:39.552848 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:39.552824 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-9633f0-7c5f7dbc86-prfm5" event={"ID":"775b25ed-d591-4102-98b8-59ce485f9f52","Type":"ContainerStarted","Data":"f75ddcf68fc25f87703c873bacb0bd4792f72a621af63d259c86dea5e8f0af25"} Oct 31 04:33:39.553105 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:39.552870 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-9633f0-7c5f7dbc86-prfm5" Oct 31 04:33:39.597045 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:39.596983 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-9633f0-7c5f7dbc86-prfm5" podStartSLOduration=2.272490779 podStartE2EDuration="2.596968674s" totalImagesPullingTime="324.477895ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:33:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:33:39.594701185 +0000 UTC m=+1767.309035823" watchObservedRunningTime="2026-10-31 04:33:39.596968674 +0000 UTC m=+1767.311303291" Oct 31 04:33:40.241048 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:40.241008 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-lvnb4"] Oct 31 04:33:40.246041 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:40.246015 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-lvnb4" Oct 31 04:33:40.267980 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:40.267945 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-lvnb4"] Oct 31 04:33:40.276252 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:40.276211 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-031745-openshift-default-dockercfg-q5sr5\"" Oct 31 04:33:40.393669 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:40.393639 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6346bb15-5ba9-4c29-b74b-3e937ba64963-workload-socket\") pod \"e2e-shared-b-w2-031745-openshift-default-7f667bc645-lvnb4\" (UID: \"6346bb15-5ba9-4c29-b74b-3e937ba64963\") " pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-lvnb4" Oct 31 04:33:40.393872 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:40.393686 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6346bb15-5ba9-4c29-b74b-3e937ba64963-istio-podinfo\") pod \"e2e-shared-b-w2-031745-openshift-default-7f667bc645-lvnb4\" (UID: \"6346bb15-5ba9-4c29-b74b-3e937ba64963\") " pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-lvnb4" Oct 31 04:33:40.393872 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:40.393823 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6346bb15-5ba9-4c29-b74b-3e937ba64963-istio-token\") pod \"e2e-shared-b-w2-031745-openshift-default-7f667bc645-lvnb4\" (UID: \"6346bb15-5ba9-4c29-b74b-3e937ba64963\") " pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-lvnb4" Oct 31 04:33:40.394028 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:40.394009 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6346bb15-5ba9-4c29-b74b-3e937ba64963-istio-envoy\") pod \"e2e-shared-b-w2-031745-openshift-default-7f667bc645-lvnb4\" (UID: \"6346bb15-5ba9-4c29-b74b-3e937ba64963\") " pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-lvnb4" Oct 31 04:33:40.394091 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:40.394072 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sq2qs\" (UniqueName: \"kubernetes.io/projected/6346bb15-5ba9-4c29-b74b-3e937ba64963-kube-api-access-sq2qs\") pod \"e2e-shared-b-w2-031745-openshift-default-7f667bc645-lvnb4\" (UID: \"6346bb15-5ba9-4c29-b74b-3e937ba64963\") " pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-lvnb4" Oct 31 04:33:40.394206 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:40.394117 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6346bb15-5ba9-4c29-b74b-3e937ba64963-workload-certs\") pod \"e2e-shared-b-w2-031745-openshift-default-7f667bc645-lvnb4\" (UID: \"6346bb15-5ba9-4c29-b74b-3e937ba64963\") " pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-lvnb4" Oct 31 04:33:40.394269 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:40.394233 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6346bb15-5ba9-4c29-b74b-3e937ba64963-istio-data\") pod \"e2e-shared-b-w2-031745-openshift-default-7f667bc645-lvnb4\" (UID: \"6346bb15-5ba9-4c29-b74b-3e937ba64963\") " pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-lvnb4" Oct 31 04:33:40.394323 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:40.394280 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6346bb15-5ba9-4c29-b74b-3e937ba64963-istiod-ca-cert\") pod \"e2e-shared-b-w2-031745-openshift-default-7f667bc645-lvnb4\" (UID: \"6346bb15-5ba9-4c29-b74b-3e937ba64963\") " pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-lvnb4" Oct 31 04:33:40.394413 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:40.394397 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6346bb15-5ba9-4c29-b74b-3e937ba64963-credential-socket\") pod \"e2e-shared-b-w2-031745-openshift-default-7f667bc645-lvnb4\" (UID: \"6346bb15-5ba9-4c29-b74b-3e937ba64963\") " pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-lvnb4" Oct 31 04:33:40.495912 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:40.495820 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6346bb15-5ba9-4c29-b74b-3e937ba64963-istio-envoy\") pod \"e2e-shared-b-w2-031745-openshift-default-7f667bc645-lvnb4\" (UID: \"6346bb15-5ba9-4c29-b74b-3e937ba64963\") " pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-lvnb4" Oct 31 04:33:40.495912 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:40.495878 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sq2qs\" (UniqueName: \"kubernetes.io/projected/6346bb15-5ba9-4c29-b74b-3e937ba64963-kube-api-access-sq2qs\") pod \"e2e-shared-b-w2-031745-openshift-default-7f667bc645-lvnb4\" (UID: \"6346bb15-5ba9-4c29-b74b-3e937ba64963\") " pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-lvnb4" Oct 31 04:33:40.495912 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:40.495908 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6346bb15-5ba9-4c29-b74b-3e937ba64963-workload-certs\") pod \"e2e-shared-b-w2-031745-openshift-default-7f667bc645-lvnb4\" (UID: \"6346bb15-5ba9-4c29-b74b-3e937ba64963\") " pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-lvnb4" Oct 31 04:33:40.496205 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:40.495960 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6346bb15-5ba9-4c29-b74b-3e937ba64963-istio-data\") pod \"e2e-shared-b-w2-031745-openshift-default-7f667bc645-lvnb4\" (UID: \"6346bb15-5ba9-4c29-b74b-3e937ba64963\") " pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-lvnb4" Oct 31 04:33:40.496205 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:40.495992 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6346bb15-5ba9-4c29-b74b-3e937ba64963-istiod-ca-cert\") pod \"e2e-shared-b-w2-031745-openshift-default-7f667bc645-lvnb4\" (UID: \"6346bb15-5ba9-4c29-b74b-3e937ba64963\") " pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-lvnb4" Oct 31 04:33:40.496205 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:40.496031 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6346bb15-5ba9-4c29-b74b-3e937ba64963-credential-socket\") pod \"e2e-shared-b-w2-031745-openshift-default-7f667bc645-lvnb4\" (UID: \"6346bb15-5ba9-4c29-b74b-3e937ba64963\") " pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-lvnb4" Oct 31 04:33:40.496205 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:40.496120 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6346bb15-5ba9-4c29-b74b-3e937ba64963-workload-socket\") pod \"e2e-shared-b-w2-031745-openshift-default-7f667bc645-lvnb4\" (UID: \"6346bb15-5ba9-4c29-b74b-3e937ba64963\") " pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-lvnb4" Oct 31 04:33:40.496205 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:40.496150 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6346bb15-5ba9-4c29-b74b-3e937ba64963-istio-podinfo\") pod \"e2e-shared-b-w2-031745-openshift-default-7f667bc645-lvnb4\" (UID: \"6346bb15-5ba9-4c29-b74b-3e937ba64963\") " pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-lvnb4" Oct 31 04:33:40.496205 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:40.496184 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6346bb15-5ba9-4c29-b74b-3e937ba64963-istio-token\") pod \"e2e-shared-b-w2-031745-openshift-default-7f667bc645-lvnb4\" (UID: \"6346bb15-5ba9-4c29-b74b-3e937ba64963\") " pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-lvnb4" Oct 31 04:33:40.496866 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:40.496838 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6346bb15-5ba9-4c29-b74b-3e937ba64963-istio-data\") pod \"e2e-shared-b-w2-031745-openshift-default-7f667bc645-lvnb4\" (UID: \"6346bb15-5ba9-4c29-b74b-3e937ba64963\") " pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-lvnb4" Oct 31 04:33:40.497115 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:40.497088 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6346bb15-5ba9-4c29-b74b-3e937ba64963-credential-socket\") pod \"e2e-shared-b-w2-031745-openshift-default-7f667bc645-lvnb4\" (UID: \"6346bb15-5ba9-4c29-b74b-3e937ba64963\") " pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-lvnb4" Oct 31 04:33:40.497566 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:40.497300 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6346bb15-5ba9-4c29-b74b-3e937ba64963-workload-socket\") pod \"e2e-shared-b-w2-031745-openshift-default-7f667bc645-lvnb4\" (UID: \"6346bb15-5ba9-4c29-b74b-3e937ba64963\") " pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-lvnb4" Oct 31 04:33:40.497566 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:40.497403 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6346bb15-5ba9-4c29-b74b-3e937ba64963-workload-certs\") pod \"e2e-shared-b-w2-031745-openshift-default-7f667bc645-lvnb4\" (UID: \"6346bb15-5ba9-4c29-b74b-3e937ba64963\") " pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-lvnb4" Oct 31 04:33:40.497566 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:40.497488 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6346bb15-5ba9-4c29-b74b-3e937ba64963-istiod-ca-cert\") pod \"e2e-shared-b-w2-031745-openshift-default-7f667bc645-lvnb4\" (UID: \"6346bb15-5ba9-4c29-b74b-3e937ba64963\") " pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-lvnb4" Oct 31 04:33:40.498972 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:40.498947 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6346bb15-5ba9-4c29-b74b-3e937ba64963-istio-envoy\") pod \"e2e-shared-b-w2-031745-openshift-default-7f667bc645-lvnb4\" (UID: \"6346bb15-5ba9-4c29-b74b-3e937ba64963\") " pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-lvnb4" Oct 31 04:33:40.499544 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:40.499527 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6346bb15-5ba9-4c29-b74b-3e937ba64963-istio-podinfo\") pod \"e2e-shared-b-w2-031745-openshift-default-7f667bc645-lvnb4\" (UID: \"6346bb15-5ba9-4c29-b74b-3e937ba64963\") " pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-lvnb4" Oct 31 04:33:40.520540 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:40.520506 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6346bb15-5ba9-4c29-b74b-3e937ba64963-istio-token\") pod \"e2e-shared-b-w2-031745-openshift-default-7f667bc645-lvnb4\" (UID: \"6346bb15-5ba9-4c29-b74b-3e937ba64963\") " pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-lvnb4" Oct 31 04:33:40.520655 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:40.520590 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sq2qs\" (UniqueName: \"kubernetes.io/projected/6346bb15-5ba9-4c29-b74b-3e937ba64963-kube-api-access-sq2qs\") pod \"e2e-shared-b-w2-031745-openshift-default-7f667bc645-lvnb4\" (UID: \"6346bb15-5ba9-4c29-b74b-3e937ba64963\") " pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-lvnb4" Oct 31 04:33:40.565161 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:40.565133 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-lvnb4" Oct 31 04:33:40.886308 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:40.886271 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-lvnb4"] Oct 31 04:33:40.900161 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:33:40.900131 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6346bb15_5ba9_4c29_b74b_3e937ba64963.slice/crio-76ca0efe825d7c34006833a57ee39305fc5516a3159c8d509d339283fd741eee WatchSource:0}: Error finding container 76ca0efe825d7c34006833a57ee39305fc5516a3159c8d509d339283fd741eee: Status 404 returned error can't find the container with id 76ca0efe825d7c34006833a57ee39305fc5516a3159c8d509d339283fd741eee Oct 31 04:33:40.909604 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:40.909562 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:33:40.909726 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:40.909656 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:33:40.909726 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:40.909703 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:33:41.018216 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:33:41.018182 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 04:33:41.565956 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:41.565913 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-lvnb4" event={"ID":"6346bb15-5ba9-4c29-b74b-3e937ba64963","Type":"ContainerStarted","Data":"d5a316e75c816b9c855bfb28ffc126afaf4070eb4cd27ae985624dd1aa548ee2"} Oct 31 04:33:41.566406 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:41.565974 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-lvnb4" event={"ID":"6346bb15-5ba9-4c29-b74b-3e937ba64963","Type":"ContainerStarted","Data":"76ca0efe825d7c34006833a57ee39305fc5516a3159c8d509d339283fd741eee"} Oct 31 04:33:41.625460 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:41.625406 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-lvnb4" podStartSLOduration=1.625390306 podStartE2EDuration="1.625390306s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:33:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:33:41.62356327 +0000 UTC m=+1769.337897912" watchObservedRunningTime="2026-10-31 04:33:41.625390306 +0000 UTC m=+1769.339724923" Oct 31 04:33:42.566360 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:42.565615 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-lvnb4" Oct 31 04:33:42.569429 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:42.569397 4376 patch_prober.go:29] interesting pod/e2e-shared-b-w2-031745-openshift-default-7f667bc645-lvnb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body= Oct 31 04:33:42.569640 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:42.569468 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-lvnb4" podUID="6346bb15-5ba9-4c29-b74b-3e937ba64963" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" Oct 31 04:33:43.566096 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:43.565874 4376 patch_prober.go:29] interesting pod/e2e-shared-b-w2-031745-openshift-default-7f667bc645-lvnb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body= Oct 31 04:33:43.566096 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:43.565933 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-lvnb4" podUID="6346bb15-5ba9-4c29-b74b-3e937ba64963" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" Oct 31 04:33:44.567530 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:44.567496 4376 patch_prober.go:29] interesting pod/e2e-shared-b-w2-031745-openshift-default-7f667bc645-lvnb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body= Oct 31 04:33:44.568069 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:44.567573 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-lvnb4" podUID="6346bb15-5ba9-4c29-b74b-3e937ba64963" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" Oct 31 04:33:45.565586 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:45.565541 4376 patch_prober.go:29] interesting pod/e2e-shared-b-w2-031745-openshift-default-7f667bc645-lvnb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body= Oct 31 04:33:45.565803 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:45.565625 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-lvnb4" podUID="6346bb15-5ba9-4c29-b74b-3e937ba64963" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" Oct 31 04:33:46.126564 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:46.126529 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-2vsk6"] Oct 31 04:33:46.131099 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:46.131081 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-2vsk6" Oct 31 04:33:46.187890 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:46.187848 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-2vsk6"] Oct 31 04:33:46.317065 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:46.317028 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8682ee97-bdd1-4755-a5c6-9fa9d4120247-istio-envoy\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-2vsk6\" (UID: \"8682ee97-bdd1-4755-a5c6-9fa9d4120247\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-2vsk6" Oct 31 04:33:46.317065 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:46.317068 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8682ee97-bdd1-4755-a5c6-9fa9d4120247-istio-data\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-2vsk6\" (UID: \"8682ee97-bdd1-4755-a5c6-9fa9d4120247\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-2vsk6" Oct 31 04:33:46.317291 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:46.317165 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8682ee97-bdd1-4755-a5c6-9fa9d4120247-istio-podinfo\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-2vsk6\" (UID: \"8682ee97-bdd1-4755-a5c6-9fa9d4120247\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-2vsk6" Oct 31 04:33:46.317291 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:46.317219 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8682ee97-bdd1-4755-a5c6-9fa9d4120247-credential-socket\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-2vsk6\" (UID: \"8682ee97-bdd1-4755-a5c6-9fa9d4120247\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-2vsk6" Oct 31 04:33:46.317291 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:46.317278 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8682ee97-bdd1-4755-a5c6-9fa9d4120247-workload-socket\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-2vsk6\" (UID: \"8682ee97-bdd1-4755-a5c6-9fa9d4120247\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-2vsk6" Oct 31 04:33:46.317424 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:46.317300 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8682ee97-bdd1-4755-a5c6-9fa9d4120247-istiod-ca-cert\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-2vsk6\" (UID: \"8682ee97-bdd1-4755-a5c6-9fa9d4120247\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-2vsk6" Oct 31 04:33:46.317424 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:46.317340 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8whx\" (UniqueName: \"kubernetes.io/projected/8682ee97-bdd1-4755-a5c6-9fa9d4120247-kube-api-access-j8whx\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-2vsk6\" (UID: \"8682ee97-bdd1-4755-a5c6-9fa9d4120247\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-2vsk6" Oct 31 04:33:46.317424 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:46.317417 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8682ee97-bdd1-4755-a5c6-9fa9d4120247-workload-certs\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-2vsk6\" (UID: \"8682ee97-bdd1-4755-a5c6-9fa9d4120247\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-2vsk6" Oct 31 04:33:46.317519 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:46.317465 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8682ee97-bdd1-4755-a5c6-9fa9d4120247-istio-token\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-2vsk6\" (UID: \"8682ee97-bdd1-4755-a5c6-9fa9d4120247\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-2vsk6" Oct 31 04:33:46.419005 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:46.418864 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8682ee97-bdd1-4755-a5c6-9fa9d4120247-istio-envoy\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-2vsk6\" (UID: \"8682ee97-bdd1-4755-a5c6-9fa9d4120247\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-2vsk6" Oct 31 04:33:46.419005 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:46.418969 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8682ee97-bdd1-4755-a5c6-9fa9d4120247-istio-data\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-2vsk6\" (UID: \"8682ee97-bdd1-4755-a5c6-9fa9d4120247\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-2vsk6" Oct 31 04:33:46.419229 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:46.419028 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8682ee97-bdd1-4755-a5c6-9fa9d4120247-istio-podinfo\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-2vsk6\" (UID: \"8682ee97-bdd1-4755-a5c6-9fa9d4120247\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-2vsk6" Oct 31 04:33:46.419229 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:46.419087 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8682ee97-bdd1-4755-a5c6-9fa9d4120247-credential-socket\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-2vsk6\" (UID: \"8682ee97-bdd1-4755-a5c6-9fa9d4120247\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-2vsk6" Oct 31 04:33:46.419229 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:46.419151 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8682ee97-bdd1-4755-a5c6-9fa9d4120247-workload-socket\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-2vsk6\" (UID: \"8682ee97-bdd1-4755-a5c6-9fa9d4120247\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-2vsk6" Oct 31 04:33:46.419229 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:46.419179 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8682ee97-bdd1-4755-a5c6-9fa9d4120247-istiod-ca-cert\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-2vsk6\" (UID: \"8682ee97-bdd1-4755-a5c6-9fa9d4120247\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-2vsk6" Oct 31 04:33:46.419229 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:46.419211 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8whx\" (UniqueName: \"kubernetes.io/projected/8682ee97-bdd1-4755-a5c6-9fa9d4120247-kube-api-access-j8whx\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-2vsk6\" (UID: \"8682ee97-bdd1-4755-a5c6-9fa9d4120247\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-2vsk6" Oct 31 04:33:46.419539 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:46.419268 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8682ee97-bdd1-4755-a5c6-9fa9d4120247-workload-certs\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-2vsk6\" (UID: \"8682ee97-bdd1-4755-a5c6-9fa9d4120247\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-2vsk6" Oct 31 04:33:46.419539 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:46.419372 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8682ee97-bdd1-4755-a5c6-9fa9d4120247-istio-token\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-2vsk6\" (UID: \"8682ee97-bdd1-4755-a5c6-9fa9d4120247\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-2vsk6" Oct 31 04:33:46.419539 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:46.419454 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8682ee97-bdd1-4755-a5c6-9fa9d4120247-istio-data\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-2vsk6\" (UID: \"8682ee97-bdd1-4755-a5c6-9fa9d4120247\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-2vsk6" Oct 31 04:33:46.419707 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:46.419588 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8682ee97-bdd1-4755-a5c6-9fa9d4120247-credential-socket\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-2vsk6\" (UID: \"8682ee97-bdd1-4755-a5c6-9fa9d4120247\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-2vsk6" Oct 31 04:33:46.419770 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:46.419725 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8682ee97-bdd1-4755-a5c6-9fa9d4120247-workload-certs\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-2vsk6\" (UID: \"8682ee97-bdd1-4755-a5c6-9fa9d4120247\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-2vsk6" Oct 31 04:33:46.419829 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:46.419792 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8682ee97-bdd1-4755-a5c6-9fa9d4120247-workload-socket\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-2vsk6\" (UID: \"8682ee97-bdd1-4755-a5c6-9fa9d4120247\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-2vsk6" Oct 31 04:33:46.420155 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:46.420130 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8682ee97-bdd1-4755-a5c6-9fa9d4120247-istiod-ca-cert\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-2vsk6\" (UID: \"8682ee97-bdd1-4755-a5c6-9fa9d4120247\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-2vsk6" Oct 31 04:33:46.421753 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:46.421705 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8682ee97-bdd1-4755-a5c6-9fa9d4120247-istio-envoy\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-2vsk6\" (UID: \"8682ee97-bdd1-4755-a5c6-9fa9d4120247\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-2vsk6" Oct 31 04:33:46.421893 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:46.421831 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8682ee97-bdd1-4755-a5c6-9fa9d4120247-istio-podinfo\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-2vsk6\" (UID: \"8682ee97-bdd1-4755-a5c6-9fa9d4120247\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-2vsk6" Oct 31 04:33:46.468002 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:46.467956 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8682ee97-bdd1-4755-a5c6-9fa9d4120247-istio-token\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-2vsk6\" (UID: \"8682ee97-bdd1-4755-a5c6-9fa9d4120247\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-2vsk6" Oct 31 04:33:46.468181 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:46.468099 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8whx\" (UniqueName: \"kubernetes.io/projected/8682ee97-bdd1-4755-a5c6-9fa9d4120247-kube-api-access-j8whx\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-2vsk6\" (UID: \"8682ee97-bdd1-4755-a5c6-9fa9d4120247\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-2vsk6" Oct 31 04:33:46.469094 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:46.469071 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-2vsk6" Oct 31 04:33:46.580525 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:46.580485 4376 patch_prober.go:29] interesting pod/e2e-shared-b-w2-031745-openshift-default-7f667bc645-lvnb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body= Oct 31 04:33:46.580703 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:46.580554 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-lvnb4" podUID="6346bb15-5ba9-4c29-b74b-3e937ba64963" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" Oct 31 04:33:46.586398 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:46.586372 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-9633f0-7c5f7dbc86-prfm5" Oct 31 04:33:46.851349 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:46.843604 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:33:46.851349 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:46.843711 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:33:46.851349 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:46.843755 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:33:46.883689 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:46.883433 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-2vsk6"] Oct 31 04:33:47.613410 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:47.613366 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-2vsk6" event={"ID":"8682ee97-bdd1-4755-a5c6-9fa9d4120247","Type":"ContainerStarted","Data":"34ff2ca539c44439336625349c26eb6ef4a8e897cd67580a007655aa862c7fff"} Oct 31 04:33:47.613944 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:47.613431 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-2vsk6" event={"ID":"8682ee97-bdd1-4755-a5c6-9fa9d4120247","Type":"ContainerStarted","Data":"76f6d6915d759e0323029805681e8394aca80c2eb4b50c495d4eca2e986ead15"} Oct 31 04:33:47.687353 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:47.679477 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-2vsk6" podStartSLOduration=1.679454295 podStartE2EDuration="1.679454295s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:33:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:33:47.677818488 +0000 UTC m=+1775.392153133" watchObservedRunningTime="2026-10-31 04:33:47.679454295 +0000 UTC m=+1775.393788928" Oct 31 04:33:47.892316 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:47.888102 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-lvnb4" Oct 31 04:33:47.892316 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:47.891503 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-lvnb4" Oct 31 04:33:47.894925 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:47.892634 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-lvnb4" Oct 31 04:33:48.469939 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:48.469901 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-2vsk6" Oct 31 04:33:48.471597 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:48.471572 4376 patch_prober.go:29] interesting pod/e2e-worker-w0-965f61-openshift-default-55854964c-2vsk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body= Oct 31 04:33:48.471722 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:48.471617 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-2vsk6" podUID="8682ee97-bdd1-4755-a5c6-9fa9d4120247" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" Oct 31 04:33:49.469977 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:49.469936 4376 patch_prober.go:29] interesting pod/e2e-worker-w0-965f61-openshift-default-55854964c-2vsk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body= Oct 31 04:33:49.470411 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:49.470000 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-2vsk6" podUID="8682ee97-bdd1-4755-a5c6-9fa9d4120247" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" Oct 31 04:33:50.470216 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:50.470182 4376 patch_prober.go:29] interesting pod/e2e-worker-w0-965f61-openshift-default-55854964c-2vsk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body= Oct 31 04:33:50.470637 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:50.470234 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-2vsk6" podUID="8682ee97-bdd1-4755-a5c6-9fa9d4120247" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" Oct 31 04:33:51.469823 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:51.469777 4376 patch_prober.go:29] interesting pod/e2e-worker-w0-965f61-openshift-default-55854964c-2vsk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body= Oct 31 04:33:51.470015 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:51.469842 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-2vsk6" podUID="8682ee97-bdd1-4755-a5c6-9fa9d4120247" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" Oct 31 04:33:52.470184 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:52.470146 4376 patch_prober.go:29] interesting pod/e2e-worker-w0-965f61-openshift-default-55854964c-2vsk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body= Oct 31 04:33:52.470708 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:52.470204 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-2vsk6" podUID="8682ee97-bdd1-4755-a5c6-9fa9d4120247" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" Oct 31 04:33:53.470566 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:53.470526 4376 patch_prober.go:29] interesting pod/e2e-worker-w0-965f61-openshift-default-55854964c-2vsk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body= Oct 31 04:33:53.470955 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:53.470586 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-2vsk6" podUID="8682ee97-bdd1-4755-a5c6-9fa9d4120247" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" Oct 31 04:33:54.742908 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:54.742875 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-2vsk6" Oct 31 04:33:54.743459 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:54.743297 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-2vsk6" Oct 31 04:33:54.744083 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:33:54.744063 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-2vsk6" Oct 31 04:34:00.222696 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:34:00.222607 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 04:34:05.933204 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:34:05.933157 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-d6ac82-8687db9c8c-c4fl6"] Oct 31 04:34:05.945352 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:34:05.937447 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-d6ac82-8687db9c8c-c4fl6" Oct 31 04:34:05.972437 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:34:05.972393 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-d6ac82-8687db9c8c-c4fl6"] Oct 31 04:34:05.983235 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:34:05.983208 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-d6ac82\"" Oct 31 04:34:05.984184 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:34:05.984113 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w1-d6ac82-dockercfg-fxjk6\"" Oct 31 04:34:05.996872 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:34:05.996844 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-d6ac82-75d47769c-bb76f"] Oct 31 04:34:06.002072 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:34:06.002038 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-d6ac82-75d47769c-bb76f" Oct 31 04:34:06.018759 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:34:06.018729 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7sbn\" (UniqueName: \"kubernetes.io/projected/adda31ec-5cfd-473c-8edd-5e7d335aec12-kube-api-access-k7sbn\") pod \"payload-pre-processing-e2e-shared-b-w1-d6ac82-8687db9c8c-c4fl6\" (UID: \"adda31ec-5cfd-473c-8edd-5e7d335aec12\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-d6ac82-8687db9c8c-c4fl6" Oct 31 04:34:06.018864 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:34:06.018804 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xz99d\" (UniqueName: \"kubernetes.io/projected/a0071e27-190b-4977-9c6f-c3458c714d64-kube-api-access-xz99d\") pod \"payload-processing-e2e-shared-b-w1-d6ac82-75d47769c-bb76f\" (UID: \"a0071e27-190b-4977-9c6f-c3458c714d64\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-d6ac82-75d47769c-bb76f" Oct 31 04:34:06.019007 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:34:06.018987 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/adda31ec-5cfd-473c-8edd-5e7d335aec12-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-d6ac82-8687db9c8c-c4fl6\" (UID: \"adda31ec-5cfd-473c-8edd-5e7d335aec12\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-d6ac82-8687db9c8c-c4fl6" Oct 31 04:34:06.019147 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:34:06.019130 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0071e27-190b-4977-9c6f-c3458c714d64-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-d6ac82-75d47769c-bb76f\" (UID: \"a0071e27-190b-4977-9c6f-c3458c714d64\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-d6ac82-75d47769c-bb76f" Oct 31 04:34:06.024488 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:34:06.024464 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-d6ac82-75d47769c-bb76f"] Oct 31 04:34:06.123370 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:34:06.121631 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7sbn\" (UniqueName: \"kubernetes.io/projected/adda31ec-5cfd-473c-8edd-5e7d335aec12-kube-api-access-k7sbn\") pod \"payload-pre-processing-e2e-shared-b-w1-d6ac82-8687db9c8c-c4fl6\" (UID: \"adda31ec-5cfd-473c-8edd-5e7d335aec12\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-d6ac82-8687db9c8c-c4fl6" Oct 31 04:34:06.123370 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:34:06.121740 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xz99d\" (UniqueName: \"kubernetes.io/projected/a0071e27-190b-4977-9c6f-c3458c714d64-kube-api-access-xz99d\") pod \"payload-processing-e2e-shared-b-w1-d6ac82-75d47769c-bb76f\" (UID: \"a0071e27-190b-4977-9c6f-c3458c714d64\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-d6ac82-75d47769c-bb76f" Oct 31 04:34:06.123370 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:34:06.121886 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/adda31ec-5cfd-473c-8edd-5e7d335aec12-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-d6ac82-8687db9c8c-c4fl6\" (UID: \"adda31ec-5cfd-473c-8edd-5e7d335aec12\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-d6ac82-8687db9c8c-c4fl6" Oct 31 04:34:06.123370 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:34:06.121989 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0071e27-190b-4977-9c6f-c3458c714d64-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-d6ac82-75d47769c-bb76f\" (UID: \"a0071e27-190b-4977-9c6f-c3458c714d64\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-d6ac82-75d47769c-bb76f" Oct 31 04:34:06.123737 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:34:06.123714 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/adda31ec-5cfd-473c-8edd-5e7d335aec12-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-d6ac82-8687db9c8c-c4fl6\" (UID: \"adda31ec-5cfd-473c-8edd-5e7d335aec12\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-d6ac82-8687db9c8c-c4fl6" Oct 31 04:34:06.124655 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:34:06.124462 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0071e27-190b-4977-9c6f-c3458c714d64-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-d6ac82-75d47769c-bb76f\" (UID: \"a0071e27-190b-4977-9c6f-c3458c714d64\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-d6ac82-75d47769c-bb76f" Oct 31 04:34:06.172373 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:34:06.172343 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7sbn\" (UniqueName: \"kubernetes.io/projected/adda31ec-5cfd-473c-8edd-5e7d335aec12-kube-api-access-k7sbn\") pod \"payload-pre-processing-e2e-shared-b-w1-d6ac82-8687db9c8c-c4fl6\" (UID: \"adda31ec-5cfd-473c-8edd-5e7d335aec12\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-d6ac82-8687db9c8c-c4fl6" Oct 31 04:34:06.181401 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:34:06.181308 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xz99d\" (UniqueName: \"kubernetes.io/projected/a0071e27-190b-4977-9c6f-c3458c714d64-kube-api-access-xz99d\") pod \"payload-processing-e2e-shared-b-w1-d6ac82-75d47769c-bb76f\" (UID: \"a0071e27-190b-4977-9c6f-c3458c714d64\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-d6ac82-75d47769c-bb76f" Oct 31 04:34:06.291001 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:34:06.290913 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-d6ac82-8687db9c8c-c4fl6" Oct 31 04:34:06.320555 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:34:06.320525 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-d6ac82-75d47769c-bb76f" Oct 31 04:34:06.673919 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:34:06.673690 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-d6ac82-8687db9c8c-c4fl6"] Oct 31 04:34:06.719876 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:34:06.719835 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0071e27_190b_4977_9c6f_c3458c714d64.slice/crio-a5aadd551e81cc654994327612114143c1894e7c5207792b58063e5148ac1b6c WatchSource:0}: Error finding container a5aadd551e81cc654994327612114143c1894e7c5207792b58063e5148ac1b6c: Status 404 returned error can't find the container with id a5aadd551e81cc654994327612114143c1894e7c5207792b58063e5148ac1b6c Oct 31 04:34:06.722204 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:34:06.722165 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-d6ac82-75d47769c-bb76f"] Oct 31 04:34:06.990477 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:34:06.988969 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-d6ac82-75d47769c-bb76f" event={"ID":"a0071e27-190b-4977-9c6f-c3458c714d64","Type":"ContainerStarted","Data":"a5aadd551e81cc654994327612114143c1894e7c5207792b58063e5148ac1b6c"} Oct 31 04:34:07.003396 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:34:07.003300 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-d6ac82-8687db9c8c-c4fl6" event={"ID":"adda31ec-5cfd-473c-8edd-5e7d335aec12","Type":"ContainerStarted","Data":"23a64a7c21ca76331d22954118444061e6cfa973f3160c34d8947fefa1f08a61"} Oct 31 04:34:08.013176 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:34:08.013133 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-d6ac82-75d47769c-bb76f" event={"ID":"a0071e27-190b-4977-9c6f-c3458c714d64","Type":"ContainerStarted","Data":"2c93fdbb4630ac291697246dd077e2008daf68851d44c57151ffe6b21ad4a622"} Oct 31 04:34:08.013687 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:34:08.013278 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-d6ac82-75d47769c-bb76f" Oct 31 04:34:08.014936 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:34:08.014906 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-d6ac82-8687db9c8c-c4fl6" event={"ID":"adda31ec-5cfd-473c-8edd-5e7d335aec12","Type":"ContainerStarted","Data":"2dbfcb68e1ad6f89a0159ea41616e9fc93671218625c0f197dd23c46cea2569f"} Oct 31 04:34:08.015053 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:34:08.014965 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-d6ac82-8687db9c8c-c4fl6" Oct 31 04:34:08.057366 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:34:08.057278 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-d6ac82-75d47769c-bb76f" podStartSLOduration=3.057258739 podStartE2EDuration="3.057258739s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:34:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:34:08.054829801 +0000 UTC m=+1795.769164440" watchObservedRunningTime="2026-10-31 04:34:08.057258739 +0000 UTC m=+1795.771593357" Oct 31 04:34:08.103142 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:34:08.103092 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-d6ac82-8687db9c8c-c4fl6" podStartSLOduration=3.103079219 podStartE2EDuration="3.103079219s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:34:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:34:08.098909432 +0000 UTC m=+1795.813244076" watchObservedRunningTime="2026-10-31 04:34:08.103079219 +0000 UTC m=+1795.817413836" Oct 31 04:34:11.002706 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:34:11.002667 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 04:34:12.063592 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:34:12.063559 4376 generic.go:356] "Generic (PLEG): container finished" podID="9e23ed56-ce2f-47b9-bd20-d22f8aa8a3ae" containerID="49d599099893cf360611bee983fceaf09d3b863db4c87b8c129e8a2c6a2b3058" exitCode=0 Oct 31 04:34:12.064046 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:34:12.063613 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-7229f2-openshift-default-7b5c69df47-7tnqr" event={"ID":"9e23ed56-ce2f-47b9-bd20-d22f8aa8a3ae","Type":"ContainerDied","Data":"49d599099893cf360611bee983fceaf09d3b863db4c87b8c129e8a2c6a2b3058"} Oct 31 04:34:12.064089 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:34:12.064068 4376 scope.go:122] "RemoveContainer" containerID="49d599099893cf360611bee983fceaf09d3b863db4c87b8c129e8a2c6a2b3058" Oct 31 04:34:12.065934 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:34:12.065901 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:34:12.066017 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:34:12.065998 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:34:12.066063 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:34:12.066050 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:34:13.143497 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:34:13.143453 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-7229f2-openshift-default-7b5c69df47-7tnqr" event={"ID":"9e23ed56-ce2f-47b9-bd20-d22f8aa8a3ae","Type":"ContainerStarted","Data":"151122b13fc1797023d86e6a7b30641f76fb43af6fee9e427c15ccbbaf04ac61"} Oct 31 04:34:13.335426 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:34:13.335388 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-7229f2-openshift-default-7b5c69df47-7tnqr" Oct 31 04:34:13.335661 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:34:13.335637 4376 patch_prober.go:29] interesting pod/e2e-autores-7229f2-openshift-default-7b5c69df47-7tnqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body= Oct 31 04:34:13.335708 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:34:13.335692 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-7229f2-openshift-default-7b5c69df47-7tnqr" podUID="9e23ed56-ce2f-47b9-bd20-d22f8aa8a3ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" Oct 31 04:34:14.335425 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:34:14.335386 4376 patch_prober.go:29] interesting pod/e2e-autores-7229f2-openshift-default-7b5c69df47-7tnqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body= Oct 31 04:34:14.335922 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:34:14.335450 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-7229f2-openshift-default-7b5c69df47-7tnqr" podUID="9e23ed56-ce2f-47b9-bd20-d22f8aa8a3ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" Oct 31 04:34:15.335893 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:34:15.335855 4376 patch_prober.go:29] interesting pod/e2e-autores-7229f2-openshift-default-7b5c69df47-7tnqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body= Oct 31 04:34:15.336313 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:34:15.335910 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-7229f2-openshift-default-7b5c69df47-7tnqr" podUID="9e23ed56-ce2f-47b9-bd20-d22f8aa8a3ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" Oct 31 04:34:16.335569 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:34:16.335533 4376 patch_prober.go:29] interesting pod/e2e-autores-7229f2-openshift-default-7b5c69df47-7tnqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body= Oct 31 04:34:16.335767 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:34:16.335604 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-7229f2-openshift-default-7b5c69df47-7tnqr" podUID="9e23ed56-ce2f-47b9-bd20-d22f8aa8a3ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" Oct 31 04:34:17.335898 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:34:17.335854 4376 patch_prober.go:29] interesting pod/e2e-autores-7229f2-openshift-default-7b5c69df47-7tnqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body= Oct 31 04:34:17.336417 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:34:17.335916 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-7229f2-openshift-default-7b5c69df47-7tnqr" podUID="9e23ed56-ce2f-47b9-bd20-d22f8aa8a3ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" Oct 31 04:34:18.268428 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:34:18.268393 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-7229f2-openshift-default-7b5c69df47-7tnqr" Oct 31 04:34:18.336393 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:34:18.336320 4376 patch_prober.go:29] interesting pod/e2e-autores-7229f2-openshift-default-7b5c69df47-7tnqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body= Oct 31 04:34:18.336910 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:34:18.336420 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-7229f2-openshift-default-7b5c69df47-7tnqr" podUID="9e23ed56-ce2f-47b9-bd20-d22f8aa8a3ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" Oct 31 04:34:19.049746 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:34:19.049711 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-d6ac82-8687db9c8c-c4fl6" Oct 31 04:34:19.050523 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:34:19.050474 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-d6ac82-75d47769c-bb76f" Oct 31 04:34:19.408350 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:34:19.407623 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-7229f2-openshift-default-7b5c69df47-7tnqr" Oct 31 04:34:19.425348 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:34:19.423587 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-7229f2-openshift-default-7b5c69df47-7tnqr" Oct 31 04:34:25.372542 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:34:25.372509 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-rs8fm"] Oct 31 04:34:25.378711 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:34:25.378687 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-rs8fm" Oct 31 04:34:25.442574 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:34:25.442528 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjxl5\" (UniqueName: \"kubernetes.io/projected/360e59d6-8295-48fd-a8d2-2e7fe1e4f402-kube-api-access-sjxl5\") pod \"e2e-shared-b-w2-031745-openshift-default-7f667bc645-rs8fm\" (UID: \"360e59d6-8295-48fd-a8d2-2e7fe1e4f402\") " pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-rs8fm" Oct 31 04:34:25.442914 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:34:25.442886 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/360e59d6-8295-48fd-a8d2-2e7fe1e4f402-istio-data\") pod \"e2e-shared-b-w2-031745-openshift-default-7f667bc645-rs8fm\" (UID: \"360e59d6-8295-48fd-a8d2-2e7fe1e4f402\") " pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-rs8fm" Oct 31 04:34:25.443244 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:34:25.443220 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/360e59d6-8295-48fd-a8d2-2e7fe1e4f402-workload-socket\") pod \"e2e-shared-b-w2-031745-openshift-default-7f667bc645-rs8fm\" (UID: \"360e59d6-8295-48fd-a8d2-2e7fe1e4f402\") " pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-rs8fm" Oct 31 04:34:25.443457 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:34:25.443437 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/360e59d6-8295-48fd-a8d2-2e7fe1e4f402-istio-podinfo\") pod \"e2e-shared-b-w2-031745-openshift-default-7f667bc645-rs8fm\" (UID: \"360e59d6-8295-48fd-a8d2-2e7fe1e4f402\") " pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-rs8fm" Oct 31 04:34:25.443628 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:34:25.443611 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/360e59d6-8295-48fd-a8d2-2e7fe1e4f402-istio-envoy\") pod \"e2e-shared-b-w2-031745-openshift-default-7f667bc645-rs8fm\" (UID: \"360e59d6-8295-48fd-a8d2-2e7fe1e4f402\") " pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-rs8fm" Oct 31 04:34:25.443837 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:34:25.443818 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/360e59d6-8295-48fd-a8d2-2e7fe1e4f402-istiod-ca-cert\") pod \"e2e-shared-b-w2-031745-openshift-default-7f667bc645-rs8fm\" (UID: \"360e59d6-8295-48fd-a8d2-2e7fe1e4f402\") " pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-rs8fm" Oct 31 04:34:25.443963 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:34:25.443946 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/360e59d6-8295-48fd-a8d2-2e7fe1e4f402-workload-certs\") pod \"e2e-shared-b-w2-031745-openshift-default-7f667bc645-rs8fm\" (UID: \"360e59d6-8295-48fd-a8d2-2e7fe1e4f402\") " pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-rs8fm" Oct 31 04:34:25.444144 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:34:25.444125 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/360e59d6-8295-48fd-a8d2-2e7fe1e4f402-credential-socket\") pod \"e2e-shared-b-w2-031745-openshift-default-7f667bc645-rs8fm\" (UID: \"360e59d6-8295-48fd-a8d2-2e7fe1e4f402\") " pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-rs8fm" Oct 31 04:34:25.444366 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:34:25.444347 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/360e59d6-8295-48fd-a8d2-2e7fe1e4f402-istio-token\") pod \"e2e-shared-b-w2-031745-openshift-default-7f667bc645-rs8fm\" (UID: \"360e59d6-8295-48fd-a8d2-2e7fe1e4f402\") " pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-rs8fm" Oct 31 04:34:25.461090 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:34:25.461054 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-rs8fm"] Oct 31 04:34:25.546058 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:34:25.546019 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/360e59d6-8295-48fd-a8d2-2e7fe1e4f402-credential-socket\") pod \"e2e-shared-b-w2-031745-openshift-default-7f667bc645-rs8fm\" (UID: \"360e59d6-8295-48fd-a8d2-2e7fe1e4f402\") " pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-rs8fm" Oct 31 04:34:25.546256 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:34:25.546107 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/360e59d6-8295-48fd-a8d2-2e7fe1e4f402-istio-token\") pod \"e2e-shared-b-w2-031745-openshift-default-7f667bc645-rs8fm\" (UID: \"360e59d6-8295-48fd-a8d2-2e7fe1e4f402\") " pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-rs8fm" Oct 31 04:34:25.546256 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:34:25.546185 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjxl5\" (UniqueName: \"kubernetes.io/projected/360e59d6-8295-48fd-a8d2-2e7fe1e4f402-kube-api-access-sjxl5\") pod \"e2e-shared-b-w2-031745-openshift-default-7f667bc645-rs8fm\" (UID: \"360e59d6-8295-48fd-a8d2-2e7fe1e4f402\") " pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-rs8fm" Oct 31 04:34:25.546256 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:34:25.546213 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/360e59d6-8295-48fd-a8d2-2e7fe1e4f402-istio-data\") pod \"e2e-shared-b-w2-031745-openshift-default-7f667bc645-rs8fm\" (UID: \"360e59d6-8295-48fd-a8d2-2e7fe1e4f402\") " pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-rs8fm" Oct 31 04:34:25.546473 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:34:25.546344 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/360e59d6-8295-48fd-a8d2-2e7fe1e4f402-workload-socket\") pod \"e2e-shared-b-w2-031745-openshift-default-7f667bc645-rs8fm\" (UID: \"360e59d6-8295-48fd-a8d2-2e7fe1e4f402\") " pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-rs8fm" Oct 31 04:34:25.546473 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:34:25.546375 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/360e59d6-8295-48fd-a8d2-2e7fe1e4f402-istio-podinfo\") pod \"e2e-shared-b-w2-031745-openshift-default-7f667bc645-rs8fm\" (UID: \"360e59d6-8295-48fd-a8d2-2e7fe1e4f402\") " pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-rs8fm" Oct 31 04:34:25.546473 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:34:25.546435 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/360e59d6-8295-48fd-a8d2-2e7fe1e4f402-istio-envoy\") pod \"e2e-shared-b-w2-031745-openshift-default-7f667bc645-rs8fm\" (UID: \"360e59d6-8295-48fd-a8d2-2e7fe1e4f402\") " pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-rs8fm" Oct 31 04:34:25.546614 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:34:25.546486 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/360e59d6-8295-48fd-a8d2-2e7fe1e4f402-credential-socket\") pod \"e2e-shared-b-w2-031745-openshift-default-7f667bc645-rs8fm\" (UID: \"360e59d6-8295-48fd-a8d2-2e7fe1e4f402\") " pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-rs8fm" Oct 31 04:34:25.546614 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:34:25.546575 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/360e59d6-8295-48fd-a8d2-2e7fe1e4f402-istiod-ca-cert\") pod \"e2e-shared-b-w2-031745-openshift-default-7f667bc645-rs8fm\" (UID: \"360e59d6-8295-48fd-a8d2-2e7fe1e4f402\") " pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-rs8fm" Oct 31 04:34:25.546705 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:34:25.546622 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/360e59d6-8295-48fd-a8d2-2e7fe1e4f402-workload-certs\") pod \"e2e-shared-b-w2-031745-openshift-default-7f667bc645-rs8fm\" (UID: \"360e59d6-8295-48fd-a8d2-2e7fe1e4f402\") " pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-rs8fm" Oct 31 04:34:25.546705 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:34:25.546666 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/360e59d6-8295-48fd-a8d2-2e7fe1e4f402-istio-data\") pod \"e2e-shared-b-w2-031745-openshift-default-7f667bc645-rs8fm\" (UID: \"360e59d6-8295-48fd-a8d2-2e7fe1e4f402\") " pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-rs8fm" Oct 31 04:34:25.546807 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:34:25.546712 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/360e59d6-8295-48fd-a8d2-2e7fe1e4f402-workload-socket\") pod \"e2e-shared-b-w2-031745-openshift-default-7f667bc645-rs8fm\" (UID: \"360e59d6-8295-48fd-a8d2-2e7fe1e4f402\") " pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-rs8fm" Oct 31 04:34:25.547001 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:34:25.546981 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/360e59d6-8295-48fd-a8d2-2e7fe1e4f402-workload-certs\") pod \"e2e-shared-b-w2-031745-openshift-default-7f667bc645-rs8fm\" (UID: \"360e59d6-8295-48fd-a8d2-2e7fe1e4f402\") " pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-rs8fm" Oct 31 04:34:25.547252 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:34:25.547230 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/360e59d6-8295-48fd-a8d2-2e7fe1e4f402-istiod-ca-cert\") pod \"e2e-shared-b-w2-031745-openshift-default-7f667bc645-rs8fm\" (UID: \"360e59d6-8295-48fd-a8d2-2e7fe1e4f402\") " pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-rs8fm" Oct 31 04:34:25.548738 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:34:25.548715 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/360e59d6-8295-48fd-a8d2-2e7fe1e4f402-istio-envoy\") pod \"e2e-shared-b-w2-031745-openshift-default-7f667bc645-rs8fm\" (UID: \"360e59d6-8295-48fd-a8d2-2e7fe1e4f402\") " pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-rs8fm" Oct 31 04:34:25.549241 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:34:25.549220 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/360e59d6-8295-48fd-a8d2-2e7fe1e4f402-istio-podinfo\") pod \"e2e-shared-b-w2-031745-openshift-default-7f667bc645-rs8fm\" (UID: \"360e59d6-8295-48fd-a8d2-2e7fe1e4f402\") " pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-rs8fm" Oct 31 04:34:25.570207 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:34:25.570186 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/360e59d6-8295-48fd-a8d2-2e7fe1e4f402-istio-token\") pod \"e2e-shared-b-w2-031745-openshift-default-7f667bc645-rs8fm\" (UID: \"360e59d6-8295-48fd-a8d2-2e7fe1e4f402\") " pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-rs8fm" Oct 31 04:34:25.570698 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:34:25.570648 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjxl5\" (UniqueName: \"kubernetes.io/projected/360e59d6-8295-48fd-a8d2-2e7fe1e4f402-kube-api-access-sjxl5\") pod \"e2e-shared-b-w2-031745-openshift-default-7f667bc645-rs8fm\" (UID: \"360e59d6-8295-48fd-a8d2-2e7fe1e4f402\") " pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-rs8fm" Oct 31 04:34:25.788466 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:34:25.788387 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-rs8fm" Oct 31 04:34:26.105949 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:34:26.105923 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod360e59d6_8295_48fd_a8d2_2e7fe1e4f402.slice/crio-b570658e1a489a3f0dd32cb35ab901342af22ac396b73e34a65483f4e8d32ff3 WatchSource:0}: Error finding container b570658e1a489a3f0dd32cb35ab901342af22ac396b73e34a65483f4e8d32ff3: Status 404 returned error can't find the container with id b570658e1a489a3f0dd32cb35ab901342af22ac396b73e34a65483f4e8d32ff3 Oct 31 04:34:26.107570 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:34:26.107541 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-rs8fm"] Oct 31 04:34:26.109157 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:34:26.109119 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:34:26.109258 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:34:26.109193 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:34:26.109258 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:34:26.109223 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:34:26.269422 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:34:26.269371 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-rs8fm" event={"ID":"360e59d6-8295-48fd-a8d2-2e7fe1e4f402","Type":"ContainerStarted","Data":"07af01032d3a5a87cd5d9e32bb49c9d0b5fc89561af67062aa6a19b20aa70153"} Oct 31 04:34:26.269604 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:34:26.269446 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-rs8fm" event={"ID":"360e59d6-8295-48fd-a8d2-2e7fe1e4f402","Type":"ContainerStarted","Data":"b570658e1a489a3f0dd32cb35ab901342af22ac396b73e34a65483f4e8d32ff3"} Oct 31 04:34:26.315531 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:34:26.315470 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-rs8fm" podStartSLOduration=1.315451313 podStartE2EDuration="1.315451313s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:34:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:34:26.315031191 +0000 UTC m=+1814.029365827" watchObservedRunningTime="2026-10-31 04:34:26.315451313 +0000 UTC m=+1814.029785930" Oct 31 04:34:26.788910 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:34:26.788876 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-rs8fm" Oct 31 04:34:27.790777 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:34:27.790738 4376 patch_prober.go:29] interesting pod/e2e-shared-b-w2-031745-openshift-default-7f667bc645-rs8fm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" start-of-body= Oct 31 04:34:27.791389 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:34:27.790796 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-rs8fm" podUID="360e59d6-8295-48fd-a8d2-2e7fe1e4f402" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" Oct 31 04:34:28.789478 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:34:28.789439 4376 patch_prober.go:29] interesting pod/e2e-shared-b-w2-031745-openshift-default-7f667bc645-rs8fm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" start-of-body= Oct 31 04:34:28.789657 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:34:28.789495 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-rs8fm" podUID="360e59d6-8295-48fd-a8d2-2e7fe1e4f402" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" Oct 31 04:34:29.789278 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:34:29.789241 4376 patch_prober.go:29] interesting pod/e2e-shared-b-w2-031745-openshift-default-7f667bc645-rs8fm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" start-of-body= Oct 31 04:34:29.789735 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:34:29.789314 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-rs8fm" podUID="360e59d6-8295-48fd-a8d2-2e7fe1e4f402" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" Oct 31 04:34:29.802945 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:34:29.802914 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w4-57dc6d/auth-test-model-57dc6d-kserve-577cb5ffd5-2wk4v"] Oct 31 04:34:29.807863 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:34:29.807831 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-57dc6d/auth-test-model-57dc6d-kserve-577cb5ffd5-2wk4v" Oct 31 04:34:29.827659 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:34:29.827632 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-57dc6d/auth-test-model-57dc6d-kserve-577cb5ffd5-2wk4v"] Oct 31 04:34:29.835674 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:34:29.835654 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-57dc6d\"/\"openshift-service-ca.crt\"" Oct 31 04:34:29.835827 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:34:29.835643 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-57dc6d\"/\"auth-test-model-57dc6d-kserve-self-signed-certs\"" Oct 31 04:34:29.835905 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:34:29.835686 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-57dc6d\"/\"default-dockercfg-4r69d\"" Oct 31 04:34:29.836089 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:34:29.836069 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-57dc6d\"/\"kube-root-ca.crt\"" Oct 31 04:34:29.971084 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:34:29.971051 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b9cb676-3050-4ea5-b38b-3d32d10ae997-model-cache\") pod \"auth-test-model-57dc6d-kserve-577cb5ffd5-2wk4v\" (UID: \"1b9cb676-3050-4ea5-b38b-3d32d10ae997\") " pod="ai-tenant-e2e-shared-a-w4-57dc6d/auth-test-model-57dc6d-kserve-577cb5ffd5-2wk4v" Oct 31 04:34:29.971358 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:34:29.971099 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b9cb676-3050-4ea5-b38b-3d32d10ae997-tmp-dir\") pod \"auth-test-model-57dc6d-kserve-577cb5ffd5-2wk4v\" (UID: \"1b9cb676-3050-4ea5-b38b-3d32d10ae997\") " pod="ai-tenant-e2e-shared-a-w4-57dc6d/auth-test-model-57dc6d-kserve-577cb5ffd5-2wk4v" Oct 31 04:34:29.971358 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:34:29.971177 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b9cb676-3050-4ea5-b38b-3d32d10ae997-home\") pod \"auth-test-model-57dc6d-kserve-577cb5ffd5-2wk4v\" (UID: \"1b9cb676-3050-4ea5-b38b-3d32d10ae997\") " pod="ai-tenant-e2e-shared-a-w4-57dc6d/auth-test-model-57dc6d-kserve-577cb5ffd5-2wk4v" Oct 31 04:34:29.971358 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:34:29.971278 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7rdg\" (UniqueName: \"kubernetes.io/projected/1b9cb676-3050-4ea5-b38b-3d32d10ae997-kube-api-access-r7rdg\") pod \"auth-test-model-57dc6d-kserve-577cb5ffd5-2wk4v\" (UID: \"1b9cb676-3050-4ea5-b38b-3d32d10ae997\") " pod="ai-tenant-e2e-shared-a-w4-57dc6d/auth-test-model-57dc6d-kserve-577cb5ffd5-2wk4v" Oct 31 04:34:29.971358 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:34:29.971310 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b9cb676-3050-4ea5-b38b-3d32d10ae997-dshm\") pod \"auth-test-model-57dc6d-kserve-577cb5ffd5-2wk4v\" (UID: \"1b9cb676-3050-4ea5-b38b-3d32d10ae997\") " pod="ai-tenant-e2e-shared-a-w4-57dc6d/auth-test-model-57dc6d-kserve-577cb5ffd5-2wk4v" Oct 31 04:34:29.971580 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:34:29.971395 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b9cb676-3050-4ea5-b38b-3d32d10ae997-tls-certs\") pod \"auth-test-model-57dc6d-kserve-577cb5ffd5-2wk4v\" (UID: \"1b9cb676-3050-4ea5-b38b-3d32d10ae997\") " pod="ai-tenant-e2e-shared-a-w4-57dc6d/auth-test-model-57dc6d-kserve-577cb5ffd5-2wk4v" Oct 31 04:34:30.077735 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:34:30.076693 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b9cb676-3050-4ea5-b38b-3d32d10ae997-model-cache\") pod \"auth-test-model-57dc6d-kserve-577cb5ffd5-2wk4v\" (UID: \"1b9cb676-3050-4ea5-b38b-3d32d10ae997\") " pod="ai-tenant-e2e-shared-a-w4-57dc6d/auth-test-model-57dc6d-kserve-577cb5ffd5-2wk4v" Oct 31 04:34:30.077735 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:34:30.076764 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b9cb676-3050-4ea5-b38b-3d32d10ae997-tmp-dir\") pod \"auth-test-model-57dc6d-kserve-577cb5ffd5-2wk4v\" (UID: \"1b9cb676-3050-4ea5-b38b-3d32d10ae997\") " pod="ai-tenant-e2e-shared-a-w4-57dc6d/auth-test-model-57dc6d-kserve-577cb5ffd5-2wk4v" Oct 31 04:34:30.077735 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:34:30.076813 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b9cb676-3050-4ea5-b38b-3d32d10ae997-home\") pod \"auth-test-model-57dc6d-kserve-577cb5ffd5-2wk4v\" (UID: \"1b9cb676-3050-4ea5-b38b-3d32d10ae997\") " pod="ai-tenant-e2e-shared-a-w4-57dc6d/auth-test-model-57dc6d-kserve-577cb5ffd5-2wk4v" Oct 31 04:34:30.077735 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:34:30.076890 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7rdg\" (UniqueName: \"kubernetes.io/projected/1b9cb676-3050-4ea5-b38b-3d32d10ae997-kube-api-access-r7rdg\") pod \"auth-test-model-57dc6d-kserve-577cb5ffd5-2wk4v\" (UID: \"1b9cb676-3050-4ea5-b38b-3d32d10ae997\") " pod="ai-tenant-e2e-shared-a-w4-57dc6d/auth-test-model-57dc6d-kserve-577cb5ffd5-2wk4v" Oct 31 04:34:30.077735 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:34:30.076920 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b9cb676-3050-4ea5-b38b-3d32d10ae997-dshm\") pod \"auth-test-model-57dc6d-kserve-577cb5ffd5-2wk4v\" (UID: \"1b9cb676-3050-4ea5-b38b-3d32d10ae997\") " pod="ai-tenant-e2e-shared-a-w4-57dc6d/auth-test-model-57dc6d-kserve-577cb5ffd5-2wk4v" Oct 31 04:34:30.077735 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:34:30.076992 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b9cb676-3050-4ea5-b38b-3d32d10ae997-tls-certs\") pod \"auth-test-model-57dc6d-kserve-577cb5ffd5-2wk4v\" (UID: \"1b9cb676-3050-4ea5-b38b-3d32d10ae997\") " pod="ai-tenant-e2e-shared-a-w4-57dc6d/auth-test-model-57dc6d-kserve-577cb5ffd5-2wk4v" Oct 31 04:34:30.078790 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:34:30.078757 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b9cb676-3050-4ea5-b38b-3d32d10ae997-home\") pod \"auth-test-model-57dc6d-kserve-577cb5ffd5-2wk4v\" (UID: \"1b9cb676-3050-4ea5-b38b-3d32d10ae997\") " pod="ai-tenant-e2e-shared-a-w4-57dc6d/auth-test-model-57dc6d-kserve-577cb5ffd5-2wk4v" Oct 31 04:34:30.079195 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:34:30.079174 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b9cb676-3050-4ea5-b38b-3d32d10ae997-model-cache\") pod \"auth-test-model-57dc6d-kserve-577cb5ffd5-2wk4v\" (UID: \"1b9cb676-3050-4ea5-b38b-3d32d10ae997\") " pod="ai-tenant-e2e-shared-a-w4-57dc6d/auth-test-model-57dc6d-kserve-577cb5ffd5-2wk4v" Oct 31 04:34:30.083656 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:34:30.083630 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b9cb676-3050-4ea5-b38b-3d32d10ae997-tmp-dir\") pod \"auth-test-model-57dc6d-kserve-577cb5ffd5-2wk4v\" (UID: \"1b9cb676-3050-4ea5-b38b-3d32d10ae997\") " pod="ai-tenant-e2e-shared-a-w4-57dc6d/auth-test-model-57dc6d-kserve-577cb5ffd5-2wk4v" Oct 31 04:34:30.089416 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:34:30.086803 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b9cb676-3050-4ea5-b38b-3d32d10ae997-tls-certs\") pod \"auth-test-model-57dc6d-kserve-577cb5ffd5-2wk4v\" (UID: \"1b9cb676-3050-4ea5-b38b-3d32d10ae997\") " pod="ai-tenant-e2e-shared-a-w4-57dc6d/auth-test-model-57dc6d-kserve-577cb5ffd5-2wk4v" Oct 31 04:34:30.104180 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:34:30.103728 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b9cb676-3050-4ea5-b38b-3d32d10ae997-dshm\") pod \"auth-test-model-57dc6d-kserve-577cb5ffd5-2wk4v\" (UID: \"1b9cb676-3050-4ea5-b38b-3d32d10ae997\") " pod="ai-tenant-e2e-shared-a-w4-57dc6d/auth-test-model-57dc6d-kserve-577cb5ffd5-2wk4v" Oct 31 04:34:30.117567 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:34:30.117469 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7rdg\" (UniqueName: \"kubernetes.io/projected/1b9cb676-3050-4ea5-b38b-3d32d10ae997-kube-api-access-r7rdg\") pod \"auth-test-model-57dc6d-kserve-577cb5ffd5-2wk4v\" (UID: \"1b9cb676-3050-4ea5-b38b-3d32d10ae997\") " pod="ai-tenant-e2e-shared-a-w4-57dc6d/auth-test-model-57dc6d-kserve-577cb5ffd5-2wk4v" Oct 31 04:34:30.124938 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:34:30.124635 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-57dc6d/auth-test-model-57dc6d-kserve-577cb5ffd5-2wk4v" Oct 31 04:34:30.575261 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:34:30.575231 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1b9cb676_3050_4ea5_b38b_3d32d10ae997.slice/crio-939dcfa6431b8031a10cda52b3e8dce2dc37c444c17f75902e10e55bdc7fd0c2 WatchSource:0}: Error finding container 939dcfa6431b8031a10cda52b3e8dce2dc37c444c17f75902e10e55bdc7fd0c2: Status 404 returned error can't find the container with id 939dcfa6431b8031a10cda52b3e8dce2dc37c444c17f75902e10e55bdc7fd0c2 Oct 31 04:34:30.577542 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:34:30.577516 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-57dc6d/auth-test-model-57dc6d-kserve-577cb5ffd5-2wk4v"] Oct 31 04:34:30.789437 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:34:30.789404 4376 patch_prober.go:29] interesting pod/e2e-shared-b-w2-031745-openshift-default-7f667bc645-rs8fm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" start-of-body= Oct 31 04:34:30.789939 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:34:30.789453 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-rs8fm" podUID="360e59d6-8295-48fd-a8d2-2e7fe1e4f402" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" Oct 31 04:34:31.321924 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:34:31.321885 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-57dc6d/auth-test-model-57dc6d-kserve-577cb5ffd5-2wk4v" event={"ID":"1b9cb676-3050-4ea5-b38b-3d32d10ae997","Type":"ContainerStarted","Data":"4c4ba863c4a2081f438ce295c6954539ad3cf932c40a67ddc686eb7138f22386"} Oct 31 04:34:31.321924 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:34:31.321930 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-57dc6d/auth-test-model-57dc6d-kserve-577cb5ffd5-2wk4v" event={"ID":"1b9cb676-3050-4ea5-b38b-3d32d10ae997","Type":"ContainerStarted","Data":"939dcfa6431b8031a10cda52b3e8dce2dc37c444c17f75902e10e55bdc7fd0c2"} Oct 31 04:34:31.368117 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:34:31.368053 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w4-57dc6d/auth-test-model-57dc6d-kserve-577cb5ffd5-2wk4v" podStartSLOduration=2.368032086 podStartE2EDuration="2.368032086s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:34:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:34:31.364779618 +0000 UTC m=+1819.079114236" watchObservedRunningTime="2026-10-31 04:34:31.368032086 +0000 UTC m=+1819.082366706" Oct 31 04:34:31.798353 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:34:31.790417 4376 patch_prober.go:29] interesting pod/e2e-shared-b-w2-031745-openshift-default-7f667bc645-rs8fm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" start-of-body= Oct 31 04:34:31.798353 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:34:31.790477 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-rs8fm" podUID="360e59d6-8295-48fd-a8d2-2e7fe1e4f402" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" Oct 31 04:34:32.125423 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:34:32.125383 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-57dc6d/auth-test-model-57dc6d-kserve-577cb5ffd5-2wk4v" Oct 31 04:34:32.905516 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:34:32.905480 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-rs8fm" Oct 31 04:34:32.906587 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:34:32.906554 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-rs8fm" Oct 31 04:34:32.908249 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:34:32.908227 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-rs8fm" Oct 31 04:34:33.009911 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:34:33.009874 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 04:34:40.126219 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:34:40.126174 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w4-57dc6d/auth-test-model-57dc6d-kserve-577cb5ffd5-2wk4v" Oct 31 04:34:40.139391 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:34:40.139359 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w4-57dc6d/auth-test-model-57dc6d-kserve-577cb5ffd5-2wk4v" Oct 31 04:34:40.394427 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:34:40.394347 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-57dc6d/auth-test-model-57dc6d-kserve-577cb5ffd5-2wk4v" Oct 31 04:34:41.001208 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:34:41.001172 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 04:35:11.000800 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:35:11.000754 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 04:35:15.375451 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:35:15.375410 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w4-9633f0/auth-test-model-9633f0-kserve-6cbd55ffd9-z9rjx"] Oct 31 04:35:15.379473 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:35:15.379451 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-9633f0/auth-test-model-9633f0-kserve-6cbd55ffd9-z9rjx" Oct 31 04:35:15.399613 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:35:15.399589 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-9633f0/auth-test-model-9633f0-kserve-6cbd55ffd9-z9rjx"] Oct 31 04:35:15.412530 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:35:15.412508 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-9633f0\"/\"kube-root-ca.crt\"" Oct 31 04:35:15.412530 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:35:15.412520 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-9633f0\"/\"openshift-service-ca.crt\"" Oct 31 04:35:15.412724 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:35:15.412585 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-9633f0\"/\"auth-test-model-9633f0-kserve-self-signed-certs\"" Oct 31 04:35:15.412724 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:35:15.412669 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-9633f0\"/\"default-dockercfg-bwbnz\"" Oct 31 04:35:15.484951 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:35:15.484916 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64j2q\" (UniqueName: \"kubernetes.io/projected/5b72272f-1227-401e-bec7-56182f8c1676-kube-api-access-64j2q\") pod \"auth-test-model-9633f0-kserve-6cbd55ffd9-z9rjx\" (UID: \"5b72272f-1227-401e-bec7-56182f8c1676\") " pod="ai-tenant-e2e-shared-b-w4-9633f0/auth-test-model-9633f0-kserve-6cbd55ffd9-z9rjx" Oct 31 04:35:15.485132 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:35:15.485058 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b72272f-1227-401e-bec7-56182f8c1676-tmp-dir\") pod \"auth-test-model-9633f0-kserve-6cbd55ffd9-z9rjx\" (UID: \"5b72272f-1227-401e-bec7-56182f8c1676\") " pod="ai-tenant-e2e-shared-b-w4-9633f0/auth-test-model-9633f0-kserve-6cbd55ffd9-z9rjx" Oct 31 04:35:15.485197 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:35:15.485174 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b72272f-1227-401e-bec7-56182f8c1676-tls-certs\") pod \"auth-test-model-9633f0-kserve-6cbd55ffd9-z9rjx\" (UID: \"5b72272f-1227-401e-bec7-56182f8c1676\") " pod="ai-tenant-e2e-shared-b-w4-9633f0/auth-test-model-9633f0-kserve-6cbd55ffd9-z9rjx" Oct 31 04:35:15.485244 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:35:15.485212 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b72272f-1227-401e-bec7-56182f8c1676-dshm\") pod \"auth-test-model-9633f0-kserve-6cbd55ffd9-z9rjx\" (UID: \"5b72272f-1227-401e-bec7-56182f8c1676\") " pod="ai-tenant-e2e-shared-b-w4-9633f0/auth-test-model-9633f0-kserve-6cbd55ffd9-z9rjx" Oct 31 04:35:15.485358 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:35:15.485308 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b72272f-1227-401e-bec7-56182f8c1676-home\") pod \"auth-test-model-9633f0-kserve-6cbd55ffd9-z9rjx\" (UID: \"5b72272f-1227-401e-bec7-56182f8c1676\") " pod="ai-tenant-e2e-shared-b-w4-9633f0/auth-test-model-9633f0-kserve-6cbd55ffd9-z9rjx" Oct 31 04:35:15.485474 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:35:15.485458 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b72272f-1227-401e-bec7-56182f8c1676-model-cache\") pod \"auth-test-model-9633f0-kserve-6cbd55ffd9-z9rjx\" (UID: \"5b72272f-1227-401e-bec7-56182f8c1676\") " pod="ai-tenant-e2e-shared-b-w4-9633f0/auth-test-model-9633f0-kserve-6cbd55ffd9-z9rjx" Oct 31 04:35:15.631392 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:35:15.619098 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b72272f-1227-401e-bec7-56182f8c1676-tmp-dir\") pod \"auth-test-model-9633f0-kserve-6cbd55ffd9-z9rjx\" (UID: \"5b72272f-1227-401e-bec7-56182f8c1676\") " pod="ai-tenant-e2e-shared-b-w4-9633f0/auth-test-model-9633f0-kserve-6cbd55ffd9-z9rjx" Oct 31 04:35:15.631392 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:35:15.619243 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b72272f-1227-401e-bec7-56182f8c1676-tls-certs\") pod \"auth-test-model-9633f0-kserve-6cbd55ffd9-z9rjx\" (UID: \"5b72272f-1227-401e-bec7-56182f8c1676\") " pod="ai-tenant-e2e-shared-b-w4-9633f0/auth-test-model-9633f0-kserve-6cbd55ffd9-z9rjx" Oct 31 04:35:15.631392 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:35:15.619282 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b72272f-1227-401e-bec7-56182f8c1676-dshm\") pod \"auth-test-model-9633f0-kserve-6cbd55ffd9-z9rjx\" (UID: \"5b72272f-1227-401e-bec7-56182f8c1676\") " pod="ai-tenant-e2e-shared-b-w4-9633f0/auth-test-model-9633f0-kserve-6cbd55ffd9-z9rjx" Oct 31 04:35:15.631392 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:35:15.619428 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b72272f-1227-401e-bec7-56182f8c1676-home\") pod \"auth-test-model-9633f0-kserve-6cbd55ffd9-z9rjx\" (UID: \"5b72272f-1227-401e-bec7-56182f8c1676\") " pod="ai-tenant-e2e-shared-b-w4-9633f0/auth-test-model-9633f0-kserve-6cbd55ffd9-z9rjx" Oct 31 04:35:15.631392 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:35:15.619586 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b72272f-1227-401e-bec7-56182f8c1676-model-cache\") pod \"auth-test-model-9633f0-kserve-6cbd55ffd9-z9rjx\" (UID: \"5b72272f-1227-401e-bec7-56182f8c1676\") " pod="ai-tenant-e2e-shared-b-w4-9633f0/auth-test-model-9633f0-kserve-6cbd55ffd9-z9rjx" Oct 31 04:35:15.631392 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:35:15.619700 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64j2q\" (UniqueName: \"kubernetes.io/projected/5b72272f-1227-401e-bec7-56182f8c1676-kube-api-access-64j2q\") pod \"auth-test-model-9633f0-kserve-6cbd55ffd9-z9rjx\" (UID: \"5b72272f-1227-401e-bec7-56182f8c1676\") " pod="ai-tenant-e2e-shared-b-w4-9633f0/auth-test-model-9633f0-kserve-6cbd55ffd9-z9rjx" Oct 31 04:35:15.631392 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:35:15.619732 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b72272f-1227-401e-bec7-56182f8c1676-tmp-dir\") pod \"auth-test-model-9633f0-kserve-6cbd55ffd9-z9rjx\" (UID: \"5b72272f-1227-401e-bec7-56182f8c1676\") " pod="ai-tenant-e2e-shared-b-w4-9633f0/auth-test-model-9633f0-kserve-6cbd55ffd9-z9rjx" Oct 31 04:35:15.631392 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:35:15.620705 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b72272f-1227-401e-bec7-56182f8c1676-home\") pod \"auth-test-model-9633f0-kserve-6cbd55ffd9-z9rjx\" (UID: \"5b72272f-1227-401e-bec7-56182f8c1676\") " pod="ai-tenant-e2e-shared-b-w4-9633f0/auth-test-model-9633f0-kserve-6cbd55ffd9-z9rjx" Oct 31 04:35:15.631392 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:35:15.620717 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b72272f-1227-401e-bec7-56182f8c1676-model-cache\") pod \"auth-test-model-9633f0-kserve-6cbd55ffd9-z9rjx\" (UID: \"5b72272f-1227-401e-bec7-56182f8c1676\") " pod="ai-tenant-e2e-shared-b-w4-9633f0/auth-test-model-9633f0-kserve-6cbd55ffd9-z9rjx" Oct 31 04:35:15.643993 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:35:15.643619 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b72272f-1227-401e-bec7-56182f8c1676-dshm\") pod \"auth-test-model-9633f0-kserve-6cbd55ffd9-z9rjx\" (UID: \"5b72272f-1227-401e-bec7-56182f8c1676\") " pod="ai-tenant-e2e-shared-b-w4-9633f0/auth-test-model-9633f0-kserve-6cbd55ffd9-z9rjx" Oct 31 04:35:15.664372 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:35:15.661639 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b72272f-1227-401e-bec7-56182f8c1676-tls-certs\") pod \"auth-test-model-9633f0-kserve-6cbd55ffd9-z9rjx\" (UID: \"5b72272f-1227-401e-bec7-56182f8c1676\") " pod="ai-tenant-e2e-shared-b-w4-9633f0/auth-test-model-9633f0-kserve-6cbd55ffd9-z9rjx" Oct 31 04:35:15.687159 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:35:15.687122 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64j2q\" (UniqueName: \"kubernetes.io/projected/5b72272f-1227-401e-bec7-56182f8c1676-kube-api-access-64j2q\") pod \"auth-test-model-9633f0-kserve-6cbd55ffd9-z9rjx\" (UID: \"5b72272f-1227-401e-bec7-56182f8c1676\") " pod="ai-tenant-e2e-shared-b-w4-9633f0/auth-test-model-9633f0-kserve-6cbd55ffd9-z9rjx" Oct 31 04:35:15.705661 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:35:15.705618 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-9633f0/auth-test-model-9633f0-kserve-6cbd55ffd9-z9rjx" Oct 31 04:35:16.093622 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:35:16.093592 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5b72272f_1227_401e_bec7_56182f8c1676.slice/crio-83f97ef109af6888444a7318d395e4f49dd59628d9b4e2ebe728a06295730d9f WatchSource:0}: Error finding container 83f97ef109af6888444a7318d395e4f49dd59628d9b4e2ebe728a06295730d9f: Status 404 returned error can't find the container with id 83f97ef109af6888444a7318d395e4f49dd59628d9b4e2ebe728a06295730d9f Oct 31 04:35:16.095866 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:35:16.095836 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-9633f0/auth-test-model-9633f0-kserve-6cbd55ffd9-z9rjx"] Oct 31 04:35:16.759577 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:35:16.759540 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-9633f0/auth-test-model-9633f0-kserve-6cbd55ffd9-z9rjx" event={"ID":"5b72272f-1227-401e-bec7-56182f8c1676","Type":"ContainerStarted","Data":"242a852c6e0a38931e92511cfb8ede33e98a65859b0dfc9faa3b6515218f81d3"} Oct 31 04:35:16.759958 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:35:16.759585 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-9633f0/auth-test-model-9633f0-kserve-6cbd55ffd9-z9rjx" event={"ID":"5b72272f-1227-401e-bec7-56182f8c1676","Type":"ContainerStarted","Data":"83f97ef109af6888444a7318d395e4f49dd59628d9b4e2ebe728a06295730d9f"} Oct 31 04:35:16.805453 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:35:16.805403 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w4-9633f0/auth-test-model-9633f0-kserve-6cbd55ffd9-z9rjx" podStartSLOduration=1.8053908939999999 podStartE2EDuration="1.805390894s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:35:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:35:16.801079366 +0000 UTC m=+1864.515414027" watchObservedRunningTime="2026-10-31 04:35:16.805390894 +0000 UTC m=+1864.519725510" Oct 31 04:35:17.706676 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:35:17.706649 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-9633f0/auth-test-model-9633f0-kserve-6cbd55ffd9-z9rjx" Oct 31 04:35:19.150604 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:35:19.150571 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41acf6c0-gxhc6"] Oct 31 04:35:19.159930 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:35:19.159906 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41acf6c0-gxhc6"] Oct 31 04:35:20.453222 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:35:20.453189 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="acc6c2f6-b64e-4940-b7f3-47a27f16a811" path="/var/lib/kubelet/pods/acc6c2f6-b64e-4940-b7f3-47a27f16a811/volumes" Oct 31 04:35:25.707055 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:35:25.707021 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w4-9633f0/auth-test-model-9633f0-kserve-6cbd55ffd9-z9rjx" Oct 31 04:35:25.719154 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:35:25.719128 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w4-9633f0/auth-test-model-9633f0-kserve-6cbd55ffd9-z9rjx" Oct 31 04:35:25.840696 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:35:25.840661 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-9633f0/auth-test-model-9633f0-kserve-6cbd55ffd9-z9rjx" Oct 31 04:35:36.523109 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:35:36.523071 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 04:35:41.002150 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:35:41.002111 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 04:35:58.931778 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:35:58.931743 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-77a153-openshift-default-7778f45d97-pzlhr"] Oct 31 04:35:58.936774 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:35:58.936752 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-77a153-openshift-default-7778f45d97-pzlhr" Oct 31 04:35:58.950026 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:35:58.949999 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/faaa68df-a884-400f-be9a-84bbb75ad520-istio-envoy\") pod \"e2e-shared-a-w1-77a153-openshift-default-7778f45d97-pzlhr\" (UID: \"faaa68df-a884-400f-be9a-84bbb75ad520\") " pod="openshift-ingress/e2e-shared-a-w1-77a153-openshift-default-7778f45d97-pzlhr" Oct 31 04:35:58.950139 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:35:58.950047 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/faaa68df-a884-400f-be9a-84bbb75ad520-istiod-ca-cert\") pod \"e2e-shared-a-w1-77a153-openshift-default-7778f45d97-pzlhr\" (UID: \"faaa68df-a884-400f-be9a-84bbb75ad520\") " pod="openshift-ingress/e2e-shared-a-w1-77a153-openshift-default-7778f45d97-pzlhr" Oct 31 04:35:58.950195 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:35:58.950155 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rgkp\" (UniqueName: \"kubernetes.io/projected/faaa68df-a884-400f-be9a-84bbb75ad520-kube-api-access-2rgkp\") pod \"e2e-shared-a-w1-77a153-openshift-default-7778f45d97-pzlhr\" (UID: \"faaa68df-a884-400f-be9a-84bbb75ad520\") " pod="openshift-ingress/e2e-shared-a-w1-77a153-openshift-default-7778f45d97-pzlhr" Oct 31 04:35:58.950240 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:35:58.950196 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/faaa68df-a884-400f-be9a-84bbb75ad520-istio-podinfo\") pod \"e2e-shared-a-w1-77a153-openshift-default-7778f45d97-pzlhr\" (UID: \"faaa68df-a884-400f-be9a-84bbb75ad520\") " pod="openshift-ingress/e2e-shared-a-w1-77a153-openshift-default-7778f45d97-pzlhr" Oct 31 04:35:58.950353 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:35:58.950302 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/faaa68df-a884-400f-be9a-84bbb75ad520-istio-token\") pod \"e2e-shared-a-w1-77a153-openshift-default-7778f45d97-pzlhr\" (UID: \"faaa68df-a884-400f-be9a-84bbb75ad520\") " pod="openshift-ingress/e2e-shared-a-w1-77a153-openshift-default-7778f45d97-pzlhr" Oct 31 04:35:58.950432 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:35:58.950413 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/faaa68df-a884-400f-be9a-84bbb75ad520-istio-data\") pod \"e2e-shared-a-w1-77a153-openshift-default-7778f45d97-pzlhr\" (UID: \"faaa68df-a884-400f-be9a-84bbb75ad520\") " pod="openshift-ingress/e2e-shared-a-w1-77a153-openshift-default-7778f45d97-pzlhr" Oct 31 04:35:58.950471 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:35:58.950457 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/faaa68df-a884-400f-be9a-84bbb75ad520-workload-certs\") pod \"e2e-shared-a-w1-77a153-openshift-default-7778f45d97-pzlhr\" (UID: \"faaa68df-a884-400f-be9a-84bbb75ad520\") " pod="openshift-ingress/e2e-shared-a-w1-77a153-openshift-default-7778f45d97-pzlhr" Oct 31 04:35:58.950540 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:35:58.950527 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/faaa68df-a884-400f-be9a-84bbb75ad520-credential-socket\") pod \"e2e-shared-a-w1-77a153-openshift-default-7778f45d97-pzlhr\" (UID: \"faaa68df-a884-400f-be9a-84bbb75ad520\") " pod="openshift-ingress/e2e-shared-a-w1-77a153-openshift-default-7778f45d97-pzlhr" Oct 31 04:35:58.950575 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:35:58.950566 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/faaa68df-a884-400f-be9a-84bbb75ad520-workload-socket\") pod \"e2e-shared-a-w1-77a153-openshift-default-7778f45d97-pzlhr\" (UID: \"faaa68df-a884-400f-be9a-84bbb75ad520\") " pod="openshift-ingress/e2e-shared-a-w1-77a153-openshift-default-7778f45d97-pzlhr" Oct 31 04:35:58.952633 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:35:58.952614 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-77a153-openshift-default-dockercfg-wr9c4\"" Oct 31 04:35:58.969983 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:35:58.969961 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-77a153-openshift-default-7778f45d97-pzlhr"] Oct 31 04:35:59.051314 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:35:59.051284 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/faaa68df-a884-400f-be9a-84bbb75ad520-istio-envoy\") pod \"e2e-shared-a-w1-77a153-openshift-default-7778f45d97-pzlhr\" (UID: \"faaa68df-a884-400f-be9a-84bbb75ad520\") " pod="openshift-ingress/e2e-shared-a-w1-77a153-openshift-default-7778f45d97-pzlhr" Oct 31 04:35:59.051495 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:35:59.051364 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/faaa68df-a884-400f-be9a-84bbb75ad520-istiod-ca-cert\") pod \"e2e-shared-a-w1-77a153-openshift-default-7778f45d97-pzlhr\" (UID: \"faaa68df-a884-400f-be9a-84bbb75ad520\") " pod="openshift-ingress/e2e-shared-a-w1-77a153-openshift-default-7778f45d97-pzlhr" Oct 31 04:35:59.051495 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:35:59.051408 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rgkp\" (UniqueName: \"kubernetes.io/projected/faaa68df-a884-400f-be9a-84bbb75ad520-kube-api-access-2rgkp\") pod \"e2e-shared-a-w1-77a153-openshift-default-7778f45d97-pzlhr\" (UID: \"faaa68df-a884-400f-be9a-84bbb75ad520\") " pod="openshift-ingress/e2e-shared-a-w1-77a153-openshift-default-7778f45d97-pzlhr" Oct 31 04:35:59.051495 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:35:59.051444 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/faaa68df-a884-400f-be9a-84bbb75ad520-istio-podinfo\") pod \"e2e-shared-a-w1-77a153-openshift-default-7778f45d97-pzlhr\" (UID: \"faaa68df-a884-400f-be9a-84bbb75ad520\") " pod="openshift-ingress/e2e-shared-a-w1-77a153-openshift-default-7778f45d97-pzlhr" Oct 31 04:35:59.051495 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:35:59.051492 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/faaa68df-a884-400f-be9a-84bbb75ad520-istio-token\") pod \"e2e-shared-a-w1-77a153-openshift-default-7778f45d97-pzlhr\" (UID: \"faaa68df-a884-400f-be9a-84bbb75ad520\") " pod="openshift-ingress/e2e-shared-a-w1-77a153-openshift-default-7778f45d97-pzlhr" Oct 31 04:35:59.051739 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:35:59.051636 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/faaa68df-a884-400f-be9a-84bbb75ad520-istio-data\") pod \"e2e-shared-a-w1-77a153-openshift-default-7778f45d97-pzlhr\" (UID: \"faaa68df-a884-400f-be9a-84bbb75ad520\") " pod="openshift-ingress/e2e-shared-a-w1-77a153-openshift-default-7778f45d97-pzlhr" Oct 31 04:35:59.051739 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:35:59.051689 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/faaa68df-a884-400f-be9a-84bbb75ad520-workload-certs\") pod \"e2e-shared-a-w1-77a153-openshift-default-7778f45d97-pzlhr\" (UID: \"faaa68df-a884-400f-be9a-84bbb75ad520\") " pod="openshift-ingress/e2e-shared-a-w1-77a153-openshift-default-7778f45d97-pzlhr" Oct 31 04:35:59.051850 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:35:59.051772 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/faaa68df-a884-400f-be9a-84bbb75ad520-credential-socket\") pod \"e2e-shared-a-w1-77a153-openshift-default-7778f45d97-pzlhr\" (UID: \"faaa68df-a884-400f-be9a-84bbb75ad520\") " pod="openshift-ingress/e2e-shared-a-w1-77a153-openshift-default-7778f45d97-pzlhr" Oct 31 04:35:59.051850 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:35:59.051823 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/faaa68df-a884-400f-be9a-84bbb75ad520-workload-socket\") pod \"e2e-shared-a-w1-77a153-openshift-default-7778f45d97-pzlhr\" (UID: \"faaa68df-a884-400f-be9a-84bbb75ad520\") " pod="openshift-ingress/e2e-shared-a-w1-77a153-openshift-default-7778f45d97-pzlhr" Oct 31 04:35:59.052071 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:35:59.051973 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/faaa68df-a884-400f-be9a-84bbb75ad520-istio-data\") pod \"e2e-shared-a-w1-77a153-openshift-default-7778f45d97-pzlhr\" (UID: \"faaa68df-a884-400f-be9a-84bbb75ad520\") " pod="openshift-ingress/e2e-shared-a-w1-77a153-openshift-default-7778f45d97-pzlhr" Oct 31 04:35:59.052071 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:35:59.052013 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/faaa68df-a884-400f-be9a-84bbb75ad520-workload-certs\") pod \"e2e-shared-a-w1-77a153-openshift-default-7778f45d97-pzlhr\" (UID: \"faaa68df-a884-400f-be9a-84bbb75ad520\") " pod="openshift-ingress/e2e-shared-a-w1-77a153-openshift-default-7778f45d97-pzlhr" Oct 31 04:35:59.052359 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:35:59.052190 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/faaa68df-a884-400f-be9a-84bbb75ad520-istiod-ca-cert\") pod \"e2e-shared-a-w1-77a153-openshift-default-7778f45d97-pzlhr\" (UID: \"faaa68df-a884-400f-be9a-84bbb75ad520\") " pod="openshift-ingress/e2e-shared-a-w1-77a153-openshift-default-7778f45d97-pzlhr" Oct 31 04:35:59.052359 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:35:59.052283 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/faaa68df-a884-400f-be9a-84bbb75ad520-credential-socket\") pod \"e2e-shared-a-w1-77a153-openshift-default-7778f45d97-pzlhr\" (UID: \"faaa68df-a884-400f-be9a-84bbb75ad520\") " pod="openshift-ingress/e2e-shared-a-w1-77a153-openshift-default-7778f45d97-pzlhr" Oct 31 04:35:59.052507 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:35:59.052397 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/faaa68df-a884-400f-be9a-84bbb75ad520-workload-socket\") pod \"e2e-shared-a-w1-77a153-openshift-default-7778f45d97-pzlhr\" (UID: \"faaa68df-a884-400f-be9a-84bbb75ad520\") " pod="openshift-ingress/e2e-shared-a-w1-77a153-openshift-default-7778f45d97-pzlhr" Oct 31 04:35:59.053781 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:35:59.053757 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/faaa68df-a884-400f-be9a-84bbb75ad520-istio-envoy\") pod \"e2e-shared-a-w1-77a153-openshift-default-7778f45d97-pzlhr\" (UID: \"faaa68df-a884-400f-be9a-84bbb75ad520\") " pod="openshift-ingress/e2e-shared-a-w1-77a153-openshift-default-7778f45d97-pzlhr" Oct 31 04:35:59.053955 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:35:59.053937 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/faaa68df-a884-400f-be9a-84bbb75ad520-istio-podinfo\") pod \"e2e-shared-a-w1-77a153-openshift-default-7778f45d97-pzlhr\" (UID: \"faaa68df-a884-400f-be9a-84bbb75ad520\") " pod="openshift-ingress/e2e-shared-a-w1-77a153-openshift-default-7778f45d97-pzlhr" Oct 31 04:35:59.075021 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:35:59.074999 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/faaa68df-a884-400f-be9a-84bbb75ad520-istio-token\") pod \"e2e-shared-a-w1-77a153-openshift-default-7778f45d97-pzlhr\" (UID: \"faaa68df-a884-400f-be9a-84bbb75ad520\") " pod="openshift-ingress/e2e-shared-a-w1-77a153-openshift-default-7778f45d97-pzlhr" Oct 31 04:35:59.075465 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:35:59.075443 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rgkp\" (UniqueName: \"kubernetes.io/projected/faaa68df-a884-400f-be9a-84bbb75ad520-kube-api-access-2rgkp\") pod \"e2e-shared-a-w1-77a153-openshift-default-7778f45d97-pzlhr\" (UID: \"faaa68df-a884-400f-be9a-84bbb75ad520\") " pod="openshift-ingress/e2e-shared-a-w1-77a153-openshift-default-7778f45d97-pzlhr" Oct 31 04:35:59.252190 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:35:59.252104 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-77a153-openshift-default-7778f45d97-pzlhr" Oct 31 04:35:59.608549 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:35:59.594112 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-77a153-openshift-default-7778f45d97-pzlhr"] Oct 31 04:35:59.621625 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:35:59.615283 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:35:59.621625 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:35:59.618514 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:35:59.621625 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:35:59.618572 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:36:00.144149 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:36:00.144108 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-77a153-openshift-default-7778f45d97-pzlhr" event={"ID":"faaa68df-a884-400f-be9a-84bbb75ad520","Type":"ContainerStarted","Data":"2cbacd986d060bd433c53bf878c3d80a4fc6242f8e7f248db63a2b2a26633f17"} Oct 31 04:36:00.144579 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:36:00.144174 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-77a153-openshift-default-7778f45d97-pzlhr" event={"ID":"faaa68df-a884-400f-be9a-84bbb75ad520","Type":"ContainerStarted","Data":"7a439005c5a6e2ade2ef8d6bacf159b08cdb236b532162271b79b66b3e9434b1"} Oct 31 04:36:00.204383 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:36:00.204236 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-77a153-openshift-default-7778f45d97-pzlhr" podStartSLOduration=2.204219569 podStartE2EDuration="2.204219569s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:35:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:36:00.203599622 +0000 UTC m=+1907.917934238" watchObservedRunningTime="2026-10-31 04:36:00.204219569 +0000 UTC m=+1907.918554188" Oct 31 04:36:00.252582 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:36:00.252551 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-77a153-openshift-default-7778f45d97-pzlhr" Oct 31 04:36:00.254204 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:36:00.254183 4376 patch_prober.go:29] interesting pod/e2e-shared-a-w1-77a153-openshift-default-7778f45d97-pzlhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body= Oct 31 04:36:00.254319 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:36:00.254221 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-77a153-openshift-default-7778f45d97-pzlhr" podUID="faaa68df-a884-400f-be9a-84bbb75ad520" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" Oct 31 04:36:01.262395 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:36:01.259939 4376 patch_prober.go:29] interesting pod/e2e-shared-a-w1-77a153-openshift-default-7778f45d97-pzlhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body= Oct 31 04:36:01.262395 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:36:01.260013 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-77a153-openshift-default-7778f45d97-pzlhr" podUID="faaa68df-a884-400f-be9a-84bbb75ad520" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" Oct 31 04:36:01.429355 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:36:01.427377 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-5n2gs"] Oct 31 04:36:01.506274 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:36:01.505763 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-5n2gs" Oct 31 04:36:01.523486 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:36:01.523404 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-5n2gs"] Oct 31 04:36:01.554351 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:36:01.535952 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-b2a372-openshift-default-dockercfg-44djb\"" Oct 31 04:36:01.611300 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:36:01.611268 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d057ba0f-e65d-44de-949f-bc94f3c20550-istiod-ca-cert\") pod \"e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-5n2gs\" (UID: \"d057ba0f-e65d-44de-949f-bc94f3c20550\") " pod="openshift-ingress/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-5n2gs" Oct 31 04:36:01.611469 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:36:01.611349 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d057ba0f-e65d-44de-949f-bc94f3c20550-workload-certs\") pod \"e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-5n2gs\" (UID: \"d057ba0f-e65d-44de-949f-bc94f3c20550\") " pod="openshift-ingress/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-5n2gs" Oct 31 04:36:01.611526 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:36:01.611467 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d057ba0f-e65d-44de-949f-bc94f3c20550-istio-podinfo\") pod \"e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-5n2gs\" (UID: \"d057ba0f-e65d-44de-949f-bc94f3c20550\") " pod="openshift-ingress/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-5n2gs" Oct 31 04:36:01.611632 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:36:01.611604 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d057ba0f-e65d-44de-949f-bc94f3c20550-workload-socket\") pod \"e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-5n2gs\" (UID: \"d057ba0f-e65d-44de-949f-bc94f3c20550\") " pod="openshift-ingress/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-5n2gs" Oct 31 04:36:01.611705 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:36:01.611676 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d057ba0f-e65d-44de-949f-bc94f3c20550-istio-envoy\") pod \"e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-5n2gs\" (UID: \"d057ba0f-e65d-44de-949f-bc94f3c20550\") " pod="openshift-ingress/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-5n2gs" Oct 31 04:36:01.611901 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:36:01.611875 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8vjx\" (UniqueName: \"kubernetes.io/projected/d057ba0f-e65d-44de-949f-bc94f3c20550-kube-api-access-q8vjx\") pod \"e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-5n2gs\" (UID: \"d057ba0f-e65d-44de-949f-bc94f3c20550\") " pod="openshift-ingress/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-5n2gs" Oct 31 04:36:01.611988 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:36:01.611972 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d057ba0f-e65d-44de-949f-bc94f3c20550-istio-data\") pod \"e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-5n2gs\" (UID: \"d057ba0f-e65d-44de-949f-bc94f3c20550\") " pod="openshift-ingress/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-5n2gs" Oct 31 04:36:01.612166 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:36:01.612142 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d057ba0f-e65d-44de-949f-bc94f3c20550-credential-socket\") pod \"e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-5n2gs\" (UID: \"d057ba0f-e65d-44de-949f-bc94f3c20550\") " pod="openshift-ingress/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-5n2gs" Oct 31 04:36:01.612243 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:36:01.612210 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d057ba0f-e65d-44de-949f-bc94f3c20550-istio-token\") pod \"e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-5n2gs\" (UID: \"d057ba0f-e65d-44de-949f-bc94f3c20550\") " pod="openshift-ingress/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-5n2gs" Oct 31 04:36:01.714016 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:36:01.713984 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d057ba0f-e65d-44de-949f-bc94f3c20550-istio-token\") pod \"e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-5n2gs\" (UID: \"d057ba0f-e65d-44de-949f-bc94f3c20550\") " pod="openshift-ingress/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-5n2gs" Oct 31 04:36:01.714185 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:36:01.714028 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d057ba0f-e65d-44de-949f-bc94f3c20550-istiod-ca-cert\") pod \"e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-5n2gs\" (UID: \"d057ba0f-e65d-44de-949f-bc94f3c20550\") " pod="openshift-ingress/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-5n2gs" Oct 31 04:36:01.714185 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:36:01.714074 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d057ba0f-e65d-44de-949f-bc94f3c20550-workload-certs\") pod \"e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-5n2gs\" (UID: \"d057ba0f-e65d-44de-949f-bc94f3c20550\") " pod="openshift-ingress/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-5n2gs" Oct 31 04:36:01.714185 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:36:01.714103 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d057ba0f-e65d-44de-949f-bc94f3c20550-istio-podinfo\") pod \"e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-5n2gs\" (UID: \"d057ba0f-e65d-44de-949f-bc94f3c20550\") " pod="openshift-ingress/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-5n2gs" Oct 31 04:36:01.714185 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:36:01.714154 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d057ba0f-e65d-44de-949f-bc94f3c20550-workload-socket\") pod \"e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-5n2gs\" (UID: \"d057ba0f-e65d-44de-949f-bc94f3c20550\") " pod="openshift-ingress/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-5n2gs" Oct 31 04:36:01.714469 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:36:01.714188 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d057ba0f-e65d-44de-949f-bc94f3c20550-istio-envoy\") pod \"e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-5n2gs\" (UID: \"d057ba0f-e65d-44de-949f-bc94f3c20550\") " pod="openshift-ingress/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-5n2gs" Oct 31 04:36:01.714469 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:36:01.714262 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8vjx\" (UniqueName: \"kubernetes.io/projected/d057ba0f-e65d-44de-949f-bc94f3c20550-kube-api-access-q8vjx\") pod \"e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-5n2gs\" (UID: \"d057ba0f-e65d-44de-949f-bc94f3c20550\") " pod="openshift-ingress/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-5n2gs" Oct 31 04:36:01.714469 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:36:01.714285 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d057ba0f-e65d-44de-949f-bc94f3c20550-istio-data\") pod \"e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-5n2gs\" (UID: \"d057ba0f-e65d-44de-949f-bc94f3c20550\") " pod="openshift-ingress/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-5n2gs" Oct 31 04:36:01.714469 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:36:01.714375 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d057ba0f-e65d-44de-949f-bc94f3c20550-credential-socket\") pod \"e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-5n2gs\" (UID: \"d057ba0f-e65d-44de-949f-bc94f3c20550\") " pod="openshift-ingress/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-5n2gs" Oct 31 04:36:01.714672 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:36:01.714528 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d057ba0f-e65d-44de-949f-bc94f3c20550-workload-certs\") pod \"e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-5n2gs\" (UID: \"d057ba0f-e65d-44de-949f-bc94f3c20550\") " pod="openshift-ingress/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-5n2gs" Oct 31 04:36:01.714672 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:36:01.714643 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d057ba0f-e65d-44de-949f-bc94f3c20550-workload-socket\") pod \"e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-5n2gs\" (UID: \"d057ba0f-e65d-44de-949f-bc94f3c20550\") " pod="openshift-ingress/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-5n2gs" Oct 31 04:36:01.714796 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:36:01.714747 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d057ba0f-e65d-44de-949f-bc94f3c20550-istio-data\") pod \"e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-5n2gs\" (UID: \"d057ba0f-e65d-44de-949f-bc94f3c20550\") " pod="openshift-ingress/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-5n2gs" Oct 31 04:36:01.714796 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:36:01.714780 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d057ba0f-e65d-44de-949f-bc94f3c20550-credential-socket\") pod \"e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-5n2gs\" (UID: \"d057ba0f-e65d-44de-949f-bc94f3c20550\") " pod="openshift-ingress/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-5n2gs" Oct 31 04:36:01.714922 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:36:01.714901 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d057ba0f-e65d-44de-949f-bc94f3c20550-istiod-ca-cert\") pod \"e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-5n2gs\" (UID: \"d057ba0f-e65d-44de-949f-bc94f3c20550\") " pod="openshift-ingress/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-5n2gs" Oct 31 04:36:01.716526 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:36:01.716503 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d057ba0f-e65d-44de-949f-bc94f3c20550-istio-envoy\") pod \"e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-5n2gs\" (UID: \"d057ba0f-e65d-44de-949f-bc94f3c20550\") " pod="openshift-ingress/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-5n2gs" Oct 31 04:36:01.717104 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:36:01.717081 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d057ba0f-e65d-44de-949f-bc94f3c20550-istio-podinfo\") pod \"e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-5n2gs\" (UID: \"d057ba0f-e65d-44de-949f-bc94f3c20550\") " pod="openshift-ingress/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-5n2gs" Oct 31 04:36:01.762351 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:36:01.753964 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8vjx\" (UniqueName: \"kubernetes.io/projected/d057ba0f-e65d-44de-949f-bc94f3c20550-kube-api-access-q8vjx\") pod \"e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-5n2gs\" (UID: \"d057ba0f-e65d-44de-949f-bc94f3c20550\") " pod="openshift-ingress/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-5n2gs" Oct 31 04:36:01.762844 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:36:01.762783 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d057ba0f-e65d-44de-949f-bc94f3c20550-istio-token\") pod \"e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-5n2gs\" (UID: \"d057ba0f-e65d-44de-949f-bc94f3c20550\") " pod="openshift-ingress/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-5n2gs" Oct 31 04:36:01.905863 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:36:01.905834 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-5n2gs" Oct 31 04:36:02.260776 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:36:02.260689 4376 patch_prober.go:29] interesting pod/e2e-shared-a-w1-77a153-openshift-default-7778f45d97-pzlhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body= Oct 31 04:36:02.261054 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:36:02.261023 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-77a153-openshift-default-7778f45d97-pzlhr" podUID="faaa68df-a884-400f-be9a-84bbb75ad520" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" Oct 31 04:36:02.265935 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:36:02.265912 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-5n2gs"] Oct 31 04:36:02.267040 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:36:02.267016 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd057ba0f_e65d_44de_949f_bc94f3c20550.slice/crio-86fa4999a692d1530210154a293aef2a9cb10538982e0f09d60e7978b81948a5 WatchSource:0}: Error finding container 86fa4999a692d1530210154a293aef2a9cb10538982e0f09d60e7978b81948a5: Status 404 returned error can't find the container with id 86fa4999a692d1530210154a293aef2a9cb10538982e0f09d60e7978b81948a5 Oct 31 04:36:02.270183 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:36:02.270153 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:36:02.270250 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:36:02.270218 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:36:02.270290 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:36:02.270247 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:36:03.236360 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:36:03.235285 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-5n2gs" event={"ID":"d057ba0f-e65d-44de-949f-bc94f3c20550","Type":"ContainerStarted","Data":"6320e141db89af9e001de1e607995acab227e72490f754e8f73b16fddce9f78d"} Oct 31 04:36:03.236360 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:36:03.235372 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-5n2gs" event={"ID":"d057ba0f-e65d-44de-949f-bc94f3c20550","Type":"ContainerStarted","Data":"86fa4999a692d1530210154a293aef2a9cb10538982e0f09d60e7978b81948a5"} Oct 31 04:36:03.262361 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:36:03.255823 4376 patch_prober.go:29] interesting pod/e2e-shared-a-w1-77a153-openshift-default-7778f45d97-pzlhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body= Oct 31 04:36:03.262361 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:36:03.255886 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-77a153-openshift-default-7778f45d97-pzlhr" podUID="faaa68df-a884-400f-be9a-84bbb75ad520" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" Oct 31 04:36:03.319352 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:36:03.312510 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-5n2gs" podStartSLOduration=2.31249056 podStartE2EDuration="2.31249056s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:36:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:36:03.31216021 +0000 UTC m=+1911.026494849" watchObservedRunningTime="2026-10-31 04:36:03.31249056 +0000 UTC m=+1911.026825177" Oct 31 04:36:03.906918 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:36:03.906834 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-5n2gs" Oct 31 04:36:03.908892 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:36:03.908864 4376 patch_prober.go:29] interesting pod/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-5n2gs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body= Oct 31 04:36:03.909018 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:36:03.908911 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-5n2gs" podUID="d057ba0f-e65d-44de-949f-bc94f3c20550" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" Oct 31 04:36:04.288447 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:36:04.286041 4376 patch_prober.go:29] interesting pod/e2e-shared-a-w1-77a153-openshift-default-7778f45d97-pzlhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body= Oct 31 04:36:04.288447 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:36:04.286123 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-77a153-openshift-default-7778f45d97-pzlhr" podUID="faaa68df-a884-400f-be9a-84bbb75ad520" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" Oct 31 04:36:04.906591 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:36:04.906559 4376 patch_prober.go:29] interesting pod/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-5n2gs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body= Oct 31 04:36:04.906961 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:36:04.906613 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-5n2gs" podUID="d057ba0f-e65d-44de-949f-bc94f3c20550" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" Oct 31 04:36:05.255824 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:36:05.255732 4376 patch_prober.go:29] interesting pod/e2e-shared-a-w1-77a153-openshift-default-7778f45d97-pzlhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body= Oct 31 04:36:05.256084 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:36:05.256053 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-77a153-openshift-default-7778f45d97-pzlhr" podUID="faaa68df-a884-400f-be9a-84bbb75ad520" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" Oct 31 04:36:05.906772 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:36:05.906734 4376 patch_prober.go:29] interesting pod/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-5n2gs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body= Oct 31 04:36:05.907234 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:36:05.906797 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-5n2gs" podUID="d057ba0f-e65d-44de-949f-bc94f3c20550" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" Oct 31 04:36:06.410315 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:36:06.410284 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-77a153-openshift-default-7778f45d97-pzlhr" Oct 31 04:36:06.411820 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:36:06.411797 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-77a153-openshift-default-7778f45d97-pzlhr" Oct 31 04:36:06.412265 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:36:06.412250 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-77a153-openshift-default-7778f45d97-pzlhr" Oct 31 04:36:06.906372 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:36:06.906309 4376 patch_prober.go:29] interesting pod/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-5n2gs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body= Oct 31 04:36:06.906537 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:36:06.906388 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-5n2gs" podUID="d057ba0f-e65d-44de-949f-bc94f3c20550" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" Oct 31 04:36:07.907874 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:36:07.907837 4376 patch_prober.go:29] interesting pod/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-5n2gs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body= Oct 31 04:36:07.908540 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:36:07.908505 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-5n2gs" podUID="d057ba0f-e65d-44de-949f-bc94f3c20550" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" Oct 31 04:36:08.906558 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:36:08.906526 4376 patch_prober.go:29] interesting pod/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-5n2gs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body= Oct 31 04:36:08.906710 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:36:08.906576 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-5n2gs" podUID="d057ba0f-e65d-44de-949f-bc94f3c20550" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" Oct 31 04:36:09.911022 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:36:09.910995 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-5n2gs" Oct 31 04:36:09.911456 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:36:09.911444 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-5n2gs" Oct 31 04:36:09.911994 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:36:09.911977 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-5n2gs" Oct 31 04:36:11.004615 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:36:11.004578 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 04:36:16.089956 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:36:16.089921 4376 scope.go:122] "RemoveContainer" containerID="d0bd93308eb9d0b0b2ec78c726416696b000c760fadfca2aae031a087702be20" Oct 31 04:36:16.100042 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:36:16.100019 4376 scope.go:122] "RemoveContainer" containerID="95bf74fe482191a98de52d27b773dc593cf08baab1d4d40517fcc70694fb57e8" Oct 31 04:36:17.997527 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:36:17.997488 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-7229f2-7b7cfc846f-4h24n"] Oct 31 04:36:22.998013 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:36:22.997965 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-7229f2-7b7cfc846f-4h24n" podUID="6cff2bce-3e01-4f33-a36d-eda47fbeae3d" containerName="payload-processing" containerID="cri-o://9d9524a952d6f91d31d8e450b527dc50509105d26de44f19986807644d4adbdf" gracePeriod=25 Oct 31 04:36:23.283075 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:36:23.283048 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-7229f2-7b7cfc846f-4h24n" Oct 31 04:36:23.413364 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:36:23.413310 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6cff2bce-3e01-4f33-a36d-eda47fbeae3d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6cff2bce-3e01-4f33-a36d-eda47fbeae3d-plugins-config-volume\") pod \"6cff2bce-3e01-4f33-a36d-eda47fbeae3d\" (UID: \"6cff2bce-3e01-4f33-a36d-eda47fbeae3d\") " Oct 31 04:36:23.413603 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:36:23.413582 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6cff2bce-3e01-4f33-a36d-eda47fbeae3d-kube-api-access-thgnp\" (UniqueName: \"kubernetes.io/projected/6cff2bce-3e01-4f33-a36d-eda47fbeae3d-kube-api-access-thgnp\") pod \"6cff2bce-3e01-4f33-a36d-eda47fbeae3d\" (UID: \"6cff2bce-3e01-4f33-a36d-eda47fbeae3d\") " Oct 31 04:36:23.413761 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:36:23.413730 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6cff2bce-3e01-4f33-a36d-eda47fbeae3d-plugins-config-volume" pod "6cff2bce-3e01-4f33-a36d-eda47fbeae3d" (UID: "6cff2bce-3e01-4f33-a36d-eda47fbeae3d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:36:23.414254 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:36:23.414228 4376 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6cff2bce-3e01-4f33-a36d-eda47fbeae3d-plugins-config-volume\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:36:23.420480 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:36:23.420452 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cff2bce-3e01-4f33-a36d-eda47fbeae3d-kube-api-access-thgnp" pod "6cff2bce-3e01-4f33-a36d-eda47fbeae3d" (UID: "6cff2bce-3e01-4f33-a36d-eda47fbeae3d"). InnerVolumeSpecName "kube-api-access-thgnp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:36:23.470313 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:36:23.470282 4376 generic.go:356] "Generic (PLEG): container finished" podID="6cff2bce-3e01-4f33-a36d-eda47fbeae3d" containerID="9d9524a952d6f91d31d8e450b527dc50509105d26de44f19986807644d4adbdf" exitCode=0 Oct 31 04:36:23.470477 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:36:23.470356 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-7229f2-7b7cfc846f-4h24n" Oct 31 04:36:23.470477 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:36:23.470366 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-7229f2-7b7cfc846f-4h24n" event={"ID":"6cff2bce-3e01-4f33-a36d-eda47fbeae3d","Type":"ContainerDied","Data":"9d9524a952d6f91d31d8e450b527dc50509105d26de44f19986807644d4adbdf"} Oct 31 04:36:23.470477 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:36:23.470419 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-7229f2-7b7cfc846f-4h24n" event={"ID":"6cff2bce-3e01-4f33-a36d-eda47fbeae3d","Type":"ContainerDied","Data":"01e39bdf2d84cfa698efa48ee744332a662b6d4da62822788b09b5ed662ec19b"} Oct 31 04:36:23.470477 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:36:23.470441 4376 scope.go:122] "RemoveContainer" containerID="9d9524a952d6f91d31d8e450b527dc50509105d26de44f19986807644d4adbdf" Oct 31 04:36:23.483673 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:36:23.483656 4376 scope.go:122] "RemoveContainer" containerID="9d9524a952d6f91d31d8e450b527dc50509105d26de44f19986807644d4adbdf" Oct 31 04:36:23.483961 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:36:23.483940 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d9524a952d6f91d31d8e450b527dc50509105d26de44f19986807644d4adbdf\": container with ID starting with 9d9524a952d6f91d31d8e450b527dc50509105d26de44f19986807644d4adbdf not found: ID does not exist" containerID="9d9524a952d6f91d31d8e450b527dc50509105d26de44f19986807644d4adbdf" Oct 31 04:36:23.484016 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:36:23.483970 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d9524a952d6f91d31d8e450b527dc50509105d26de44f19986807644d4adbdf"} err="failed to get container status \"9d9524a952d6f91d31d8e450b527dc50509105d26de44f19986807644d4adbdf\": rpc error: code = NotFound desc = could not find container \"9d9524a952d6f91d31d8e450b527dc50509105d26de44f19986807644d4adbdf\": container with ID starting with 9d9524a952d6f91d31d8e450b527dc50509105d26de44f19986807644d4adbdf not found: ID does not exist" Oct 31 04:36:23.516430 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:36:23.516367 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-thgnp\" (UniqueName: \"kubernetes.io/projected/6cff2bce-3e01-4f33-a36d-eda47fbeae3d-kube-api-access-thgnp\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:36:23.525738 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:36:23.525715 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-7229f2-7b7cfc846f-4h24n"] Oct 31 04:36:23.532454 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:36:23.532432 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-7229f2-7b7cfc846f-4h24n"] Oct 31 04:36:24.444923 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:36:24.444893 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6cff2bce-3e01-4f33-a36d-eda47fbeae3d" path="/var/lib/kubelet/pods/6cff2bce-3e01-4f33-a36d-eda47fbeae3d/volumes" Oct 31 04:36:25.776042 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:36:25.776000 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 04:36:41.003758 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:36:41.003712 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 04:36:41.203561 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:36:41.203525 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-7229f2-openshift-default-7b5c69df47-7tnqr"] Oct 31 04:36:41.203873 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:36:41.203822 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-7229f2-openshift-default-7b5c69df47-7tnqr" podUID="9e23ed56-ce2f-47b9-bd20-d22f8aa8a3ae" containerName="istio-proxy" containerID="cri-o://151122b13fc1797023d86e6a7b30641f76fb43af6fee9e427c15ccbbaf04ac61" gracePeriod=30 Oct 31 04:36:46.482982 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:36:46.482960 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-7229f2-openshift-default-7b5c69df47-7tnqr" Oct 31 04:36:46.575794 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:36:46.575761 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e23ed56-ce2f-47b9-bd20-d22f8aa8a3ae-kube-api-access-xztnh\" (UniqueName: \"kubernetes.io/projected/9e23ed56-ce2f-47b9-bd20-d22f8aa8a3ae-kube-api-access-xztnh\") pod \"9e23ed56-ce2f-47b9-bd20-d22f8aa8a3ae\" (UID: \"9e23ed56-ce2f-47b9-bd20-d22f8aa8a3ae\") " Oct 31 04:36:46.576019 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:36:46.575830 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e23ed56-ce2f-47b9-bd20-d22f8aa8a3ae-istio-token\" (UniqueName: \"kubernetes.io/projected/9e23ed56-ce2f-47b9-bd20-d22f8aa8a3ae-istio-token\") pod \"9e23ed56-ce2f-47b9-bd20-d22f8aa8a3ae\" (UID: \"9e23ed56-ce2f-47b9-bd20-d22f8aa8a3ae\") " Oct 31 04:36:46.576019 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:36:46.575960 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e23ed56-ce2f-47b9-bd20-d22f8aa8a3ae-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e23ed56-ce2f-47b9-bd20-d22f8aa8a3ae-credential-socket\") pod \"9e23ed56-ce2f-47b9-bd20-d22f8aa8a3ae\" (UID: \"9e23ed56-ce2f-47b9-bd20-d22f8aa8a3ae\") " Oct 31 04:36:46.576320 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:36:46.576021 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e23ed56-ce2f-47b9-bd20-d22f8aa8a3ae-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e23ed56-ce2f-47b9-bd20-d22f8aa8a3ae-workload-certs\") pod \"9e23ed56-ce2f-47b9-bd20-d22f8aa8a3ae\" (UID: \"9e23ed56-ce2f-47b9-bd20-d22f8aa8a3ae\") " Oct 31 04:36:46.576320 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:36:46.576173 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e23ed56-ce2f-47b9-bd20-d22f8aa8a3ae-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e23ed56-ce2f-47b9-bd20-d22f8aa8a3ae-istio-data\") pod \"9e23ed56-ce2f-47b9-bd20-d22f8aa8a3ae\" (UID: \"9e23ed56-ce2f-47b9-bd20-d22f8aa8a3ae\") " Oct 31 04:36:46.576725 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:36:46.576601 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e23ed56-ce2f-47b9-bd20-d22f8aa8a3ae-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e23ed56-ce2f-47b9-bd20-d22f8aa8a3ae-workload-socket\") pod \"9e23ed56-ce2f-47b9-bd20-d22f8aa8a3ae\" (UID: \"9e23ed56-ce2f-47b9-bd20-d22f8aa8a3ae\") " Oct 31 04:36:46.576725 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:36:46.576647 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e23ed56-ce2f-47b9-bd20-d22f8aa8a3ae-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e23ed56-ce2f-47b9-bd20-d22f8aa8a3ae-istiod-ca-cert\") pod \"9e23ed56-ce2f-47b9-bd20-d22f8aa8a3ae\" (UID: \"9e23ed56-ce2f-47b9-bd20-d22f8aa8a3ae\") " Oct 31 04:36:46.576905 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:36:46.576734 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9e23ed56-ce2f-47b9-bd20-d22f8aa8a3ae-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e23ed56-ce2f-47b9-bd20-d22f8aa8a3ae-istio-podinfo\") pod \"9e23ed56-ce2f-47b9-bd20-d22f8aa8a3ae\" (UID: \"9e23ed56-ce2f-47b9-bd20-d22f8aa8a3ae\") " Oct 31 04:36:46.576905 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:36:46.576791 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e23ed56-ce2f-47b9-bd20-d22f8aa8a3ae-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e23ed56-ce2f-47b9-bd20-d22f8aa8a3ae-istio-envoy\") pod \"9e23ed56-ce2f-47b9-bd20-d22f8aa8a3ae\" (UID: \"9e23ed56-ce2f-47b9-bd20-d22f8aa8a3ae\") " Oct 31 04:36:46.578150 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:36:46.578121 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e23ed56-ce2f-47b9-bd20-d22f8aa8a3ae-credential-socket" pod "9e23ed56-ce2f-47b9-bd20-d22f8aa8a3ae" (UID: "9e23ed56-ce2f-47b9-bd20-d22f8aa8a3ae"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:36:46.578980 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:36:46.578952 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e23ed56-ce2f-47b9-bd20-d22f8aa8a3ae-workload-certs" pod "9e23ed56-ce2f-47b9-bd20-d22f8aa8a3ae" (UID: "9e23ed56-ce2f-47b9-bd20-d22f8aa8a3ae"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:36:46.579083 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:36:46.579030 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e23ed56-ce2f-47b9-bd20-d22f8aa8a3ae-workload-socket" pod "9e23ed56-ce2f-47b9-bd20-d22f8aa8a3ae" (UID: "9e23ed56-ce2f-47b9-bd20-d22f8aa8a3ae"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:36:46.579792 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:36:46.579754 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e23ed56-ce2f-47b9-bd20-d22f8aa8a3ae-kube-api-access-xztnh" pod "9e23ed56-ce2f-47b9-bd20-d22f8aa8a3ae" (UID: "9e23ed56-ce2f-47b9-bd20-d22f8aa8a3ae"). InnerVolumeSpecName "kube-api-access-xztnh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:36:46.580217 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:36:46.580188 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e23ed56-ce2f-47b9-bd20-d22f8aa8a3ae-istio-token" pod "9e23ed56-ce2f-47b9-bd20-d22f8aa8a3ae" (UID: "9e23ed56-ce2f-47b9-bd20-d22f8aa8a3ae"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:36:46.580396 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:36:46.580381 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e23ed56-ce2f-47b9-bd20-d22f8aa8a3ae-istio-envoy" pod "9e23ed56-ce2f-47b9-bd20-d22f8aa8a3ae" (UID: "9e23ed56-ce2f-47b9-bd20-d22f8aa8a3ae"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:36:46.581433 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:36:46.581407 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e23ed56-ce2f-47b9-bd20-d22f8aa8a3ae-istiod-ca-cert" pod "9e23ed56-ce2f-47b9-bd20-d22f8aa8a3ae" (UID: "9e23ed56-ce2f-47b9-bd20-d22f8aa8a3ae"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:36:46.581541 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:36:46.581530 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e23ed56-ce2f-47b9-bd20-d22f8aa8a3ae-istio-data" pod "9e23ed56-ce2f-47b9-bd20-d22f8aa8a3ae" (UID: "9e23ed56-ce2f-47b9-bd20-d22f8aa8a3ae"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:36:46.582006 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:36:46.581980 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9e23ed56-ce2f-47b9-bd20-d22f8aa8a3ae-istio-podinfo" pod "9e23ed56-ce2f-47b9-bd20-d22f8aa8a3ae" (UID: "9e23ed56-ce2f-47b9-bd20-d22f8aa8a3ae"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 04:36:46.678937 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:36:46.678904 4376 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e23ed56-ce2f-47b9-bd20-d22f8aa8a3ae-istio-data\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:36:46.678937 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:36:46.678934 4376 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e23ed56-ce2f-47b9-bd20-d22f8aa8a3ae-workload-socket\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:36:46.679139 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:36:46.678946 4376 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e23ed56-ce2f-47b9-bd20-d22f8aa8a3ae-istiod-ca-cert\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:36:46.679139 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:36:46.678958 4376 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e23ed56-ce2f-47b9-bd20-d22f8aa8a3ae-istio-podinfo\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:36:46.679139 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:36:46.678968 4376 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e23ed56-ce2f-47b9-bd20-d22f8aa8a3ae-istio-envoy\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:36:46.679139 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:36:46.678981 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xztnh\" (UniqueName: \"kubernetes.io/projected/9e23ed56-ce2f-47b9-bd20-d22f8aa8a3ae-kube-api-access-xztnh\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:36:46.679139 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:36:46.678992 4376 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e23ed56-ce2f-47b9-bd20-d22f8aa8a3ae-istio-token\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:36:46.679139 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:36:46.679002 4376 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e23ed56-ce2f-47b9-bd20-d22f8aa8a3ae-credential-socket\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:36:46.679139 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:36:46.679013 4376 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e23ed56-ce2f-47b9-bd20-d22f8aa8a3ae-workload-certs\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:36:46.703161 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:36:46.703106 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-7229f2-openshift-default-7b5c69df47-7tnqr" Oct 31 04:36:46.703324 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:36:46.703040 4376 generic.go:356] "Generic (PLEG): container finished" podID="9e23ed56-ce2f-47b9-bd20-d22f8aa8a3ae" containerID="151122b13fc1797023d86e6a7b30641f76fb43af6fee9e427c15ccbbaf04ac61" exitCode=0 Oct 31 04:36:46.703324 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:36:46.703226 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-7229f2-openshift-default-7b5c69df47-7tnqr" event={"ID":"9e23ed56-ce2f-47b9-bd20-d22f8aa8a3ae","Type":"ContainerDied","Data":"151122b13fc1797023d86e6a7b30641f76fb43af6fee9e427c15ccbbaf04ac61"} Oct 31 04:36:46.703324 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:36:46.703277 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-7229f2-openshift-default-7b5c69df47-7tnqr" event={"ID":"9e23ed56-ce2f-47b9-bd20-d22f8aa8a3ae","Type":"ContainerDied","Data":"88d17da629a8a0b15d5bdd45ee6fb8d65c363f1742837db4be149c460ba2b2ae"} Oct 31 04:36:46.703324 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:36:46.703306 4376 scope.go:122] "RemoveContainer" containerID="151122b13fc1797023d86e6a7b30641f76fb43af6fee9e427c15ccbbaf04ac61" Oct 31 04:36:46.715608 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:36:46.715588 4376 scope.go:122] "RemoveContainer" containerID="49d599099893cf360611bee983fceaf09d3b863db4c87b8c129e8a2c6a2b3058" Oct 31 04:36:46.724674 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:36:46.724656 4376 scope.go:122] "RemoveContainer" containerID="151122b13fc1797023d86e6a7b30641f76fb43af6fee9e427c15ccbbaf04ac61" Oct 31 04:36:46.724962 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:36:46.724942 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"151122b13fc1797023d86e6a7b30641f76fb43af6fee9e427c15ccbbaf04ac61\": container with ID starting with 151122b13fc1797023d86e6a7b30641f76fb43af6fee9e427c15ccbbaf04ac61 not found: ID does not exist" containerID="151122b13fc1797023d86e6a7b30641f76fb43af6fee9e427c15ccbbaf04ac61" Oct 31 04:36:46.725029 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:36:46.724972 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"151122b13fc1797023d86e6a7b30641f76fb43af6fee9e427c15ccbbaf04ac61"} err="failed to get container status \"151122b13fc1797023d86e6a7b30641f76fb43af6fee9e427c15ccbbaf04ac61\": rpc error: code = NotFound desc = could not find container \"151122b13fc1797023d86e6a7b30641f76fb43af6fee9e427c15ccbbaf04ac61\": container with ID starting with 151122b13fc1797023d86e6a7b30641f76fb43af6fee9e427c15ccbbaf04ac61 not found: ID does not exist" Oct 31 04:36:46.725029 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:36:46.724989 4376 scope.go:122] "RemoveContainer" containerID="49d599099893cf360611bee983fceaf09d3b863db4c87b8c129e8a2c6a2b3058" Oct 31 04:36:46.725265 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:36:46.725247 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49d599099893cf360611bee983fceaf09d3b863db4c87b8c129e8a2c6a2b3058\": container with ID starting with 49d599099893cf360611bee983fceaf09d3b863db4c87b8c129e8a2c6a2b3058 not found: ID does not exist" containerID="49d599099893cf360611bee983fceaf09d3b863db4c87b8c129e8a2c6a2b3058" Oct 31 04:36:46.725317 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:36:46.725271 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49d599099893cf360611bee983fceaf09d3b863db4c87b8c129e8a2c6a2b3058"} err="failed to get container status \"49d599099893cf360611bee983fceaf09d3b863db4c87b8c129e8a2c6a2b3058\": rpc error: code = NotFound desc = could not find container \"49d599099893cf360611bee983fceaf09d3b863db4c87b8c129e8a2c6a2b3058\": container with ID starting with 49d599099893cf360611bee983fceaf09d3b863db4c87b8c129e8a2c6a2b3058 not found: ID does not exist" Oct 31 04:36:46.757932 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:36:46.757910 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-7229f2-openshift-default-7b5c69df47-7tnqr"] Oct 31 04:36:46.763551 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:36:46.763529 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-7229f2-openshift-default-7b5c69df47-7tnqr"] Oct 31 04:36:48.515619 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:36:48.515432 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e23ed56-ce2f-47b9-bd20-d22f8aa8a3ae" path="/var/lib/kubelet/pods/9e23ed56-ce2f-47b9-bd20-d22f8aa8a3ae/volumes" Oct 31 04:37:11.003516 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:37:11.003477 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 04:37:18.018282 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:37:18.018252 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 04:37:36.921491 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:37:36.921456 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-d5gfx"] Oct 31 04:37:36.921943 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:37:36.921697 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-d5gfx" podUID="4d2c5af6-838a-4bb6-bec8-8803bbb03c22" containerName="istio-proxy" containerID="cri-o://4a2b8b2ebb49be8a92bbb62db2239c1c63aed46c083c7407995c1a3d273240e9" gracePeriod=30 Oct 31 04:37:36.934967 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:37:36.934940 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-dw4vw"] Oct 31 04:37:37.126461 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:37:37.126412 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-dw4vw" podUID="74d939ce-44dd-4d3c-b51e-0ba8fa7be377" containerName="istio-proxy" containerID="cri-o://43dda6c731a7c063331c51c61fd7449e6d4a7bbdadee291a0886056d22c384fd" gracePeriod=30 Oct 31 04:37:41.000312 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:37:41.000276 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 04:37:42.166543 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:37:42.166514 4376 generic.go:356] "Generic (PLEG): container finished" podID="4d2c5af6-838a-4bb6-bec8-8803bbb03c22" containerID="4a2b8b2ebb49be8a92bbb62db2239c1c63aed46c083c7407995c1a3d273240e9" exitCode=0 Oct 31 04:37:42.166888 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:37:42.166659 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-d5gfx" event={"ID":"4d2c5af6-838a-4bb6-bec8-8803bbb03c22","Type":"ContainerDied","Data":"4a2b8b2ebb49be8a92bbb62db2239c1c63aed46c083c7407995c1a3d273240e9"} Oct 31 04:37:42.262213 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:37:42.262182 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-d5gfx" Oct 31 04:37:42.329128 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:37:42.329093 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4d2c5af6-838a-4bb6-bec8-8803bbb03c22-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d2c5af6-838a-4bb6-bec8-8803bbb03c22-istio-podinfo\") pod \"4d2c5af6-838a-4bb6-bec8-8803bbb03c22\" (UID: \"4d2c5af6-838a-4bb6-bec8-8803bbb03c22\") " Oct 31 04:37:42.329342 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:37:42.329310 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d2c5af6-838a-4bb6-bec8-8803bbb03c22-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d2c5af6-838a-4bb6-bec8-8803bbb03c22-credential-socket\") pod \"4d2c5af6-838a-4bb6-bec8-8803bbb03c22\" (UID: \"4d2c5af6-838a-4bb6-bec8-8803bbb03c22\") " Oct 31 04:37:42.329439 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:37:42.329404 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d2c5af6-838a-4bb6-bec8-8803bbb03c22-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d2c5af6-838a-4bb6-bec8-8803bbb03c22-istio-envoy\") pod \"4d2c5af6-838a-4bb6-bec8-8803bbb03c22\" (UID: \"4d2c5af6-838a-4bb6-bec8-8803bbb03c22\") " Oct 31 04:37:42.329499 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:37:42.329474 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d2c5af6-838a-4bb6-bec8-8803bbb03c22-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d2c5af6-838a-4bb6-bec8-8803bbb03c22-workload-socket\") pod \"4d2c5af6-838a-4bb6-bec8-8803bbb03c22\" (UID: \"4d2c5af6-838a-4bb6-bec8-8803bbb03c22\") " Oct 31 04:37:42.329579 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:37:42.329564 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d2c5af6-838a-4bb6-bec8-8803bbb03c22-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d2c5af6-838a-4bb6-bec8-8803bbb03c22-workload-certs\") pod \"4d2c5af6-838a-4bb6-bec8-8803bbb03c22\" (UID: \"4d2c5af6-838a-4bb6-bec8-8803bbb03c22\") " Oct 31 04:37:42.329651 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:37:42.329636 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d2c5af6-838a-4bb6-bec8-8803bbb03c22-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d2c5af6-838a-4bb6-bec8-8803bbb03c22-istio-data\") pod \"4d2c5af6-838a-4bb6-bec8-8803bbb03c22\" (UID: \"4d2c5af6-838a-4bb6-bec8-8803bbb03c22\") " Oct 31 04:37:42.329741 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:37:42.329722 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d2c5af6-838a-4bb6-bec8-8803bbb03c22-kube-api-access-m4dtb\" (UniqueName: \"kubernetes.io/projected/4d2c5af6-838a-4bb6-bec8-8803bbb03c22-kube-api-access-m4dtb\") pod \"4d2c5af6-838a-4bb6-bec8-8803bbb03c22\" (UID: \"4d2c5af6-838a-4bb6-bec8-8803bbb03c22\") " Oct 31 04:37:42.329810 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:37:42.329795 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d2c5af6-838a-4bb6-bec8-8803bbb03c22-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d2c5af6-838a-4bb6-bec8-8803bbb03c22-istiod-ca-cert\") pod \"4d2c5af6-838a-4bb6-bec8-8803bbb03c22\" (UID: \"4d2c5af6-838a-4bb6-bec8-8803bbb03c22\") " Oct 31 04:37:42.330162 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:37:42.329870 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d2c5af6-838a-4bb6-bec8-8803bbb03c22-istio-token\" (UniqueName: \"kubernetes.io/projected/4d2c5af6-838a-4bb6-bec8-8803bbb03c22-istio-token\") pod \"4d2c5af6-838a-4bb6-bec8-8803bbb03c22\" (UID: \"4d2c5af6-838a-4bb6-bec8-8803bbb03c22\") " Oct 31 04:37:42.330280 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:37:42.330192 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d2c5af6-838a-4bb6-bec8-8803bbb03c22-workload-certs" pod "4d2c5af6-838a-4bb6-bec8-8803bbb03c22" (UID: "4d2c5af6-838a-4bb6-bec8-8803bbb03c22"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:37:42.330434 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:37:42.330412 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d2c5af6-838a-4bb6-bec8-8803bbb03c22-credential-socket" pod "4d2c5af6-838a-4bb6-bec8-8803bbb03c22" (UID: "4d2c5af6-838a-4bb6-bec8-8803bbb03c22"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:37:42.330955 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:37:42.330929 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d2c5af6-838a-4bb6-bec8-8803bbb03c22-workload-socket" pod "4d2c5af6-838a-4bb6-bec8-8803bbb03c22" (UID: "4d2c5af6-838a-4bb6-bec8-8803bbb03c22"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:37:42.331560 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:37:42.331536 4376 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d2c5af6-838a-4bb6-bec8-8803bbb03c22-credential-socket\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:37:42.331668 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:37:42.331565 4376 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d2c5af6-838a-4bb6-bec8-8803bbb03c22-workload-socket\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:37:42.331668 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:37:42.331580 4376 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d2c5af6-838a-4bb6-bec8-8803bbb03c22-workload-certs\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:37:42.332164 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:37:42.332131 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d2c5af6-838a-4bb6-bec8-8803bbb03c22-istiod-ca-cert" pod "4d2c5af6-838a-4bb6-bec8-8803bbb03c22" (UID: "4d2c5af6-838a-4bb6-bec8-8803bbb03c22"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:37:42.332347 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:37:42.332262 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d2c5af6-838a-4bb6-bec8-8803bbb03c22-istio-data" pod "4d2c5af6-838a-4bb6-bec8-8803bbb03c22" (UID: "4d2c5af6-838a-4bb6-bec8-8803bbb03c22"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:37:42.334079 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:37:42.334044 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d2c5af6-838a-4bb6-bec8-8803bbb03c22-istio-token" pod "4d2c5af6-838a-4bb6-bec8-8803bbb03c22" (UID: "4d2c5af6-838a-4bb6-bec8-8803bbb03c22"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:37:42.335615 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:37:42.335577 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d2c5af6-838a-4bb6-bec8-8803bbb03c22-kube-api-access-m4dtb" pod "4d2c5af6-838a-4bb6-bec8-8803bbb03c22" (UID: "4d2c5af6-838a-4bb6-bec8-8803bbb03c22"). InnerVolumeSpecName "kube-api-access-m4dtb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:37:42.336743 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:37:42.336718 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d2c5af6-838a-4bb6-bec8-8803bbb03c22-istio-envoy" pod "4d2c5af6-838a-4bb6-bec8-8803bbb03c22" (UID: "4d2c5af6-838a-4bb6-bec8-8803bbb03c22"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:37:42.338148 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:37:42.338118 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4d2c5af6-838a-4bb6-bec8-8803bbb03c22-istio-podinfo" pod "4d2c5af6-838a-4bb6-bec8-8803bbb03c22" (UID: "4d2c5af6-838a-4bb6-bec8-8803bbb03c22"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 04:37:42.392296 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:37:42.392262 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-dw4vw" Oct 31 04:37:42.434357 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:37:42.434183 4376 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d2c5af6-838a-4bb6-bec8-8803bbb03c22-istio-podinfo\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:37:42.434357 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:37:42.434217 4376 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d2c5af6-838a-4bb6-bec8-8803bbb03c22-istio-envoy\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:37:42.434357 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:37:42.434231 4376 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d2c5af6-838a-4bb6-bec8-8803bbb03c22-istio-data\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:37:42.434357 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:37:42.434244 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4dtb\" (UniqueName: \"kubernetes.io/projected/4d2c5af6-838a-4bb6-bec8-8803bbb03c22-kube-api-access-m4dtb\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:37:42.434357 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:37:42.434259 4376 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d2c5af6-838a-4bb6-bec8-8803bbb03c22-istiod-ca-cert\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:37:42.434357 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:37:42.434272 4376 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d2c5af6-838a-4bb6-bec8-8803bbb03c22-istio-token\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:37:42.535568 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:37:42.535534 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74d939ce-44dd-4d3c-b51e-0ba8fa7be377-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74d939ce-44dd-4d3c-b51e-0ba8fa7be377-credential-socket\") pod \"74d939ce-44dd-4d3c-b51e-0ba8fa7be377\" (UID: \"74d939ce-44dd-4d3c-b51e-0ba8fa7be377\") " Oct 31 04:37:42.535740 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:37:42.535691 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/74d939ce-44dd-4d3c-b51e-0ba8fa7be377-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74d939ce-44dd-4d3c-b51e-0ba8fa7be377-istio-podinfo\") pod \"74d939ce-44dd-4d3c-b51e-0ba8fa7be377\" (UID: \"74d939ce-44dd-4d3c-b51e-0ba8fa7be377\") " Oct 31 04:37:42.535791 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:37:42.535739 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74d939ce-44dd-4d3c-b51e-0ba8fa7be377-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74d939ce-44dd-4d3c-b51e-0ba8fa7be377-istio-envoy\") pod \"74d939ce-44dd-4d3c-b51e-0ba8fa7be377\" (UID: \"74d939ce-44dd-4d3c-b51e-0ba8fa7be377\") " Oct 31 04:37:42.535860 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:37:42.535831 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74d939ce-44dd-4d3c-b51e-0ba8fa7be377-credential-socket" pod "74d939ce-44dd-4d3c-b51e-0ba8fa7be377" (UID: "74d939ce-44dd-4d3c-b51e-0ba8fa7be377"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:37:42.535915 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:37:42.535896 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74d939ce-44dd-4d3c-b51e-0ba8fa7be377-kube-api-access-767bs\" (UniqueName: \"kubernetes.io/projected/74d939ce-44dd-4d3c-b51e-0ba8fa7be377-kube-api-access-767bs\") pod \"74d939ce-44dd-4d3c-b51e-0ba8fa7be377\" (UID: \"74d939ce-44dd-4d3c-b51e-0ba8fa7be377\") " Oct 31 04:37:42.535973 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:37:42.535920 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74d939ce-44dd-4d3c-b51e-0ba8fa7be377-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74d939ce-44dd-4d3c-b51e-0ba8fa7be377-workload-socket\") pod \"74d939ce-44dd-4d3c-b51e-0ba8fa7be377\" (UID: \"74d939ce-44dd-4d3c-b51e-0ba8fa7be377\") " Oct 31 04:37:42.536031 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:37:42.535972 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74d939ce-44dd-4d3c-b51e-0ba8fa7be377-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74d939ce-44dd-4d3c-b51e-0ba8fa7be377-istio-data\") pod \"74d939ce-44dd-4d3c-b51e-0ba8fa7be377\" (UID: \"74d939ce-44dd-4d3c-b51e-0ba8fa7be377\") " Oct 31 04:37:42.536031 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:37:42.535997 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74d939ce-44dd-4d3c-b51e-0ba8fa7be377-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74d939ce-44dd-4d3c-b51e-0ba8fa7be377-istiod-ca-cert\") pod \"74d939ce-44dd-4d3c-b51e-0ba8fa7be377\" (UID: \"74d939ce-44dd-4d3c-b51e-0ba8fa7be377\") " Oct 31 04:37:42.536146 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:37:42.536030 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74d939ce-44dd-4d3c-b51e-0ba8fa7be377-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74d939ce-44dd-4d3c-b51e-0ba8fa7be377-workload-certs\") pod \"74d939ce-44dd-4d3c-b51e-0ba8fa7be377\" (UID: \"74d939ce-44dd-4d3c-b51e-0ba8fa7be377\") " Oct 31 04:37:42.536146 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:37:42.536080 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74d939ce-44dd-4d3c-b51e-0ba8fa7be377-istio-token\" (UniqueName: \"kubernetes.io/projected/74d939ce-44dd-4d3c-b51e-0ba8fa7be377-istio-token\") pod \"74d939ce-44dd-4d3c-b51e-0ba8fa7be377\" (UID: \"74d939ce-44dd-4d3c-b51e-0ba8fa7be377\") " Oct 31 04:37:42.536438 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:37:42.536401 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74d939ce-44dd-4d3c-b51e-0ba8fa7be377-workload-certs" pod "74d939ce-44dd-4d3c-b51e-0ba8fa7be377" (UID: "74d939ce-44dd-4d3c-b51e-0ba8fa7be377"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:37:42.536626 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:37:42.536601 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74d939ce-44dd-4d3c-b51e-0ba8fa7be377-workload-socket" pod "74d939ce-44dd-4d3c-b51e-0ba8fa7be377" (UID: "74d939ce-44dd-4d3c-b51e-0ba8fa7be377"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:37:42.536898 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:37:42.536873 4376 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74d939ce-44dd-4d3c-b51e-0ba8fa7be377-credential-socket\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:37:42.537002 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:37:42.536898 4376 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74d939ce-44dd-4d3c-b51e-0ba8fa7be377-workload-socket\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:37:42.537002 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:37:42.536915 4376 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74d939ce-44dd-4d3c-b51e-0ba8fa7be377-workload-certs\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:37:42.537204 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:37:42.537168 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74d939ce-44dd-4d3c-b51e-0ba8fa7be377-istio-data" pod "74d939ce-44dd-4d3c-b51e-0ba8fa7be377" (UID: "74d939ce-44dd-4d3c-b51e-0ba8fa7be377"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:37:42.537307 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:37:42.537287 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74d939ce-44dd-4d3c-b51e-0ba8fa7be377-istiod-ca-cert" pod "74d939ce-44dd-4d3c-b51e-0ba8fa7be377" (UID: "74d939ce-44dd-4d3c-b51e-0ba8fa7be377"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:37:42.538201 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:37:42.538158 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74d939ce-44dd-4d3c-b51e-0ba8fa7be377-istio-envoy" pod "74d939ce-44dd-4d3c-b51e-0ba8fa7be377" (UID: "74d939ce-44dd-4d3c-b51e-0ba8fa7be377"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:37:42.538318 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:37:42.538296 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/74d939ce-44dd-4d3c-b51e-0ba8fa7be377-istio-podinfo" pod "74d939ce-44dd-4d3c-b51e-0ba8fa7be377" (UID: "74d939ce-44dd-4d3c-b51e-0ba8fa7be377"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 04:37:42.538549 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:37:42.538529 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74d939ce-44dd-4d3c-b51e-0ba8fa7be377-kube-api-access-767bs" pod "74d939ce-44dd-4d3c-b51e-0ba8fa7be377" (UID: "74d939ce-44dd-4d3c-b51e-0ba8fa7be377"). InnerVolumeSpecName "kube-api-access-767bs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:37:42.538654 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:37:42.538638 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74d939ce-44dd-4d3c-b51e-0ba8fa7be377-istio-token" pod "74d939ce-44dd-4d3c-b51e-0ba8fa7be377" (UID: "74d939ce-44dd-4d3c-b51e-0ba8fa7be377"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:37:42.638344 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:37:42.638294 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-767bs\" (UniqueName: \"kubernetes.io/projected/74d939ce-44dd-4d3c-b51e-0ba8fa7be377-kube-api-access-767bs\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:37:42.638344 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:37:42.638322 4376 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74d939ce-44dd-4d3c-b51e-0ba8fa7be377-istio-data\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:37:42.638344 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:37:42.638350 4376 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74d939ce-44dd-4d3c-b51e-0ba8fa7be377-istiod-ca-cert\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:37:42.638585 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:37:42.638371 4376 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74d939ce-44dd-4d3c-b51e-0ba8fa7be377-istio-token\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:37:42.638585 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:37:42.638383 4376 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74d939ce-44dd-4d3c-b51e-0ba8fa7be377-istio-podinfo\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:37:42.638585 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:37:42.638394 4376 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74d939ce-44dd-4d3c-b51e-0ba8fa7be377-istio-envoy\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:37:43.174345 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:37:43.174309 4376 generic.go:356] "Generic (PLEG): container finished" podID="74d939ce-44dd-4d3c-b51e-0ba8fa7be377" containerID="43dda6c731a7c063331c51c61fd7449e6d4a7bbdadee291a0886056d22c384fd" exitCode=0 Oct 31 04:37:43.174779 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:37:43.174384 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-dw4vw" event={"ID":"74d939ce-44dd-4d3c-b51e-0ba8fa7be377","Type":"ContainerDied","Data":"43dda6c731a7c063331c51c61fd7449e6d4a7bbdadee291a0886056d22c384fd"} Oct 31 04:37:43.174779 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:37:43.174434 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-dw4vw" Oct 31 04:37:43.174779 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:37:43.174445 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-dw4vw" event={"ID":"74d939ce-44dd-4d3c-b51e-0ba8fa7be377","Type":"ContainerDied","Data":"b9d0f681eb3e19230ae095ccaea0bf09fe629c41a7d3a77f69e73dd3c854b44c"} Oct 31 04:37:43.174779 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:37:43.174469 4376 scope.go:122] "RemoveContainer" containerID="43dda6c731a7c063331c51c61fd7449e6d4a7bbdadee291a0886056d22c384fd" Oct 31 04:37:43.176105 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:37:43.176082 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-d5gfx" event={"ID":"4d2c5af6-838a-4bb6-bec8-8803bbb03c22","Type":"ContainerDied","Data":"c04f735da9f18ed1c55cac7e514a3d8bf6c00a3128eb25842cf67fd7629de22c"} Oct 31 04:37:43.176223 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:37:43.176167 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-d5gfx" Oct 31 04:37:43.185578 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:37:43.185565 4376 scope.go:122] "RemoveContainer" containerID="43dda6c731a7c063331c51c61fd7449e6d4a7bbdadee291a0886056d22c384fd" Oct 31 04:37:43.185834 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:37:43.185814 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43dda6c731a7c063331c51c61fd7449e6d4a7bbdadee291a0886056d22c384fd\": container with ID starting with 43dda6c731a7c063331c51c61fd7449e6d4a7bbdadee291a0886056d22c384fd not found: ID does not exist" containerID="43dda6c731a7c063331c51c61fd7449e6d4a7bbdadee291a0886056d22c384fd" Oct 31 04:37:43.185886 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:37:43.185845 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43dda6c731a7c063331c51c61fd7449e6d4a7bbdadee291a0886056d22c384fd"} err="failed to get container status \"43dda6c731a7c063331c51c61fd7449e6d4a7bbdadee291a0886056d22c384fd\": rpc error: code = NotFound desc = could not find container \"43dda6c731a7c063331c51c61fd7449e6d4a7bbdadee291a0886056d22c384fd\": container with ID starting with 43dda6c731a7c063331c51c61fd7449e6d4a7bbdadee291a0886056d22c384fd not found: ID does not exist" Oct 31 04:37:43.185886 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:37:43.185862 4376 scope.go:122] "RemoveContainer" containerID="4a2b8b2ebb49be8a92bbb62db2239c1c63aed46c083c7407995c1a3d273240e9" Oct 31 04:37:43.229435 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:37:43.229374 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-d5gfx"] Oct 31 04:37:43.242542 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:37:43.242517 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-d5gfx"] Oct 31 04:37:43.271382 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:37:43.271356 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-dw4vw"] Oct 31 04:37:43.274755 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:37:43.274735 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-dw4vw"] Oct 31 04:37:44.444922 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:37:44.444890 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d2c5af6-838a-4bb6-bec8-8803bbb03c22" path="/var/lib/kubelet/pods/4d2c5af6-838a-4bb6-bec8-8803bbb03c22/volumes" Oct 31 04:37:44.445404 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:37:44.445389 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74d939ce-44dd-4d3c-b51e-0ba8fa7be377" path="/var/lib/kubelet/pods/74d939ce-44dd-4d3c-b51e-0ba8fa7be377/volumes" Oct 31 04:37:54.537447 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:37:54.537415 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7096-a211b5"] Oct 31 04:37:54.538223 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:37:54.538197 4376 cpu_manager.go:414] "containerMap: removing container" podUID="6cff2bce-3e01-4f33-a36d-eda47fbeae3d" containerName="payload-processing" Oct 31 04:37:54.538223 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:37:54.538223 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6cff2bce-3e01-4f33-a36d-eda47fbeae3d" containerName="payload-processing" Oct 31 04:37:54.538463 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:37:54.538253 4376 cpu_manager.go:414] "containerMap: removing container" podUID="4d2c5af6-838a-4bb6-bec8-8803bbb03c22" containerName="istio-proxy" Oct 31 04:37:54.538463 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:37:54.538262 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d2c5af6-838a-4bb6-bec8-8803bbb03c22" containerName="istio-proxy" Oct 31 04:37:54.538463 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:37:54.538284 4376 cpu_manager.go:414] "containerMap: removing container" podUID="74d939ce-44dd-4d3c-b51e-0ba8fa7be377" containerName="istio-proxy" Oct 31 04:37:54.538463 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:37:54.538293 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74d939ce-44dd-4d3c-b51e-0ba8fa7be377" containerName="istio-proxy" Oct 31 04:37:54.538463 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:37:54.538303 4376 cpu_manager.go:414] "containerMap: removing container" podUID="9e23ed56-ce2f-47b9-bd20-d22f8aa8a3ae" containerName="istio-proxy" Oct 31 04:37:54.538463 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:37:54.538310 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e23ed56-ce2f-47b9-bd20-d22f8aa8a3ae" containerName="istio-proxy" Oct 31 04:37:54.538463 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:37:54.538340 4376 cpu_manager.go:414] "containerMap: removing container" podUID="9e23ed56-ce2f-47b9-bd20-d22f8aa8a3ae" containerName="istio-proxy" Oct 31 04:37:54.538463 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:37:54.538347 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e23ed56-ce2f-47b9-bd20-d22f8aa8a3ae" containerName="istio-proxy" Oct 31 04:37:54.538855 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:37:54.538539 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="6cff2bce-3e01-4f33-a36d-eda47fbeae3d" containerName="payload-processing" Oct 31 04:37:54.538855 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:37:54.538554 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d2c5af6-838a-4bb6-bec8-8803bbb03c22" containerName="istio-proxy" Oct 31 04:37:54.538855 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:37:54.538562 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="74d939ce-44dd-4d3c-b51e-0ba8fa7be377" containerName="istio-proxy" Oct 31 04:37:54.538855 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:37:54.538568 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e23ed56-ce2f-47b9-bd20-d22f8aa8a3ae" containerName="istio-proxy" Oct 31 04:37:54.538855 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:37:54.538575 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e23ed56-ce2f-47b9-bd20-d22f8aa8a3ae" containerName="istio-proxy" Oct 31 04:37:54.543643 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:37:54.543626 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7096-a211b5" Oct 31 04:37:54.559218 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:37:54.559193 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7096-a211b5"] Oct 31 04:37:54.566545 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:37:54.566506 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-prswc\"" Oct 31 04:37:54.719449 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:37:54.719416 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s67zv\" (UniqueName: \"kubernetes.io/projected/42bd7fdf-2308-4364-8685-da1dae892b00-kube-api-access-s67zv\") pod \"test-curl-7096-a211b5\" (UID: \"42bd7fdf-2308-4364-8685-da1dae892b00\") " pod="opendatahub/test-curl-7096-a211b5" Oct 31 04:37:54.820712 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:37:54.820606 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s67zv\" (UniqueName: \"kubernetes.io/projected/42bd7fdf-2308-4364-8685-da1dae892b00-kube-api-access-s67zv\") pod \"test-curl-7096-a211b5\" (UID: \"42bd7fdf-2308-4364-8685-da1dae892b00\") " pod="opendatahub/test-curl-7096-a211b5" Oct 31 04:37:54.844559 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:37:54.844526 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s67zv\" (UniqueName: \"kubernetes.io/projected/42bd7fdf-2308-4364-8685-da1dae892b00-kube-api-access-s67zv\") pod \"test-curl-7096-a211b5\" (UID: \"42bd7fdf-2308-4364-8685-da1dae892b00\") " pod="opendatahub/test-curl-7096-a211b5" Oct 31 04:37:54.856915 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:37:54.856889 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7096-a211b5" Oct 31 04:37:55.154686 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:37:55.154652 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod42bd7fdf_2308_4364_8685_da1dae892b00.slice/crio-17ceee786159d96e16d8390db03e581eb8d59fb6f2bdbf3a0cf96297942e953a WatchSource:0}: Error finding container 17ceee786159d96e16d8390db03e581eb8d59fb6f2bdbf3a0cf96297942e953a: Status 404 returned error can't find the container with id 17ceee786159d96e16d8390db03e581eb8d59fb6f2bdbf3a0cf96297942e953a Oct 31 04:37:55.155295 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:37:55.155275 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7096-a211b5"] Oct 31 04:37:55.261447 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:37:55.261407 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7096-a211b5" event={"ID":"42bd7fdf-2308-4364-8685-da1dae892b00","Type":"ContainerStarted","Data":"9d4ec0912315f46e9cdbd84705a3ce52ef74ecae0fbda8f4d8d6bb9fe95cc340"} Oct 31 04:37:55.261447 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:37:55.261453 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7096-a211b5" event={"ID":"42bd7fdf-2308-4364-8685-da1dae892b00","Type":"ContainerStarted","Data":"17ceee786159d96e16d8390db03e581eb8d59fb6f2bdbf3a0cf96297942e953a"} Oct 31 04:37:55.303463 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:37:55.303417 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7096-a211b5" podStartSLOduration=1.3034036740000001 podStartE2EDuration="1.303403674s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:37:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:37:55.301002265 +0000 UTC m=+2023.015336883" watchObservedRunningTime="2026-10-31 04:37:55.303403674 +0000 UTC m=+2023.017738291" Oct 31 04:37:56.198744 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:37:56.198710 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7096-a211b5"] Oct 31 04:37:56.202419 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:37:56.202382 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7096-a211b5"] Oct 31 04:37:56.352744 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:37:56.352716 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7096-f899c2"] Oct 31 04:37:56.353392 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:37:56.353375 4376 cpu_manager.go:414] "containerMap: removing container" podUID="42bd7fdf-2308-4364-8685-da1dae892b00" containerName="test-curl-7096-a211b5" Oct 31 04:37:56.353443 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:37:56.353392 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42bd7fdf-2308-4364-8685-da1dae892b00" containerName="test-curl-7096-a211b5" Oct 31 04:37:56.353535 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:37:56.353525 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="42bd7fdf-2308-4364-8685-da1dae892b00" containerName="test-curl-7096-a211b5" Oct 31 04:37:56.357230 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:37:56.357211 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7096-f899c2" Oct 31 04:37:56.374981 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:37:56.374956 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7096-f899c2"] Oct 31 04:37:56.469602 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:37:56.469527 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghtts\" (UniqueName: \"kubernetes.io/projected/f8fe5232-5adc-4a38-817b-d46c38589c61-kube-api-access-ghtts\") pod \"test-curl-7096-f899c2\" (UID: \"f8fe5232-5adc-4a38-817b-d46c38589c61\") " pod="opendatahub/test-curl-7096-f899c2" Oct 31 04:37:56.571166 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:37:56.571140 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghtts\" (UniqueName: \"kubernetes.io/projected/f8fe5232-5adc-4a38-817b-d46c38589c61-kube-api-access-ghtts\") pod \"test-curl-7096-f899c2\" (UID: \"f8fe5232-5adc-4a38-817b-d46c38589c61\") " pod="opendatahub/test-curl-7096-f899c2" Oct 31 04:37:56.594733 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:37:56.594699 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghtts\" (UniqueName: \"kubernetes.io/projected/f8fe5232-5adc-4a38-817b-d46c38589c61-kube-api-access-ghtts\") pod \"test-curl-7096-f899c2\" (UID: \"f8fe5232-5adc-4a38-817b-d46c38589c61\") " pod="opendatahub/test-curl-7096-f899c2" Oct 31 04:37:56.624898 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:37:56.624872 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-zs49c"] Oct 31 04:37:56.625075 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:37:56.625056 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-zs49c" podUID="d0c24766-7a33-47fa-a162-cf7593c2d41c" containerName="istio-proxy" containerID="cri-o://a62e3e3373a2a3d2897a18b4aeb2c57154ba16a90b8e42d0b99b49e5fa64cc21" gracePeriod=30 Oct 31 04:37:56.669431 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:37:56.669396 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7096-f899c2" Oct 31 04:37:56.689519 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:37:56.689495 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 04:37:56.972101 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:37:56.972069 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf8fe5232_5adc_4a38_817b_d46c38589c61.slice/crio-bb206e7432cf0dee8192c643ece6193a8f0d9ae16c926173d873b4e70f5ffc2f WatchSource:0}: Error finding container bb206e7432cf0dee8192c643ece6193a8f0d9ae16c926173d873b4e70f5ffc2f: Status 404 returned error can't find the container with id bb206e7432cf0dee8192c643ece6193a8f0d9ae16c926173d873b4e70f5ffc2f Oct 31 04:37:56.974694 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:37:56.974668 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7096-f899c2"] Oct 31 04:37:57.333665 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:37:57.333568 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7096-f899c2" event={"ID":"f8fe5232-5adc-4a38-817b-d46c38589c61","Type":"ContainerStarted","Data":"f31970fd4e6b99a488458bb8a35b511b819471010321d45bfbcac0ff419089bd"} Oct 31 04:37:57.334221 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:37:57.334187 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7096-f899c2" event={"ID":"f8fe5232-5adc-4a38-817b-d46c38589c61","Type":"ContainerStarted","Data":"bb206e7432cf0dee8192c643ece6193a8f0d9ae16c926173d873b4e70f5ffc2f"} Oct 31 04:37:57.334480 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:37:57.334442 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7096-a211b5" podUID="42bd7fdf-2308-4364-8685-da1dae892b00" containerName="test-curl-7096-a211b5" containerID="cri-o://9d4ec0912315f46e9cdbd84705a3ce52ef74ecae0fbda8f4d8d6bb9fe95cc340" gracePeriod=2 Oct 31 04:37:58.832313 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:37:58.832248 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7096-f899c2" podStartSLOduration=2.832231372 podStartE2EDuration="2.832231372s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:37:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:37:57.3891353 +0000 UTC m=+2025.103469933" watchObservedRunningTime="2026-10-31 04:37:58.832231372 +0000 UTC m=+2026.546565991" Oct 31 04:37:58.833053 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:37:58.833019 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7096-f899c2"] Oct 31 04:37:58.837237 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:37:58.837211 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7096-f899c2"] Oct 31 04:37:59.337548 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:37:59.337510 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7096-f899c2" podUID="f8fe5232-5adc-4a38-817b-d46c38589c61" containerName="test-curl-7096-f899c2" containerID="cri-o://f31970fd4e6b99a488458bb8a35b511b819471010321d45bfbcac0ff419089bd" gracePeriod=2 Oct 31 04:37:59.351995 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:37:59.351957 4376 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7096-f899c2\" is forbidden: User \"system:node:ip-10-0-134-155.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-155.ec2.internal' and this object" podUID="f8fe5232-5adc-4a38-817b-d46c38589c61" pod="opendatahub/test-curl-7096-f899c2" Oct 31 04:37:59.678836 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:37:59.678512 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7096-a211b5" Oct 31 04:37:59.693066 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:37:59.693020 4376 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7096-f899c2\" is forbidden: User \"system:node:ip-10-0-134-155.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-155.ec2.internal' and this object" podUID="f8fe5232-5adc-4a38-817b-d46c38589c61" pod="opendatahub/test-curl-7096-f899c2" Oct 31 04:37:59.770464 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:37:59.770428 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42bd7fdf-2308-4364-8685-da1dae892b00-kube-api-access-s67zv\" (UniqueName: \"kubernetes.io/projected/42bd7fdf-2308-4364-8685-da1dae892b00-kube-api-access-s67zv\") pod \"42bd7fdf-2308-4364-8685-da1dae892b00\" (UID: \"42bd7fdf-2308-4364-8685-da1dae892b00\") " Oct 31 04:37:59.772693 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:37:59.772664 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42bd7fdf-2308-4364-8685-da1dae892b00-kube-api-access-s67zv" pod "42bd7fdf-2308-4364-8685-da1dae892b00" (UID: "42bd7fdf-2308-4364-8685-da1dae892b00"). InnerVolumeSpecName "kube-api-access-s67zv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:37:59.872733 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:37:59.872698 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s67zv\" (UniqueName: \"kubernetes.io/projected/42bd7fdf-2308-4364-8685-da1dae892b00-kube-api-access-s67zv\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:38:00.345884 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:00.345805 4376 generic.go:356] "Generic (PLEG): container finished" podID="42bd7fdf-2308-4364-8685-da1dae892b00" containerID="9d4ec0912315f46e9cdbd84705a3ce52ef74ecae0fbda8f4d8d6bb9fe95cc340" exitCode=137 Oct 31 04:38:00.345884 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:00.345844 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7096-a211b5" Oct 31 04:38:00.346087 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:00.345907 4376 scope.go:122] "RemoveContainer" containerID="9d4ec0912315f46e9cdbd84705a3ce52ef74ecae0fbda8f4d8d6bb9fe95cc340" Oct 31 04:38:00.356435 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:00.356400 4376 scope.go:122] "RemoveContainer" containerID="9d4ec0912315f46e9cdbd84705a3ce52ef74ecae0fbda8f4d8d6bb9fe95cc340" Oct 31 04:38:00.356692 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:38:00.356674 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d4ec0912315f46e9cdbd84705a3ce52ef74ecae0fbda8f4d8d6bb9fe95cc340\": container with ID starting with 9d4ec0912315f46e9cdbd84705a3ce52ef74ecae0fbda8f4d8d6bb9fe95cc340 not found: ID does not exist" containerID="9d4ec0912315f46e9cdbd84705a3ce52ef74ecae0fbda8f4d8d6bb9fe95cc340" Oct 31 04:38:00.356770 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:00.356702 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d4ec0912315f46e9cdbd84705a3ce52ef74ecae0fbda8f4d8d6bb9fe95cc340"} err="failed to get container status \"9d4ec0912315f46e9cdbd84705a3ce52ef74ecae0fbda8f4d8d6bb9fe95cc340\": rpc error: code = NotFound desc = could not find container \"9d4ec0912315f46e9cdbd84705a3ce52ef74ecae0fbda8f4d8d6bb9fe95cc340\": container with ID starting with 9d4ec0912315f46e9cdbd84705a3ce52ef74ecae0fbda8f4d8d6bb9fe95cc340 not found: ID does not exist" Oct 31 04:38:00.360279 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:38:00.360257 4376 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7096-f899c2\" is forbidden: User \"system:node:ip-10-0-134-155.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-155.ec2.internal' and this object" podUID="f8fe5232-5adc-4a38-817b-d46c38589c61" pod="opendatahub/test-curl-7096-f899c2" Oct 31 04:38:00.374512 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:38:00.374483 4376 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7096-f899c2\" is forbidden: User \"system:node:ip-10-0-134-155.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-155.ec2.internal' and this object" podUID="f8fe5232-5adc-4a38-817b-d46c38589c61" pod="opendatahub/test-curl-7096-f899c2" Oct 31 04:38:00.388685 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:38:00.388657 4376 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7096-a211b5\" is forbidden: User \"system:node:ip-10-0-134-155.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-155.ec2.internal' and this object" podUID="42bd7fdf-2308-4364-8685-da1dae892b00" pod="opendatahub/test-curl-7096-a211b5" Oct 31 04:38:00.443994 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:00.443961 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42bd7fdf-2308-4364-8685-da1dae892b00" path="/var/lib/kubelet/pods/42bd7fdf-2308-4364-8685-da1dae892b00/volumes" Oct 31 04:38:01.767891 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:01.767752 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7096-f899c2" Oct 31 04:38:01.795096 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:01.795064 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8fe5232-5adc-4a38-817b-d46c38589c61-kube-api-access-ghtts\" (UniqueName: \"kubernetes.io/projected/f8fe5232-5adc-4a38-817b-d46c38589c61-kube-api-access-ghtts\") pod \"f8fe5232-5adc-4a38-817b-d46c38589c61\" (UID: \"f8fe5232-5adc-4a38-817b-d46c38589c61\") " Oct 31 04:38:01.797199 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:01.797163 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8fe5232-5adc-4a38-817b-d46c38589c61-kube-api-access-ghtts" pod "f8fe5232-5adc-4a38-817b-d46c38589c61" (UID: "f8fe5232-5adc-4a38-817b-d46c38589c61"). InnerVolumeSpecName "kube-api-access-ghtts". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:38:01.897310 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:01.897284 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghtts\" (UniqueName: \"kubernetes.io/projected/f8fe5232-5adc-4a38-817b-d46c38589c61-kube-api-access-ghtts\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:38:01.902245 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:01.902224 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-zs49c" Oct 31 04:38:01.999061 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:01.999030 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0c24766-7a33-47fa-a162-cf7593c2d41c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0c24766-7a33-47fa-a162-cf7593c2d41c-workload-socket\") pod \"d0c24766-7a33-47fa-a162-cf7593c2d41c\" (UID: \"d0c24766-7a33-47fa-a162-cf7593c2d41c\") " Oct 31 04:38:01.999245 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:01.999128 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0c24766-7a33-47fa-a162-cf7593c2d41c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0c24766-7a33-47fa-a162-cf7593c2d41c-istio-envoy\") pod \"d0c24766-7a33-47fa-a162-cf7593c2d41c\" (UID: \"d0c24766-7a33-47fa-a162-cf7593c2d41c\") " Oct 31 04:38:01.999245 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:01.999170 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0c24766-7a33-47fa-a162-cf7593c2d41c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0c24766-7a33-47fa-a162-cf7593c2d41c-workload-certs\") pod \"d0c24766-7a33-47fa-a162-cf7593c2d41c\" (UID: \"d0c24766-7a33-47fa-a162-cf7593c2d41c\") " Oct 31 04:38:01.999402 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:01.999264 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0c24766-7a33-47fa-a162-cf7593c2d41c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0c24766-7a33-47fa-a162-cf7593c2d41c-istiod-ca-cert\") pod \"d0c24766-7a33-47fa-a162-cf7593c2d41c\" (UID: \"d0c24766-7a33-47fa-a162-cf7593c2d41c\") " Oct 31 04:38:01.999402 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:01.999284 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0c24766-7a33-47fa-a162-cf7593c2d41c-workload-socket" pod "d0c24766-7a33-47fa-a162-cf7593c2d41c" (UID: "d0c24766-7a33-47fa-a162-cf7593c2d41c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:38:01.999402 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:01.999314 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d0c24766-7a33-47fa-a162-cf7593c2d41c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0c24766-7a33-47fa-a162-cf7593c2d41c-istio-podinfo\") pod \"d0c24766-7a33-47fa-a162-cf7593c2d41c\" (UID: \"d0c24766-7a33-47fa-a162-cf7593c2d41c\") " Oct 31 04:38:01.999578 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:01.999463 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0c24766-7a33-47fa-a162-cf7593c2d41c-kube-api-access-4sqww\" (UniqueName: \"kubernetes.io/projected/d0c24766-7a33-47fa-a162-cf7593c2d41c-kube-api-access-4sqww\") pod \"d0c24766-7a33-47fa-a162-cf7593c2d41c\" (UID: \"d0c24766-7a33-47fa-a162-cf7593c2d41c\") " Oct 31 04:38:01.999578 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:01.999538 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0c24766-7a33-47fa-a162-cf7593c2d41c-workload-certs" pod "d0c24766-7a33-47fa-a162-cf7593c2d41c" (UID: "d0c24766-7a33-47fa-a162-cf7593c2d41c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:38:01.999707 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:01.999609 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0c24766-7a33-47fa-a162-cf7593c2d41c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0c24766-7a33-47fa-a162-cf7593c2d41c-istio-data\") pod \"d0c24766-7a33-47fa-a162-cf7593c2d41c\" (UID: \"d0c24766-7a33-47fa-a162-cf7593c2d41c\") " Oct 31 04:38:01.999707 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:01.999689 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0c24766-7a33-47fa-a162-cf7593c2d41c-istiod-ca-cert" pod "d0c24766-7a33-47fa-a162-cf7593c2d41c" (UID: "d0c24766-7a33-47fa-a162-cf7593c2d41c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:38:01.999791 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:01.999720 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0c24766-7a33-47fa-a162-cf7593c2d41c-istio-token\" (UniqueName: \"kubernetes.io/projected/d0c24766-7a33-47fa-a162-cf7593c2d41c-istio-token\") pod \"d0c24766-7a33-47fa-a162-cf7593c2d41c\" (UID: \"d0c24766-7a33-47fa-a162-cf7593c2d41c\") " Oct 31 04:38:01.999847 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:01.999809 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0c24766-7a33-47fa-a162-cf7593c2d41c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0c24766-7a33-47fa-a162-cf7593c2d41c-credential-socket\") pod \"d0c24766-7a33-47fa-a162-cf7593c2d41c\" (UID: \"d0c24766-7a33-47fa-a162-cf7593c2d41c\") " Oct 31 04:38:01.999916 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:01.999848 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0c24766-7a33-47fa-a162-cf7593c2d41c-istio-data" pod "d0c24766-7a33-47fa-a162-cf7593c2d41c" (UID: "d0c24766-7a33-47fa-a162-cf7593c2d41c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:38:02.000111 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:02.000090 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0c24766-7a33-47fa-a162-cf7593c2d41c-credential-socket" pod "d0c24766-7a33-47fa-a162-cf7593c2d41c" (UID: "d0c24766-7a33-47fa-a162-cf7593c2d41c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:38:02.000459 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:02.000439 4376 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0c24766-7a33-47fa-a162-cf7593c2d41c-istiod-ca-cert\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:38:02.000459 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:02.000460 4376 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0c24766-7a33-47fa-a162-cf7593c2d41c-istio-data\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:38:02.000635 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:02.000468 4376 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0c24766-7a33-47fa-a162-cf7593c2d41c-credential-socket\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:38:02.000635 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:02.000477 4376 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0c24766-7a33-47fa-a162-cf7593c2d41c-workload-socket\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:38:02.000635 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:02.000486 4376 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0c24766-7a33-47fa-a162-cf7593c2d41c-workload-certs\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:38:02.001804 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:02.001786 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0c24766-7a33-47fa-a162-cf7593c2d41c-istio-token" pod "d0c24766-7a33-47fa-a162-cf7593c2d41c" (UID: "d0c24766-7a33-47fa-a162-cf7593c2d41c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:38:02.001966 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:02.001948 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d0c24766-7a33-47fa-a162-cf7593c2d41c-istio-podinfo" pod "d0c24766-7a33-47fa-a162-cf7593c2d41c" (UID: "d0c24766-7a33-47fa-a162-cf7593c2d41c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 04:38:02.001966 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:02.001955 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0c24766-7a33-47fa-a162-cf7593c2d41c-istio-envoy" pod "d0c24766-7a33-47fa-a162-cf7593c2d41c" (UID: "d0c24766-7a33-47fa-a162-cf7593c2d41c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:38:02.002069 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:02.002047 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0c24766-7a33-47fa-a162-cf7593c2d41c-kube-api-access-4sqww" pod "d0c24766-7a33-47fa-a162-cf7593c2d41c" (UID: "d0c24766-7a33-47fa-a162-cf7593c2d41c"). InnerVolumeSpecName "kube-api-access-4sqww". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:38:02.101387 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:02.101357 4376 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0c24766-7a33-47fa-a162-cf7593c2d41c-istio-envoy\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:38:02.101387 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:02.101382 4376 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0c24766-7a33-47fa-a162-cf7593c2d41c-istio-podinfo\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:38:02.101387 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:02.101391 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4sqww\" (UniqueName: \"kubernetes.io/projected/d0c24766-7a33-47fa-a162-cf7593c2d41c-kube-api-access-4sqww\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:38:02.101718 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:02.101401 4376 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0c24766-7a33-47fa-a162-cf7593c2d41c-istio-token\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:38:02.361707 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:02.361677 4376 generic.go:356] "Generic (PLEG): container finished" podID="f8fe5232-5adc-4a38-817b-d46c38589c61" containerID="f31970fd4e6b99a488458bb8a35b511b819471010321d45bfbcac0ff419089bd" exitCode=137 Oct 31 04:38:02.361854 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:02.361736 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7096-f899c2" Oct 31 04:38:02.361854 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:02.361777 4376 scope.go:122] "RemoveContainer" containerID="f31970fd4e6b99a488458bb8a35b511b819471010321d45bfbcac0ff419089bd" Oct 31 04:38:02.363655 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:02.363634 4376 generic.go:356] "Generic (PLEG): container finished" podID="d0c24766-7a33-47fa-a162-cf7593c2d41c" containerID="a62e3e3373a2a3d2897a18b4aeb2c57154ba16a90b8e42d0b99b49e5fa64cc21" exitCode=0 Oct 31 04:38:02.363740 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:02.363690 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-zs49c" Oct 31 04:38:02.363740 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:02.363703 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-zs49c" event={"ID":"d0c24766-7a33-47fa-a162-cf7593c2d41c","Type":"ContainerDied","Data":"a62e3e3373a2a3d2897a18b4aeb2c57154ba16a90b8e42d0b99b49e5fa64cc21"} Oct 31 04:38:02.363824 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:02.363750 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-zs49c" event={"ID":"d0c24766-7a33-47fa-a162-cf7593c2d41c","Type":"ContainerDied","Data":"39f66c91db5e588d948c0718addc4a9f9989d9afa209a0ce675240236ceee9c6"} Oct 31 04:38:02.375974 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:02.375764 4376 scope.go:122] "RemoveContainer" containerID="f31970fd4e6b99a488458bb8a35b511b819471010321d45bfbcac0ff419089bd" Oct 31 04:38:02.376166 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:38:02.376144 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f31970fd4e6b99a488458bb8a35b511b819471010321d45bfbcac0ff419089bd\": container with ID starting with f31970fd4e6b99a488458bb8a35b511b819471010321d45bfbcac0ff419089bd not found: ID does not exist" containerID="f31970fd4e6b99a488458bb8a35b511b819471010321d45bfbcac0ff419089bd" Oct 31 04:38:02.376233 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:02.376177 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f31970fd4e6b99a488458bb8a35b511b819471010321d45bfbcac0ff419089bd"} err="failed to get container status \"f31970fd4e6b99a488458bb8a35b511b819471010321d45bfbcac0ff419089bd\": rpc error: code = NotFound desc = could not find container \"f31970fd4e6b99a488458bb8a35b511b819471010321d45bfbcac0ff419089bd\": container with ID starting with f31970fd4e6b99a488458bb8a35b511b819471010321d45bfbcac0ff419089bd not found: ID does not exist" Oct 31 04:38:02.376233 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:02.376199 4376 scope.go:122] "RemoveContainer" containerID="a62e3e3373a2a3d2897a18b4aeb2c57154ba16a90b8e42d0b99b49e5fa64cc21" Oct 31 04:38:02.398960 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:02.396696 4376 scope.go:122] "RemoveContainer" containerID="a62e3e3373a2a3d2897a18b4aeb2c57154ba16a90b8e42d0b99b49e5fa64cc21" Oct 31 04:38:02.400064 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:38:02.399968 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a62e3e3373a2a3d2897a18b4aeb2c57154ba16a90b8e42d0b99b49e5fa64cc21\": container with ID starting with a62e3e3373a2a3d2897a18b4aeb2c57154ba16a90b8e42d0b99b49e5fa64cc21 not found: ID does not exist" containerID="a62e3e3373a2a3d2897a18b4aeb2c57154ba16a90b8e42d0b99b49e5fa64cc21" Oct 31 04:38:02.400064 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:02.400003 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a62e3e3373a2a3d2897a18b4aeb2c57154ba16a90b8e42d0b99b49e5fa64cc21"} err="failed to get container status \"a62e3e3373a2a3d2897a18b4aeb2c57154ba16a90b8e42d0b99b49e5fa64cc21\": rpc error: code = NotFound desc = could not find container \"a62e3e3373a2a3d2897a18b4aeb2c57154ba16a90b8e42d0b99b49e5fa64cc21\": container with ID starting with a62e3e3373a2a3d2897a18b4aeb2c57154ba16a90b8e42d0b99b49e5fa64cc21 not found: ID does not exist" Oct 31 04:38:02.422682 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:02.422657 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-zs49c"] Oct 31 04:38:02.435132 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:02.435109 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-zs49c"] Oct 31 04:38:02.436782 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:38:02.436755 4376 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7096-f899c2\" is forbidden: User \"system:node:ip-10-0-134-155.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-155.ec2.internal' and this object" podUID="f8fe5232-5adc-4a38-817b-d46c38589c61" pod="opendatahub/test-curl-7096-f899c2" Oct 31 04:38:02.444078 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:02.444058 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0c24766-7a33-47fa-a162-cf7593c2d41c" path="/var/lib/kubelet/pods/d0c24766-7a33-47fa-a162-cf7593c2d41c/volumes" Oct 31 04:38:02.444475 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:02.444461 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8fe5232-5adc-4a38-817b-d46c38589c61" path="/var/lib/kubelet/pods/f8fe5232-5adc-4a38-817b-d46c38589c61/volumes" Oct 31 04:38:02.454431 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:38:02.454410 4376 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7096-f899c2\" is forbidden: User \"system:node:ip-10-0-134-155.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-155.ec2.internal' and this object" podUID="f8fe5232-5adc-4a38-817b-d46c38589c61" pod="opendatahub/test-curl-7096-f899c2" Oct 31 04:38:05.817035 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:05.817000 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-766728-69794bbd5f-s8zfz"] Oct 31 04:38:05.817875 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:05.817855 4376 cpu_manager.go:414] "containerMap: removing container" podUID="f8fe5232-5adc-4a38-817b-d46c38589c61" containerName="test-curl-7096-f899c2" Oct 31 04:38:05.817958 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:05.817878 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8fe5232-5adc-4a38-817b-d46c38589c61" containerName="test-curl-7096-f899c2" Oct 31 04:38:05.817958 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:05.817919 4376 cpu_manager.go:414] "containerMap: removing container" podUID="d0c24766-7a33-47fa-a162-cf7593c2d41c" containerName="istio-proxy" Oct 31 04:38:05.817958 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:05.817928 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0c24766-7a33-47fa-a162-cf7593c2d41c" containerName="istio-proxy" Oct 31 04:38:05.818073 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:05.818054 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0c24766-7a33-47fa-a162-cf7593c2d41c" containerName="istio-proxy" Oct 31 04:38:05.818073 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:05.818068 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8fe5232-5adc-4a38-817b-d46c38589c61" containerName="test-curl-7096-f899c2" Oct 31 04:38:05.823216 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:05.823197 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-766728-69794bbd5f-s8zfz" Oct 31 04:38:05.845226 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:05.845202 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-766728-69794bbd5f-s8zfz"] Oct 31 04:38:05.853684 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:05.853663 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w0-766728\"" Oct 31 04:38:05.853794 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:05.853757 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w0-766728-dockercfg-q84zp\"" Oct 31 04:38:05.890535 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:05.890495 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vd59h\" (UniqueName: \"kubernetes.io/projected/b9faf32d-45c1-4167-bb23-663987f3e602-kube-api-access-vd59h\") pod \"payload-pre-processing-e2e-shared-a-w0-766728-69794bbd5f-s8zfz\" (UID: \"b9faf32d-45c1-4167-bb23-663987f3e602\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-766728-69794bbd5f-s8zfz" Oct 31 04:38:05.890826 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:05.890804 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9faf32d-45c1-4167-bb23-663987f3e602-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-766728-69794bbd5f-s8zfz\" (UID: \"b9faf32d-45c1-4167-bb23-663987f3e602\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-766728-69794bbd5f-s8zfz" Oct 31 04:38:06.011354 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:06.011265 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vd59h\" (UniqueName: \"kubernetes.io/projected/b9faf32d-45c1-4167-bb23-663987f3e602-kube-api-access-vd59h\") pod \"payload-pre-processing-e2e-shared-a-w0-766728-69794bbd5f-s8zfz\" (UID: \"b9faf32d-45c1-4167-bb23-663987f3e602\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-766728-69794bbd5f-s8zfz" Oct 31 04:38:06.011545 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:06.011375 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9faf32d-45c1-4167-bb23-663987f3e602-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-766728-69794bbd5f-s8zfz\" (UID: \"b9faf32d-45c1-4167-bb23-663987f3e602\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-766728-69794bbd5f-s8zfz" Oct 31 04:38:06.023400 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:06.012506 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9faf32d-45c1-4167-bb23-663987f3e602-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-766728-69794bbd5f-s8zfz\" (UID: \"b9faf32d-45c1-4167-bb23-663987f3e602\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-766728-69794bbd5f-s8zfz" Oct 31 04:38:06.093653 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:06.065388 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-766728-d66dffc49-llqqr"] Oct 31 04:38:06.149524 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:06.127687 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-766728-d66dffc49-llqqr" Oct 31 04:38:06.181492 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:06.181449 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vd59h\" (UniqueName: \"kubernetes.io/projected/b9faf32d-45c1-4167-bb23-663987f3e602-kube-api-access-vd59h\") pod \"payload-pre-processing-e2e-shared-a-w0-766728-69794bbd5f-s8zfz\" (UID: \"b9faf32d-45c1-4167-bb23-663987f3e602\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-766728-69794bbd5f-s8zfz" Oct 31 04:38:06.183579 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:06.183540 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-766728-d66dffc49-llqqr"] Oct 31 04:38:06.187527 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:06.187501 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-766728-69794bbd5f-s8zfz" Oct 31 04:38:06.253861 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:06.253796 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f05ba444-d067-4e81-a4ca-cb9ef426c2a1-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-766728-d66dffc49-llqqr\" (UID: \"f05ba444-d067-4e81-a4ca-cb9ef426c2a1\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-766728-d66dffc49-llqqr" Oct 31 04:38:06.254360 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:06.254310 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6n58b\" (UniqueName: \"kubernetes.io/projected/f05ba444-d067-4e81-a4ca-cb9ef426c2a1-kube-api-access-6n58b\") pod \"payload-processing-e2e-shared-a-w0-766728-d66dffc49-llqqr\" (UID: \"f05ba444-d067-4e81-a4ca-cb9ef426c2a1\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-766728-d66dffc49-llqqr" Oct 31 04:38:06.356005 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:06.355930 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f05ba444-d067-4e81-a4ca-cb9ef426c2a1-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-766728-d66dffc49-llqqr\" (UID: \"f05ba444-d067-4e81-a4ca-cb9ef426c2a1\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-766728-d66dffc49-llqqr" Oct 31 04:38:06.356144 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:06.356066 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6n58b\" (UniqueName: \"kubernetes.io/projected/f05ba444-d067-4e81-a4ca-cb9ef426c2a1-kube-api-access-6n58b\") pod \"payload-processing-e2e-shared-a-w0-766728-d66dffc49-llqqr\" (UID: \"f05ba444-d067-4e81-a4ca-cb9ef426c2a1\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-766728-d66dffc49-llqqr" Oct 31 04:38:06.356657 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:06.356636 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f05ba444-d067-4e81-a4ca-cb9ef426c2a1-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-766728-d66dffc49-llqqr\" (UID: \"f05ba444-d067-4e81-a4ca-cb9ef426c2a1\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-766728-d66dffc49-llqqr" Oct 31 04:38:06.380137 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:06.380115 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6n58b\" (UniqueName: \"kubernetes.io/projected/f05ba444-d067-4e81-a4ca-cb9ef426c2a1-kube-api-access-6n58b\") pod \"payload-processing-e2e-shared-a-w0-766728-d66dffc49-llqqr\" (UID: \"f05ba444-d067-4e81-a4ca-cb9ef426c2a1\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-766728-d66dffc49-llqqr" Oct 31 04:38:06.502782 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:38:06.502751 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9faf32d_45c1_4167_bb23_663987f3e602.slice/crio-5286b69d00501647ee6dc7bc285f551a6f0918ef7ed706050737f4608a9fbb2c WatchSource:0}: Error finding container 5286b69d00501647ee6dc7bc285f551a6f0918ef7ed706050737f4608a9fbb2c: Status 404 returned error can't find the container with id 5286b69d00501647ee6dc7bc285f551a6f0918ef7ed706050737f4608a9fbb2c Oct 31 04:38:06.504279 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:06.504257 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-766728-69794bbd5f-s8zfz"] Oct 31 04:38:06.542287 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:06.542267 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-766728-d66dffc49-llqqr" Oct 31 04:38:06.844163 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:38:06.844128 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf05ba444_d067_4e81_a4ca_cb9ef426c2a1.slice/crio-178b91ab5a71aed1689d629a670a3dbebed60a8e2b41b0b0cd111e0f31544010 WatchSource:0}: Error finding container 178b91ab5a71aed1689d629a670a3dbebed60a8e2b41b0b0cd111e0f31544010: Status 404 returned error can't find the container with id 178b91ab5a71aed1689d629a670a3dbebed60a8e2b41b0b0cd111e0f31544010 Oct 31 04:38:06.847123 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:06.847099 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-766728-d66dffc49-llqqr"] Oct 31 04:38:07.409753 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:07.409714 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-766728-69794bbd5f-s8zfz" event={"ID":"b9faf32d-45c1-4167-bb23-663987f3e602","Type":"ContainerStarted","Data":"a672b1062cd591142810625dd353b3fd9354a6c9bd3b42763d8cd1c2e727f474"} Oct 31 04:38:07.409959 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:07.409782 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-766728-69794bbd5f-s8zfz" Oct 31 04:38:07.409959 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:07.409806 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-766728-69794bbd5f-s8zfz" event={"ID":"b9faf32d-45c1-4167-bb23-663987f3e602","Type":"ContainerStarted","Data":"5286b69d00501647ee6dc7bc285f551a6f0918ef7ed706050737f4608a9fbb2c"} Oct 31 04:38:07.411533 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:07.411504 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-766728-d66dffc49-llqqr" event={"ID":"f05ba444-d067-4e81-a4ca-cb9ef426c2a1","Type":"ContainerStarted","Data":"6c89e8848671e9859f7b4c56e2e5e709a6ff1cd0c8818e41a10467a489e3afda"} Oct 31 04:38:07.411658 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:07.411557 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-766728-d66dffc49-llqqr" event={"ID":"f05ba444-d067-4e81-a4ca-cb9ef426c2a1","Type":"ContainerStarted","Data":"178b91ab5a71aed1689d629a670a3dbebed60a8e2b41b0b0cd111e0f31544010"} Oct 31 04:38:07.411658 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:07.411589 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-766728-d66dffc49-llqqr" Oct 31 04:38:07.453776 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:07.453732 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-766728-69794bbd5f-s8zfz" podStartSLOduration=2.453719981 podStartE2EDuration="2.453719981s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:38:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:38:07.451491301 +0000 UTC m=+2035.165825919" watchObservedRunningTime="2026-10-31 04:38:07.453719981 +0000 UTC m=+2035.168054597" Oct 31 04:38:07.495464 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:07.495418 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-766728-d66dffc49-llqqr" podStartSLOduration=2.495405826 podStartE2EDuration="2.495405826s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:38:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:38:07.494828054 +0000 UTC m=+2035.209162671" watchObservedRunningTime="2026-10-31 04:38:07.495405826 +0000 UTC m=+2035.209740440" Oct 31 04:38:11.001394 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:38:11.001358 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 04:38:18.423006 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:18.422970 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-766728-d66dffc49-llqqr" Oct 31 04:38:18.423457 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:18.423047 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-766728-69794bbd5f-s8zfz" Oct 31 04:38:23.548743 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:23.548708 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-57dc6d/auth-test-model-57dc6d-kserve-577cb5ffd5-2wk4v"] Oct 31 04:38:26.741303 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:26.741265 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bgh5f"] Oct 31 04:38:26.742284 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:26.742222 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bgh5f" podUID="594f2d93-2f76-4632-904d-402fa126740c" containerName="istio-proxy" containerID="cri-o://a907730c8fd50e968e3d15ba2fd87f6db56a4e6102701e117130f34be25d9b5d" gracePeriod=30 Oct 31 04:38:26.746355 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:26.746314 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-cjmdq"] Oct 31 04:38:26.746558 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:26.746519 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-cjmdq" podUID="6213a9f9-302f-4896-a1ce-dc0c7323082c" containerName="istio-proxy" containerID="cri-o://76596c78a36ea8bc117ade55c61089b261bb2d8c0405ac33a1c0324f5787eccb" gracePeriod=30 Oct 31 04:38:26.785552 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:26.785515 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-rw9sz"] Oct 31 04:38:26.785811 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:26.785786 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-rw9sz" podUID="ead2084a-70a7-47d2-a757-f340d971bfb6" containerName="istio-proxy" containerID="cri-o://c4227668546e211ff6d5f7c2783bec5fc415a777c6fe37cf575f22e947f32ba2" gracePeriod=30 Oct 31 04:38:26.789764 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:26.789713 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-wq9cv"] Oct 31 04:38:26.789979 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:26.789950 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-wq9cv" podUID="17ea2397-450e-4505-9a48-0cf93f76065e" containerName="istio-proxy" containerID="cri-o://ff9953f770bf9c4b20cfb7ca85ab92c25a709850cf99cd7c3ab370b9718fd576" gracePeriod=30 Oct 31 04:38:27.911354 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:27.904892 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-crjjl"] Oct 31 04:38:27.911354 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:27.905909 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-crjjl" podUID="63486e50-4059-425f-b04a-4adea538d427" containerName="istio-proxy" containerID="cri-o://ed822ccdccebd79101c8df29b99ac5823650b406625a22bb6e931795f33bada2" gracePeriod=30 Oct 31 04:38:27.940348 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:27.931176 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mrx9s"] Oct 31 04:38:27.940348 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:27.931424 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mrx9s" podUID="70120ad8-48aa-44b4-b396-64eab56ce59b" containerName="istio-proxy" containerID="cri-o://1dea307cb0f07118a743766d464d20933d0bf91533edea2b93501e78a8f198ca" gracePeriod=30 Oct 31 04:38:28.010350 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:27.999812 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-9633f0/auth-test-model-9633f0-kserve-6cbd55ffd9-z9rjx"] Oct 31 04:38:31.957974 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:31.956911 4376 patch_prober.go:29] interesting pod/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bgh5f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Oct 31 04:38:31.957974 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:31.956971 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bgh5f" podUID="594f2d93-2f76-4632-904d-402fa126740c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Oct 31 04:38:32.168449 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:32.168422 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7108-0d3be9"] Oct 31 04:38:32.176229 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:32.176205 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7108-0d3be9"] Oct 31 04:38:32.176592 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:32.176572 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7108-0d3be9" Oct 31 04:38:32.193525 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:32.192812 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-prswc\"" Oct 31 04:38:32.198582 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:32.198551 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-cjmdq" Oct 31 04:38:32.221350 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:32.221312 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-rw9sz" Oct 31 04:38:32.333260 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:32.333235 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bgh5f" Oct 31 04:38:32.341316 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:32.341289 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-wq9cv" Oct 31 04:38:32.344386 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:32.344359 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ead2084a-70a7-47d2-a757-f340d971bfb6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ead2084a-70a7-47d2-a757-f340d971bfb6-workload-socket\") pod \"ead2084a-70a7-47d2-a757-f340d971bfb6\" (UID: \"ead2084a-70a7-47d2-a757-f340d971bfb6\") " Oct 31 04:38:32.344516 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:32.344401 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6213a9f9-302f-4896-a1ce-dc0c7323082c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6213a9f9-302f-4896-a1ce-dc0c7323082c-istio-data\") pod \"6213a9f9-302f-4896-a1ce-dc0c7323082c\" (UID: \"6213a9f9-302f-4896-a1ce-dc0c7323082c\") " Oct 31 04:38:32.344516 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:32.344426 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ead2084a-70a7-47d2-a757-f340d971bfb6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ead2084a-70a7-47d2-a757-f340d971bfb6-istio-envoy\") pod \"ead2084a-70a7-47d2-a757-f340d971bfb6\" (UID: \"ead2084a-70a7-47d2-a757-f340d971bfb6\") " Oct 31 04:38:32.344516 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:32.344460 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ead2084a-70a7-47d2-a757-f340d971bfb6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ead2084a-70a7-47d2-a757-f340d971bfb6-istio-podinfo\") pod \"ead2084a-70a7-47d2-a757-f340d971bfb6\" (UID: \"ead2084a-70a7-47d2-a757-f340d971bfb6\") " Oct 31 04:38:32.344683 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:32.344585 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6213a9f9-302f-4896-a1ce-dc0c7323082c-kube-api-access-n7pq2\" (UniqueName: \"kubernetes.io/projected/6213a9f9-302f-4896-a1ce-dc0c7323082c-kube-api-access-n7pq2\") pod \"6213a9f9-302f-4896-a1ce-dc0c7323082c\" (UID: \"6213a9f9-302f-4896-a1ce-dc0c7323082c\") " Oct 31 04:38:32.344683 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:32.344665 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6213a9f9-302f-4896-a1ce-dc0c7323082c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6213a9f9-302f-4896-a1ce-dc0c7323082c-istio-envoy\") pod \"6213a9f9-302f-4896-a1ce-dc0c7323082c\" (UID: \"6213a9f9-302f-4896-a1ce-dc0c7323082c\") " Oct 31 04:38:32.344793 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:32.344709 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6213a9f9-302f-4896-a1ce-dc0c7323082c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6213a9f9-302f-4896-a1ce-dc0c7323082c-istiod-ca-cert\") pod \"6213a9f9-302f-4896-a1ce-dc0c7323082c\" (UID: \"6213a9f9-302f-4896-a1ce-dc0c7323082c\") " Oct 31 04:38:32.344793 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:32.344753 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6213a9f9-302f-4896-a1ce-dc0c7323082c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6213a9f9-302f-4896-a1ce-dc0c7323082c-istio-podinfo\") pod \"6213a9f9-302f-4896-a1ce-dc0c7323082c\" (UID: \"6213a9f9-302f-4896-a1ce-dc0c7323082c\") " Oct 31 04:38:32.344899 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:32.344830 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ead2084a-70a7-47d2-a757-f340d971bfb6-kube-api-access-mzhw7\" (UniqueName: \"kubernetes.io/projected/ead2084a-70a7-47d2-a757-f340d971bfb6-kube-api-access-mzhw7\") pod \"ead2084a-70a7-47d2-a757-f340d971bfb6\" (UID: \"ead2084a-70a7-47d2-a757-f340d971bfb6\") " Oct 31 04:38:32.344899 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:32.344864 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6213a9f9-302f-4896-a1ce-dc0c7323082c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6213a9f9-302f-4896-a1ce-dc0c7323082c-credential-socket\") pod \"6213a9f9-302f-4896-a1ce-dc0c7323082c\" (UID: \"6213a9f9-302f-4896-a1ce-dc0c7323082c\") " Oct 31 04:38:32.345004 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:32.344900 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ead2084a-70a7-47d2-a757-f340d971bfb6-istio-token\" (UniqueName: \"kubernetes.io/projected/ead2084a-70a7-47d2-a757-f340d971bfb6-istio-token\") pod \"ead2084a-70a7-47d2-a757-f340d971bfb6\" (UID: \"ead2084a-70a7-47d2-a757-f340d971bfb6\") " Oct 31 04:38:32.345004 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:32.344993 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6213a9f9-302f-4896-a1ce-dc0c7323082c-istio-token\" (UniqueName: \"kubernetes.io/projected/6213a9f9-302f-4896-a1ce-dc0c7323082c-istio-token\") pod \"6213a9f9-302f-4896-a1ce-dc0c7323082c\" (UID: \"6213a9f9-302f-4896-a1ce-dc0c7323082c\") " Oct 31 04:38:32.345109 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:32.345096 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6213a9f9-302f-4896-a1ce-dc0c7323082c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6213a9f9-302f-4896-a1ce-dc0c7323082c-workload-certs\") pod \"6213a9f9-302f-4896-a1ce-dc0c7323082c\" (UID: \"6213a9f9-302f-4896-a1ce-dc0c7323082c\") " Oct 31 04:38:32.345184 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:32.345168 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ead2084a-70a7-47d2-a757-f340d971bfb6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ead2084a-70a7-47d2-a757-f340d971bfb6-workload-certs\") pod \"ead2084a-70a7-47d2-a757-f340d971bfb6\" (UID: \"ead2084a-70a7-47d2-a757-f340d971bfb6\") " Oct 31 04:38:32.345242 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:32.345226 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ead2084a-70a7-47d2-a757-f340d971bfb6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ead2084a-70a7-47d2-a757-f340d971bfb6-istio-data\") pod \"ead2084a-70a7-47d2-a757-f340d971bfb6\" (UID: \"ead2084a-70a7-47d2-a757-f340d971bfb6\") " Oct 31 04:38:32.345298 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:32.345274 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6213a9f9-302f-4896-a1ce-dc0c7323082c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6213a9f9-302f-4896-a1ce-dc0c7323082c-workload-socket\") pod \"6213a9f9-302f-4896-a1ce-dc0c7323082c\" (UID: \"6213a9f9-302f-4896-a1ce-dc0c7323082c\") " Oct 31 04:38:32.345367 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:32.345318 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ead2084a-70a7-47d2-a757-f340d971bfb6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ead2084a-70a7-47d2-a757-f340d971bfb6-istiod-ca-cert\") pod \"ead2084a-70a7-47d2-a757-f340d971bfb6\" (UID: \"ead2084a-70a7-47d2-a757-f340d971bfb6\") " Oct 31 04:38:32.345437 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:32.345419 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ead2084a-70a7-47d2-a757-f340d971bfb6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ead2084a-70a7-47d2-a757-f340d971bfb6-credential-socket\") pod \"ead2084a-70a7-47d2-a757-f340d971bfb6\" (UID: \"ead2084a-70a7-47d2-a757-f340d971bfb6\") " Oct 31 04:38:32.345804 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:32.345783 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngjkv\" (UniqueName: \"kubernetes.io/projected/7701398e-558d-48c9-8e08-1010af929cc7-kube-api-access-ngjkv\") pod \"test-curl-7108-0d3be9\" (UID: \"7701398e-558d-48c9-8e08-1010af929cc7\") " pod="opendatahub/test-curl-7108-0d3be9" Oct 31 04:38:32.346218 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:32.346196 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ead2084a-70a7-47d2-a757-f340d971bfb6-workload-socket" pod "ead2084a-70a7-47d2-a757-f340d971bfb6" (UID: "ead2084a-70a7-47d2-a757-f340d971bfb6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:38:32.346511 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:32.346415 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6213a9f9-302f-4896-a1ce-dc0c7323082c-istio-data" pod "6213a9f9-302f-4896-a1ce-dc0c7323082c" (UID: "6213a9f9-302f-4896-a1ce-dc0c7323082c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:38:32.346612 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:32.346562 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6213a9f9-302f-4896-a1ce-dc0c7323082c-credential-socket" pod "6213a9f9-302f-4896-a1ce-dc0c7323082c" (UID: "6213a9f9-302f-4896-a1ce-dc0c7323082c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:38:32.346783 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:32.346763 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ead2084a-70a7-47d2-a757-f340d971bfb6-istio-data" pod "ead2084a-70a7-47d2-a757-f340d971bfb6" (UID: "ead2084a-70a7-47d2-a757-f340d971bfb6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:38:32.346952 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:32.346936 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ead2084a-70a7-47d2-a757-f340d971bfb6-credential-socket" pod "ead2084a-70a7-47d2-a757-f340d971bfb6" (UID: "ead2084a-70a7-47d2-a757-f340d971bfb6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:38:32.347008 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:32.346953 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ead2084a-70a7-47d2-a757-f340d971bfb6-istiod-ca-cert" pod "ead2084a-70a7-47d2-a757-f340d971bfb6" (UID: "ead2084a-70a7-47d2-a757-f340d971bfb6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:38:32.347589 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:32.347567 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6213a9f9-302f-4896-a1ce-dc0c7323082c-workload-socket" pod "6213a9f9-302f-4896-a1ce-dc0c7323082c" (UID: "6213a9f9-302f-4896-a1ce-dc0c7323082c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:38:32.347770 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:32.347750 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6213a9f9-302f-4896-a1ce-dc0c7323082c-workload-certs" pod "6213a9f9-302f-4896-a1ce-dc0c7323082c" (UID: "6213a9f9-302f-4896-a1ce-dc0c7323082c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:38:32.347770 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:32.347756 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ead2084a-70a7-47d2-a757-f340d971bfb6-workload-certs" pod "ead2084a-70a7-47d2-a757-f340d971bfb6" (UID: "ead2084a-70a7-47d2-a757-f340d971bfb6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:38:32.348638 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:32.348616 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6213a9f9-302f-4896-a1ce-dc0c7323082c-istiod-ca-cert" pod "6213a9f9-302f-4896-a1ce-dc0c7323082c" (UID: "6213a9f9-302f-4896-a1ce-dc0c7323082c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:38:32.350595 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:32.350572 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ead2084a-70a7-47d2-a757-f340d971bfb6-istio-podinfo" pod "ead2084a-70a7-47d2-a757-f340d971bfb6" (UID: "ead2084a-70a7-47d2-a757-f340d971bfb6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 04:38:32.350705 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:32.350606 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6213a9f9-302f-4896-a1ce-dc0c7323082c-istio-token" pod "6213a9f9-302f-4896-a1ce-dc0c7323082c" (UID: "6213a9f9-302f-4896-a1ce-dc0c7323082c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:38:32.351955 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:32.351932 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6213a9f9-302f-4896-a1ce-dc0c7323082c-istio-podinfo" pod "6213a9f9-302f-4896-a1ce-dc0c7323082c" (UID: "6213a9f9-302f-4896-a1ce-dc0c7323082c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 04:38:32.352122 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:32.352104 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6213a9f9-302f-4896-a1ce-dc0c7323082c-kube-api-access-n7pq2" pod "6213a9f9-302f-4896-a1ce-dc0c7323082c" (UID: "6213a9f9-302f-4896-a1ce-dc0c7323082c"). InnerVolumeSpecName "kube-api-access-n7pq2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:38:32.352382 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:32.352363 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6213a9f9-302f-4896-a1ce-dc0c7323082c-istio-envoy" pod "6213a9f9-302f-4896-a1ce-dc0c7323082c" (UID: "6213a9f9-302f-4896-a1ce-dc0c7323082c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:38:32.354407 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:32.354380 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ead2084a-70a7-47d2-a757-f340d971bfb6-istio-envoy" pod "ead2084a-70a7-47d2-a757-f340d971bfb6" (UID: "ead2084a-70a7-47d2-a757-f340d971bfb6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:38:32.354594 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:32.354576 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ead2084a-70a7-47d2-a757-f340d971bfb6-kube-api-access-mzhw7" pod "ead2084a-70a7-47d2-a757-f340d971bfb6" (UID: "ead2084a-70a7-47d2-a757-f340d971bfb6"). InnerVolumeSpecName "kube-api-access-mzhw7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:38:32.355825 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:32.355801 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ead2084a-70a7-47d2-a757-f340d971bfb6-istio-token" pod "ead2084a-70a7-47d2-a757-f340d971bfb6" (UID: "ead2084a-70a7-47d2-a757-f340d971bfb6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:38:32.447392 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:32.447132 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/594f2d93-2f76-4632-904d-402fa126740c-istio-token\" (UniqueName: \"kubernetes.io/projected/594f2d93-2f76-4632-904d-402fa126740c-istio-token\") pod \"594f2d93-2f76-4632-904d-402fa126740c\" (UID: \"594f2d93-2f76-4632-904d-402fa126740c\") " Oct 31 04:38:32.447392 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:32.447182 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17ea2397-450e-4505-9a48-0cf93f76065e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17ea2397-450e-4505-9a48-0cf93f76065e-credential-socket\") pod \"17ea2397-450e-4505-9a48-0cf93f76065e\" (UID: \"17ea2397-450e-4505-9a48-0cf93f76065e\") " Oct 31 04:38:32.447392 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:32.447230 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/594f2d93-2f76-4632-904d-402fa126740c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/594f2d93-2f76-4632-904d-402fa126740c-istio-podinfo\") pod \"594f2d93-2f76-4632-904d-402fa126740c\" (UID: \"594f2d93-2f76-4632-904d-402fa126740c\") " Oct 31 04:38:32.447392 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:32.447267 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/594f2d93-2f76-4632-904d-402fa126740c-kube-api-access-7pfm2\" (UniqueName: \"kubernetes.io/projected/594f2d93-2f76-4632-904d-402fa126740c-kube-api-access-7pfm2\") pod \"594f2d93-2f76-4632-904d-402fa126740c\" (UID: \"594f2d93-2f76-4632-904d-402fa126740c\") " Oct 31 04:38:32.447392 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:32.447307 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17ea2397-450e-4505-9a48-0cf93f76065e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17ea2397-450e-4505-9a48-0cf93f76065e-istio-data\") pod \"17ea2397-450e-4505-9a48-0cf93f76065e\" (UID: \"17ea2397-450e-4505-9a48-0cf93f76065e\") " Oct 31 04:38:32.447392 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:32.447349 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/17ea2397-450e-4505-9a48-0cf93f76065e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17ea2397-450e-4505-9a48-0cf93f76065e-istio-podinfo\") pod \"17ea2397-450e-4505-9a48-0cf93f76065e\" (UID: \"17ea2397-450e-4505-9a48-0cf93f76065e\") " Oct 31 04:38:32.447392 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:32.447384 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17ea2397-450e-4505-9a48-0cf93f76065e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17ea2397-450e-4505-9a48-0cf93f76065e-workload-certs\") pod \"17ea2397-450e-4505-9a48-0cf93f76065e\" (UID: \"17ea2397-450e-4505-9a48-0cf93f76065e\") " Oct 31 04:38:32.447898 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:32.447432 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17ea2397-450e-4505-9a48-0cf93f76065e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17ea2397-450e-4505-9a48-0cf93f76065e-istiod-ca-cert\") pod \"17ea2397-450e-4505-9a48-0cf93f76065e\" (UID: \"17ea2397-450e-4505-9a48-0cf93f76065e\") " Oct 31 04:38:32.447898 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:32.447480 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/594f2d93-2f76-4632-904d-402fa126740c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/594f2d93-2f76-4632-904d-402fa126740c-credential-socket\") pod \"594f2d93-2f76-4632-904d-402fa126740c\" (UID: \"594f2d93-2f76-4632-904d-402fa126740c\") " Oct 31 04:38:32.447898 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:32.447516 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/594f2d93-2f76-4632-904d-402fa126740c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/594f2d93-2f76-4632-904d-402fa126740c-workload-certs\") pod \"594f2d93-2f76-4632-904d-402fa126740c\" (UID: \"594f2d93-2f76-4632-904d-402fa126740c\") " Oct 31 04:38:32.447898 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:32.447576 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17ea2397-450e-4505-9a48-0cf93f76065e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17ea2397-450e-4505-9a48-0cf93f76065e-istio-envoy\") pod \"17ea2397-450e-4505-9a48-0cf93f76065e\" (UID: \"17ea2397-450e-4505-9a48-0cf93f76065e\") " Oct 31 04:38:32.447898 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:32.447611 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/594f2d93-2f76-4632-904d-402fa126740c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/594f2d93-2f76-4632-904d-402fa126740c-istiod-ca-cert\") pod \"594f2d93-2f76-4632-904d-402fa126740c\" (UID: \"594f2d93-2f76-4632-904d-402fa126740c\") " Oct 31 04:38:32.447898 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:32.447649 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17ea2397-450e-4505-9a48-0cf93f76065e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17ea2397-450e-4505-9a48-0cf93f76065e-workload-socket\") pod \"17ea2397-450e-4505-9a48-0cf93f76065e\" (UID: \"17ea2397-450e-4505-9a48-0cf93f76065e\") " Oct 31 04:38:32.447898 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:32.447743 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17ea2397-450e-4505-9a48-0cf93f76065e-kube-api-access-n8qdc\" (UniqueName: \"kubernetes.io/projected/17ea2397-450e-4505-9a48-0cf93f76065e-kube-api-access-n8qdc\") pod \"17ea2397-450e-4505-9a48-0cf93f76065e\" (UID: \"17ea2397-450e-4505-9a48-0cf93f76065e\") " Oct 31 04:38:32.447898 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:32.447782 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17ea2397-450e-4505-9a48-0cf93f76065e-istio-token\" (UniqueName: \"kubernetes.io/projected/17ea2397-450e-4505-9a48-0cf93f76065e-istio-token\") pod \"17ea2397-450e-4505-9a48-0cf93f76065e\" (UID: \"17ea2397-450e-4505-9a48-0cf93f76065e\") " Oct 31 04:38:32.447898 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:32.447842 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/594f2d93-2f76-4632-904d-402fa126740c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/594f2d93-2f76-4632-904d-402fa126740c-istio-data\") pod \"594f2d93-2f76-4632-904d-402fa126740c\" (UID: \"594f2d93-2f76-4632-904d-402fa126740c\") " Oct 31 04:38:32.450971 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:32.447917 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/594f2d93-2f76-4632-904d-402fa126740c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/594f2d93-2f76-4632-904d-402fa126740c-istio-envoy\") pod \"594f2d93-2f76-4632-904d-402fa126740c\" (UID: \"594f2d93-2f76-4632-904d-402fa126740c\") " Oct 31 04:38:32.450971 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:32.447938 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/594f2d93-2f76-4632-904d-402fa126740c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/594f2d93-2f76-4632-904d-402fa126740c-workload-socket\") pod \"594f2d93-2f76-4632-904d-402fa126740c\" (UID: \"594f2d93-2f76-4632-904d-402fa126740c\") " Oct 31 04:38:32.450971 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:32.448210 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngjkv\" (UniqueName: \"kubernetes.io/projected/7701398e-558d-48c9-8e08-1010af929cc7-kube-api-access-ngjkv\") pod \"test-curl-7108-0d3be9\" (UID: \"7701398e-558d-48c9-8e08-1010af929cc7\") " pod="opendatahub/test-curl-7108-0d3be9" Oct 31 04:38:32.450971 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:32.448603 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzhw7\" (UniqueName: \"kubernetes.io/projected/ead2084a-70a7-47d2-a757-f340d971bfb6-kube-api-access-mzhw7\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:38:32.450971 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:32.448622 4376 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6213a9f9-302f-4896-a1ce-dc0c7323082c-credential-socket\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:38:32.450971 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:32.448638 4376 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ead2084a-70a7-47d2-a757-f340d971bfb6-istio-token\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:38:32.450971 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:32.448651 4376 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6213a9f9-302f-4896-a1ce-dc0c7323082c-istio-token\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:38:32.450971 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:32.448661 4376 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6213a9f9-302f-4896-a1ce-dc0c7323082c-workload-certs\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:38:32.450971 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:32.448669 4376 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ead2084a-70a7-47d2-a757-f340d971bfb6-workload-certs\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:38:32.450971 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:32.448677 4376 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ead2084a-70a7-47d2-a757-f340d971bfb6-istio-data\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:38:32.450971 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:32.448685 4376 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6213a9f9-302f-4896-a1ce-dc0c7323082c-workload-socket\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:38:32.450971 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:32.448693 4376 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ead2084a-70a7-47d2-a757-f340d971bfb6-istiod-ca-cert\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:38:32.450971 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:32.448702 4376 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ead2084a-70a7-47d2-a757-f340d971bfb6-credential-socket\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:38:32.450971 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:32.448712 4376 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ead2084a-70a7-47d2-a757-f340d971bfb6-workload-socket\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:38:32.450971 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:32.448720 4376 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6213a9f9-302f-4896-a1ce-dc0c7323082c-istio-data\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:38:32.450971 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:32.448729 4376 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ead2084a-70a7-47d2-a757-f340d971bfb6-istio-envoy\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:38:32.450971 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:32.448737 4376 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ead2084a-70a7-47d2-a757-f340d971bfb6-istio-podinfo\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:38:32.450971 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:32.448747 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n7pq2\" (UniqueName: \"kubernetes.io/projected/6213a9f9-302f-4896-a1ce-dc0c7323082c-kube-api-access-n7pq2\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:38:32.450971 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:32.448757 4376 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6213a9f9-302f-4896-a1ce-dc0c7323082c-istio-envoy\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:38:32.450971 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:32.448769 4376 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6213a9f9-302f-4896-a1ce-dc0c7323082c-istiod-ca-cert\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:38:32.450971 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:32.448788 4376 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6213a9f9-302f-4896-a1ce-dc0c7323082c-istio-podinfo\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:38:32.459377 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:32.452988 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/594f2d93-2f76-4632-904d-402fa126740c-istio-token" pod "594f2d93-2f76-4632-904d-402fa126740c" (UID: "594f2d93-2f76-4632-904d-402fa126740c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:38:32.459377 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:32.453633 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/594f2d93-2f76-4632-904d-402fa126740c-istiod-ca-cert" pod "594f2d93-2f76-4632-904d-402fa126740c" (UID: "594f2d93-2f76-4632-904d-402fa126740c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:38:32.459377 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:32.454089 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17ea2397-450e-4505-9a48-0cf93f76065e-workload-socket" pod "17ea2397-450e-4505-9a48-0cf93f76065e" (UID: "17ea2397-450e-4505-9a48-0cf93f76065e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:38:32.459377 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:32.454162 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17ea2397-450e-4505-9a48-0cf93f76065e-istio-data" pod "17ea2397-450e-4505-9a48-0cf93f76065e" (UID: "17ea2397-450e-4505-9a48-0cf93f76065e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:38:32.459377 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:32.454401 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17ea2397-450e-4505-9a48-0cf93f76065e-workload-certs" pod "17ea2397-450e-4505-9a48-0cf93f76065e" (UID: "17ea2397-450e-4505-9a48-0cf93f76065e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:38:32.459377 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:32.455871 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17ea2397-450e-4505-9a48-0cf93f76065e-credential-socket" pod "17ea2397-450e-4505-9a48-0cf93f76065e" (UID: "17ea2397-450e-4505-9a48-0cf93f76065e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:38:32.459377 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:32.456172 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/594f2d93-2f76-4632-904d-402fa126740c-istio-podinfo" pod "594f2d93-2f76-4632-904d-402fa126740c" (UID: "594f2d93-2f76-4632-904d-402fa126740c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 04:38:32.459377 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:32.457944 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/17ea2397-450e-4505-9a48-0cf93f76065e-istio-podinfo" pod "17ea2397-450e-4505-9a48-0cf93f76065e" (UID: "17ea2397-450e-4505-9a48-0cf93f76065e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 04:38:32.459377 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:32.458224 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17ea2397-450e-4505-9a48-0cf93f76065e-istiod-ca-cert" pod "17ea2397-450e-4505-9a48-0cf93f76065e" (UID: "17ea2397-450e-4505-9a48-0cf93f76065e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:38:32.459377 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:32.458753 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/594f2d93-2f76-4632-904d-402fa126740c-credential-socket" pod "594f2d93-2f76-4632-904d-402fa126740c" (UID: "594f2d93-2f76-4632-904d-402fa126740c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:38:32.459377 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:32.459123 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/594f2d93-2f76-4632-904d-402fa126740c-istio-data" pod "594f2d93-2f76-4632-904d-402fa126740c" (UID: "594f2d93-2f76-4632-904d-402fa126740c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:38:32.459377 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:32.459323 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/594f2d93-2f76-4632-904d-402fa126740c-workload-socket" pod "594f2d93-2f76-4632-904d-402fa126740c" (UID: "594f2d93-2f76-4632-904d-402fa126740c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:38:32.460397 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:32.460361 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/594f2d93-2f76-4632-904d-402fa126740c-workload-certs" pod "594f2d93-2f76-4632-904d-402fa126740c" (UID: "594f2d93-2f76-4632-904d-402fa126740c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:38:32.462669 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:32.462527 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17ea2397-450e-4505-9a48-0cf93f76065e-istio-token" pod "17ea2397-450e-4505-9a48-0cf93f76065e" (UID: "17ea2397-450e-4505-9a48-0cf93f76065e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:38:32.462669 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:32.462624 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17ea2397-450e-4505-9a48-0cf93f76065e-istio-envoy" pod "17ea2397-450e-4505-9a48-0cf93f76065e" (UID: "17ea2397-450e-4505-9a48-0cf93f76065e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:38:32.463308 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:32.463275 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17ea2397-450e-4505-9a48-0cf93f76065e-kube-api-access-n8qdc" pod "17ea2397-450e-4505-9a48-0cf93f76065e" (UID: "17ea2397-450e-4505-9a48-0cf93f76065e"). InnerVolumeSpecName "kube-api-access-n8qdc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:38:32.464165 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:32.463904 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/594f2d93-2f76-4632-904d-402fa126740c-kube-api-access-7pfm2" pod "594f2d93-2f76-4632-904d-402fa126740c" (UID: "594f2d93-2f76-4632-904d-402fa126740c"). InnerVolumeSpecName "kube-api-access-7pfm2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:38:32.464165 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:32.464082 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/594f2d93-2f76-4632-904d-402fa126740c-istio-envoy" pod "594f2d93-2f76-4632-904d-402fa126740c" (UID: "594f2d93-2f76-4632-904d-402fa126740c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:38:32.470914 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:32.470109 4376 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-mrx9s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 04:38:32.470914 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:32.470193 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mrx9s" podUID="70120ad8-48aa-44b4-b396-64eab56ce59b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 04:38:32.473305 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:32.473270 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngjkv\" (UniqueName: \"kubernetes.io/projected/7701398e-558d-48c9-8e08-1010af929cc7-kube-api-access-ngjkv\") pod \"test-curl-7108-0d3be9\" (UID: \"7701398e-558d-48c9-8e08-1010af929cc7\") " pod="opendatahub/test-curl-7108-0d3be9" Oct 31 04:38:32.519051 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:32.519024 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7108-0d3be9" Oct 31 04:38:32.551148 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:32.551116 4376 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/594f2d93-2f76-4632-904d-402fa126740c-istio-envoy\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:38:32.551148 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:32.551153 4376 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/594f2d93-2f76-4632-904d-402fa126740c-workload-socket\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:38:32.551424 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:32.551167 4376 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/594f2d93-2f76-4632-904d-402fa126740c-istio-token\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:38:32.551424 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:32.551180 4376 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17ea2397-450e-4505-9a48-0cf93f76065e-credential-socket\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:38:32.551424 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:32.551193 4376 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/594f2d93-2f76-4632-904d-402fa126740c-istio-podinfo\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:38:32.551424 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:32.551205 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7pfm2\" (UniqueName: \"kubernetes.io/projected/594f2d93-2f76-4632-904d-402fa126740c-kube-api-access-7pfm2\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:38:32.551424 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:32.551219 4376 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17ea2397-450e-4505-9a48-0cf93f76065e-istio-data\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:38:32.551424 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:32.551231 4376 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17ea2397-450e-4505-9a48-0cf93f76065e-istio-podinfo\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:38:32.551424 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:32.551242 4376 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17ea2397-450e-4505-9a48-0cf93f76065e-workload-certs\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:38:32.551424 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:32.551253 4376 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17ea2397-450e-4505-9a48-0cf93f76065e-istiod-ca-cert\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:38:32.551424 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:32.551265 4376 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/594f2d93-2f76-4632-904d-402fa126740c-credential-socket\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:38:32.551424 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:32.551277 4376 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/594f2d93-2f76-4632-904d-402fa126740c-workload-certs\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:38:32.551424 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:32.551291 4376 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17ea2397-450e-4505-9a48-0cf93f76065e-istio-envoy\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:38:32.551424 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:32.551305 4376 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/594f2d93-2f76-4632-904d-402fa126740c-istiod-ca-cert\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:38:32.551424 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:32.551318 4376 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17ea2397-450e-4505-9a48-0cf93f76065e-workload-socket\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:38:32.551424 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:32.551350 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8qdc\" (UniqueName: \"kubernetes.io/projected/17ea2397-450e-4505-9a48-0cf93f76065e-kube-api-access-n8qdc\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:38:32.551424 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:32.551362 4376 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17ea2397-450e-4505-9a48-0cf93f76065e-istio-token\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:38:32.551424 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:32.551375 4376 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/594f2d93-2f76-4632-904d-402fa126740c-istio-data\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:38:32.669445 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:32.669390 4376 generic.go:356] "Generic (PLEG): container finished" podID="ead2084a-70a7-47d2-a757-f340d971bfb6" containerID="c4227668546e211ff6d5f7c2783bec5fc415a777c6fe37cf575f22e947f32ba2" exitCode=0 Oct 31 04:38:32.669445 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:32.669439 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-rw9sz" Oct 31 04:38:32.669687 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:32.669525 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-rw9sz" event={"ID":"ead2084a-70a7-47d2-a757-f340d971bfb6","Type":"ContainerDied","Data":"c4227668546e211ff6d5f7c2783bec5fc415a777c6fe37cf575f22e947f32ba2"} Oct 31 04:38:32.669687 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:32.669609 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-rw9sz" event={"ID":"ead2084a-70a7-47d2-a757-f340d971bfb6","Type":"ContainerDied","Data":"d2f1ab09e3c9cf8f4282e5d066ba3e7eb6ae7c2c53d1e1d1bb69806df29f82d2"} Oct 31 04:38:32.669687 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:32.669639 4376 scope.go:122] "RemoveContainer" containerID="c4227668546e211ff6d5f7c2783bec5fc415a777c6fe37cf575f22e947f32ba2" Oct 31 04:38:32.671429 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:32.671079 4376 generic.go:356] "Generic (PLEG): container finished" podID="17ea2397-450e-4505-9a48-0cf93f76065e" containerID="ff9953f770bf9c4b20cfb7ca85ab92c25a709850cf99cd7c3ab370b9718fd576" exitCode=0 Oct 31 04:38:32.671429 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:32.671116 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-wq9cv" event={"ID":"17ea2397-450e-4505-9a48-0cf93f76065e","Type":"ContainerDied","Data":"ff9953f770bf9c4b20cfb7ca85ab92c25a709850cf99cd7c3ab370b9718fd576"} Oct 31 04:38:32.671429 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:32.671143 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-wq9cv" Oct 31 04:38:32.671429 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:32.671178 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-wq9cv" event={"ID":"17ea2397-450e-4505-9a48-0cf93f76065e","Type":"ContainerDied","Data":"eda7de36a1d9c178c94540398525c070e1ebea1992f5b73d330f3d423e1560af"} Oct 31 04:38:32.673065 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:32.673045 4376 generic.go:356] "Generic (PLEG): container finished" podID="594f2d93-2f76-4632-904d-402fa126740c" containerID="a907730c8fd50e968e3d15ba2fd87f6db56a4e6102701e117130f34be25d9b5d" exitCode=0 Oct 31 04:38:32.673185 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:32.673171 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bgh5f" Oct 31 04:38:32.673355 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:32.673305 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bgh5f" event={"ID":"594f2d93-2f76-4632-904d-402fa126740c","Type":"ContainerDied","Data":"a907730c8fd50e968e3d15ba2fd87f6db56a4e6102701e117130f34be25d9b5d"} Oct 31 04:38:32.673464 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:32.673368 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bgh5f" event={"ID":"594f2d93-2f76-4632-904d-402fa126740c","Type":"ContainerDied","Data":"d48c395637a943f586d76ae3be49b97d9e35e76bfe452fb4146ec72928e2a82e"} Oct 31 04:38:32.674876 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:32.674853 4376 generic.go:356] "Generic (PLEG): container finished" podID="6213a9f9-302f-4896-a1ce-dc0c7323082c" containerID="76596c78a36ea8bc117ade55c61089b261bb2d8c0405ac33a1c0324f5787eccb" exitCode=0 Oct 31 04:38:32.674992 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:32.674887 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-cjmdq" event={"ID":"6213a9f9-302f-4896-a1ce-dc0c7323082c","Type":"ContainerDied","Data":"76596c78a36ea8bc117ade55c61089b261bb2d8c0405ac33a1c0324f5787eccb"} Oct 31 04:38:32.674992 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:32.674905 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-cjmdq" Oct 31 04:38:32.674992 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:32.674913 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-cjmdq" event={"ID":"6213a9f9-302f-4896-a1ce-dc0c7323082c","Type":"ContainerDied","Data":"87e82dd28b2a85b773db740984da597bd9ddb9ac7cbdc2bf75078d00aadc9e66"} Oct 31 04:38:32.681411 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:32.681392 4376 scope.go:122] "RemoveContainer" containerID="c4227668546e211ff6d5f7c2783bec5fc415a777c6fe37cf575f22e947f32ba2" Oct 31 04:38:32.681715 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:38:32.681690 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4227668546e211ff6d5f7c2783bec5fc415a777c6fe37cf575f22e947f32ba2\": container with ID starting with c4227668546e211ff6d5f7c2783bec5fc415a777c6fe37cf575f22e947f32ba2 not found: ID does not exist" containerID="c4227668546e211ff6d5f7c2783bec5fc415a777c6fe37cf575f22e947f32ba2" Oct 31 04:38:32.681815 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:32.681720 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4227668546e211ff6d5f7c2783bec5fc415a777c6fe37cf575f22e947f32ba2"} err="failed to get container status \"c4227668546e211ff6d5f7c2783bec5fc415a777c6fe37cf575f22e947f32ba2\": rpc error: code = NotFound desc = could not find container \"c4227668546e211ff6d5f7c2783bec5fc415a777c6fe37cf575f22e947f32ba2\": container with ID starting with c4227668546e211ff6d5f7c2783bec5fc415a777c6fe37cf575f22e947f32ba2 not found: ID does not exist" Oct 31 04:38:32.681815 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:32.681741 4376 scope.go:122] "RemoveContainer" containerID="ff9953f770bf9c4b20cfb7ca85ab92c25a709850cf99cd7c3ab370b9718fd576" Oct 31 04:38:32.691858 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:32.691837 4376 scope.go:122] "RemoveContainer" containerID="ff9953f770bf9c4b20cfb7ca85ab92c25a709850cf99cd7c3ab370b9718fd576" Oct 31 04:38:32.692099 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:38:32.692083 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff9953f770bf9c4b20cfb7ca85ab92c25a709850cf99cd7c3ab370b9718fd576\": container with ID starting with ff9953f770bf9c4b20cfb7ca85ab92c25a709850cf99cd7c3ab370b9718fd576 not found: ID does not exist" containerID="ff9953f770bf9c4b20cfb7ca85ab92c25a709850cf99cd7c3ab370b9718fd576" Oct 31 04:38:32.692150 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:32.692105 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff9953f770bf9c4b20cfb7ca85ab92c25a709850cf99cd7c3ab370b9718fd576"} err="failed to get container status \"ff9953f770bf9c4b20cfb7ca85ab92c25a709850cf99cd7c3ab370b9718fd576\": rpc error: code = NotFound desc = could not find container \"ff9953f770bf9c4b20cfb7ca85ab92c25a709850cf99cd7c3ab370b9718fd576\": container with ID starting with ff9953f770bf9c4b20cfb7ca85ab92c25a709850cf99cd7c3ab370b9718fd576 not found: ID does not exist" Oct 31 04:38:32.692150 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:32.692120 4376 scope.go:122] "RemoveContainer" containerID="a907730c8fd50e968e3d15ba2fd87f6db56a4e6102701e117130f34be25d9b5d" Oct 31 04:38:32.700918 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:32.700906 4376 scope.go:122] "RemoveContainer" containerID="a907730c8fd50e968e3d15ba2fd87f6db56a4e6102701e117130f34be25d9b5d" Oct 31 04:38:32.701138 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:38:32.701119 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a907730c8fd50e968e3d15ba2fd87f6db56a4e6102701e117130f34be25d9b5d\": container with ID starting with a907730c8fd50e968e3d15ba2fd87f6db56a4e6102701e117130f34be25d9b5d not found: ID does not exist" containerID="a907730c8fd50e968e3d15ba2fd87f6db56a4e6102701e117130f34be25d9b5d" Oct 31 04:38:32.701189 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:32.701145 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a907730c8fd50e968e3d15ba2fd87f6db56a4e6102701e117130f34be25d9b5d"} err="failed to get container status \"a907730c8fd50e968e3d15ba2fd87f6db56a4e6102701e117130f34be25d9b5d\": rpc error: code = NotFound desc = could not find container \"a907730c8fd50e968e3d15ba2fd87f6db56a4e6102701e117130f34be25d9b5d\": container with ID starting with a907730c8fd50e968e3d15ba2fd87f6db56a4e6102701e117130f34be25d9b5d not found: ID does not exist" Oct 31 04:38:32.701189 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:32.701160 4376 scope.go:122] "RemoveContainer" containerID="76596c78a36ea8bc117ade55c61089b261bb2d8c0405ac33a1c0324f5787eccb" Oct 31 04:38:32.709716 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:32.709696 4376 scope.go:122] "RemoveContainer" containerID="76596c78a36ea8bc117ade55c61089b261bb2d8c0405ac33a1c0324f5787eccb" Oct 31 04:38:32.709967 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:38:32.709950 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76596c78a36ea8bc117ade55c61089b261bb2d8c0405ac33a1c0324f5787eccb\": container with ID starting with 76596c78a36ea8bc117ade55c61089b261bb2d8c0405ac33a1c0324f5787eccb not found: ID does not exist" containerID="76596c78a36ea8bc117ade55c61089b261bb2d8c0405ac33a1c0324f5787eccb" Oct 31 04:38:32.710023 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:32.709971 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76596c78a36ea8bc117ade55c61089b261bb2d8c0405ac33a1c0324f5787eccb"} err="failed to get container status \"76596c78a36ea8bc117ade55c61089b261bb2d8c0405ac33a1c0324f5787eccb\": rpc error: code = NotFound desc = could not find container \"76596c78a36ea8bc117ade55c61089b261bb2d8c0405ac33a1c0324f5787eccb\": container with ID starting with 76596c78a36ea8bc117ade55c61089b261bb2d8c0405ac33a1c0324f5787eccb not found: ID does not exist" Oct 31 04:38:32.725750 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:32.725732 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-wq9cv"] Oct 31 04:38:32.733274 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:32.733157 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-wq9cv"] Oct 31 04:38:32.764695 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:32.764667 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-rw9sz"] Oct 31 04:38:32.770618 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:32.770594 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-rw9sz"] Oct 31 04:38:32.804688 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:32.804664 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bgh5f"] Oct 31 04:38:32.808714 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:32.808691 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bgh5f"] Oct 31 04:38:32.830013 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:38:32.829987 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7701398e_558d_48c9_8e08_1010af929cc7.slice/crio-d543e67af38ceb9087e0668576ecf00fa53674be89937b13cd4356385cdbd5dd WatchSource:0}: Error finding container d543e67af38ceb9087e0668576ecf00fa53674be89937b13cd4356385cdbd5dd: Status 404 returned error can't find the container with id d543e67af38ceb9087e0668576ecf00fa53674be89937b13cd4356385cdbd5dd Oct 31 04:38:32.830904 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:32.830884 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7108-0d3be9"] Oct 31 04:38:32.844153 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:32.844119 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-cjmdq"] Oct 31 04:38:32.848271 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:32.848238 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-cjmdq"] Oct 31 04:38:33.203746 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:33.203709 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-crjjl" Oct 31 04:38:33.280740 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:33.280716 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mrx9s" Oct 31 04:38:33.373680 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:33.373646 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63486e50-4059-425f-b04a-4adea538d427-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63486e50-4059-425f-b04a-4adea538d427-credential-socket\") pod \"63486e50-4059-425f-b04a-4adea538d427\" (UID: \"63486e50-4059-425f-b04a-4adea538d427\") " Oct 31 04:38:33.373868 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:33.373698 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70120ad8-48aa-44b4-b396-64eab56ce59b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70120ad8-48aa-44b4-b396-64eab56ce59b-istio-envoy\") pod \"70120ad8-48aa-44b4-b396-64eab56ce59b\" (UID: \"70120ad8-48aa-44b4-b396-64eab56ce59b\") " Oct 31 04:38:33.373868 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:33.373738 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70120ad8-48aa-44b4-b396-64eab56ce59b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70120ad8-48aa-44b4-b396-64eab56ce59b-istiod-ca-cert\") pod \"70120ad8-48aa-44b4-b396-64eab56ce59b\" (UID: \"70120ad8-48aa-44b4-b396-64eab56ce59b\") " Oct 31 04:38:33.373868 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:33.373791 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70120ad8-48aa-44b4-b396-64eab56ce59b-istio-token\" (UniqueName: \"kubernetes.io/projected/70120ad8-48aa-44b4-b396-64eab56ce59b-istio-token\") pod \"70120ad8-48aa-44b4-b396-64eab56ce59b\" (UID: \"70120ad8-48aa-44b4-b396-64eab56ce59b\") " Oct 31 04:38:33.374050 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:33.373875 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63486e50-4059-425f-b04a-4adea538d427-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63486e50-4059-425f-b04a-4adea538d427-workload-socket\") pod \"63486e50-4059-425f-b04a-4adea538d427\" (UID: \"63486e50-4059-425f-b04a-4adea538d427\") " Oct 31 04:38:33.374050 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:33.373935 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63486e50-4059-425f-b04a-4adea538d427-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63486e50-4059-425f-b04a-4adea538d427-istio-data\") pod \"63486e50-4059-425f-b04a-4adea538d427\" (UID: \"63486e50-4059-425f-b04a-4adea538d427\") " Oct 31 04:38:33.374050 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:33.373945 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63486e50-4059-425f-b04a-4adea538d427-credential-socket" pod "63486e50-4059-425f-b04a-4adea538d427" (UID: "63486e50-4059-425f-b04a-4adea538d427"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:38:33.374050 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:33.373963 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70120ad8-48aa-44b4-b396-64eab56ce59b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70120ad8-48aa-44b4-b396-64eab56ce59b-workload-certs\") pod \"70120ad8-48aa-44b4-b396-64eab56ce59b\" (UID: \"70120ad8-48aa-44b4-b396-64eab56ce59b\") " Oct 31 04:38:33.374050 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:33.373995 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/70120ad8-48aa-44b4-b396-64eab56ce59b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70120ad8-48aa-44b4-b396-64eab56ce59b-istio-podinfo\") pod \"70120ad8-48aa-44b4-b396-64eab56ce59b\" (UID: \"70120ad8-48aa-44b4-b396-64eab56ce59b\") " Oct 31 04:38:33.374050 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:33.374037 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70120ad8-48aa-44b4-b396-64eab56ce59b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70120ad8-48aa-44b4-b396-64eab56ce59b-credential-socket\") pod \"70120ad8-48aa-44b4-b396-64eab56ce59b\" (UID: \"70120ad8-48aa-44b4-b396-64eab56ce59b\") " Oct 31 04:38:33.374389 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:33.374114 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70120ad8-48aa-44b4-b396-64eab56ce59b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70120ad8-48aa-44b4-b396-64eab56ce59b-workload-socket\") pod \"70120ad8-48aa-44b4-b396-64eab56ce59b\" (UID: \"70120ad8-48aa-44b4-b396-64eab56ce59b\") " Oct 31 04:38:33.374389 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:33.374165 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63486e50-4059-425f-b04a-4adea538d427-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63486e50-4059-425f-b04a-4adea538d427-workload-certs\") pod \"63486e50-4059-425f-b04a-4adea538d427\" (UID: \"63486e50-4059-425f-b04a-4adea538d427\") " Oct 31 04:38:33.374389 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:33.374255 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70120ad8-48aa-44b4-b396-64eab56ce59b-workload-certs" pod "70120ad8-48aa-44b4-b396-64eab56ce59b" (UID: "70120ad8-48aa-44b4-b396-64eab56ce59b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:38:33.374562 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:33.374514 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63486e50-4059-425f-b04a-4adea538d427-workload-certs" pod "63486e50-4059-425f-b04a-4adea538d427" (UID: "63486e50-4059-425f-b04a-4adea538d427"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:38:33.374562 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:33.374528 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63486e50-4059-425f-b04a-4adea538d427-workload-socket" pod "63486e50-4059-425f-b04a-4adea538d427" (UID: "63486e50-4059-425f-b04a-4adea538d427"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:38:33.374671 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:33.374592 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63486e50-4059-425f-b04a-4adea538d427-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63486e50-4059-425f-b04a-4adea538d427-istiod-ca-cert\") pod \"63486e50-4059-425f-b04a-4adea538d427\" (UID: \"63486e50-4059-425f-b04a-4adea538d427\") " Oct 31 04:38:33.374671 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:33.374614 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70120ad8-48aa-44b4-b396-64eab56ce59b-workload-socket" pod "70120ad8-48aa-44b4-b396-64eab56ce59b" (UID: "70120ad8-48aa-44b4-b396-64eab56ce59b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:38:33.374671 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:33.374629 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70120ad8-48aa-44b4-b396-64eab56ce59b-credential-socket" pod "70120ad8-48aa-44b4-b396-64eab56ce59b" (UID: "70120ad8-48aa-44b4-b396-64eab56ce59b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:38:33.374829 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:33.374695 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63486e50-4059-425f-b04a-4adea538d427-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63486e50-4059-425f-b04a-4adea538d427-istio-envoy\") pod \"63486e50-4059-425f-b04a-4adea538d427\" (UID: \"63486e50-4059-425f-b04a-4adea538d427\") " Oct 31 04:38:33.374829 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:33.374729 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70120ad8-48aa-44b4-b396-64eab56ce59b-kube-api-access-546rh\" (UniqueName: \"kubernetes.io/projected/70120ad8-48aa-44b4-b396-64eab56ce59b-kube-api-access-546rh\") pod \"70120ad8-48aa-44b4-b396-64eab56ce59b\" (UID: \"70120ad8-48aa-44b4-b396-64eab56ce59b\") " Oct 31 04:38:33.374829 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:33.374776 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/63486e50-4059-425f-b04a-4adea538d427-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63486e50-4059-425f-b04a-4adea538d427-istio-podinfo\") pod \"63486e50-4059-425f-b04a-4adea538d427\" (UID: \"63486e50-4059-425f-b04a-4adea538d427\") " Oct 31 04:38:33.374829 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:33.374818 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63486e50-4059-425f-b04a-4adea538d427-kube-api-access-nvgkj\" (UniqueName: \"kubernetes.io/projected/63486e50-4059-425f-b04a-4adea538d427-kube-api-access-nvgkj\") pod \"63486e50-4059-425f-b04a-4adea538d427\" (UID: \"63486e50-4059-425f-b04a-4adea538d427\") " Oct 31 04:38:33.375045 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:33.374887 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63486e50-4059-425f-b04a-4adea538d427-istio-token\" (UniqueName: \"kubernetes.io/projected/63486e50-4059-425f-b04a-4adea538d427-istio-token\") pod \"63486e50-4059-425f-b04a-4adea538d427\" (UID: \"63486e50-4059-425f-b04a-4adea538d427\") " Oct 31 04:38:33.375045 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:33.374919 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70120ad8-48aa-44b4-b396-64eab56ce59b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70120ad8-48aa-44b4-b396-64eab56ce59b-istio-data\") pod \"70120ad8-48aa-44b4-b396-64eab56ce59b\" (UID: \"70120ad8-48aa-44b4-b396-64eab56ce59b\") " Oct 31 04:38:33.375045 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:33.374988 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63486e50-4059-425f-b04a-4adea538d427-istiod-ca-cert" pod "63486e50-4059-425f-b04a-4adea538d427" (UID: "63486e50-4059-425f-b04a-4adea538d427"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:38:33.375207 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:33.374811 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70120ad8-48aa-44b4-b396-64eab56ce59b-istiod-ca-cert" pod "70120ad8-48aa-44b4-b396-64eab56ce59b" (UID: "70120ad8-48aa-44b4-b396-64eab56ce59b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:38:33.375410 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:33.375387 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70120ad8-48aa-44b4-b396-64eab56ce59b-istio-data" pod "70120ad8-48aa-44b4-b396-64eab56ce59b" (UID: "70120ad8-48aa-44b4-b396-64eab56ce59b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:38:33.375722 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:33.375697 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63486e50-4059-425f-b04a-4adea538d427-istio-data" pod "63486e50-4059-425f-b04a-4adea538d427" (UID: "63486e50-4059-425f-b04a-4adea538d427"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:38:33.375815 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:33.375724 4376 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70120ad8-48aa-44b4-b396-64eab56ce59b-istio-data\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:38:33.375815 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:33.375747 4376 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63486e50-4059-425f-b04a-4adea538d427-credential-socket\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:38:33.375815 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:33.375762 4376 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70120ad8-48aa-44b4-b396-64eab56ce59b-istiod-ca-cert\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:38:33.375815 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:33.375777 4376 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63486e50-4059-425f-b04a-4adea538d427-workload-socket\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:38:33.375815 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:33.375791 4376 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70120ad8-48aa-44b4-b396-64eab56ce59b-workload-certs\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:38:33.375815 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:33.375805 4376 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70120ad8-48aa-44b4-b396-64eab56ce59b-credential-socket\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:38:33.376127 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:33.375821 4376 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70120ad8-48aa-44b4-b396-64eab56ce59b-workload-socket\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:38:33.376127 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:33.375835 4376 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63486e50-4059-425f-b04a-4adea538d427-workload-certs\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:38:33.376127 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:33.375853 4376 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63486e50-4059-425f-b04a-4adea538d427-istiod-ca-cert\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:38:33.377143 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:33.377117 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/70120ad8-48aa-44b4-b396-64eab56ce59b-istio-podinfo" pod "70120ad8-48aa-44b4-b396-64eab56ce59b" (UID: "70120ad8-48aa-44b4-b396-64eab56ce59b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 04:38:33.377639 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:33.377613 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70120ad8-48aa-44b4-b396-64eab56ce59b-istio-token" pod "70120ad8-48aa-44b4-b396-64eab56ce59b" (UID: "70120ad8-48aa-44b4-b396-64eab56ce59b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:38:33.378351 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:33.378296 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70120ad8-48aa-44b4-b396-64eab56ce59b-istio-envoy" pod "70120ad8-48aa-44b4-b396-64eab56ce59b" (UID: "70120ad8-48aa-44b4-b396-64eab56ce59b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:38:33.378824 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:33.378796 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63486e50-4059-425f-b04a-4adea538d427-kube-api-access-nvgkj" pod "63486e50-4059-425f-b04a-4adea538d427" (UID: "63486e50-4059-425f-b04a-4adea538d427"). InnerVolumeSpecName "kube-api-access-nvgkj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:38:33.379441 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:33.379322 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63486e50-4059-425f-b04a-4adea538d427-istio-envoy" pod "63486e50-4059-425f-b04a-4adea538d427" (UID: "63486e50-4059-425f-b04a-4adea538d427"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:38:33.379600 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:33.379579 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63486e50-4059-425f-b04a-4adea538d427-istio-token" pod "63486e50-4059-425f-b04a-4adea538d427" (UID: "63486e50-4059-425f-b04a-4adea538d427"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:38:33.379929 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:33.379900 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70120ad8-48aa-44b4-b396-64eab56ce59b-kube-api-access-546rh" pod "70120ad8-48aa-44b4-b396-64eab56ce59b" (UID: "70120ad8-48aa-44b4-b396-64eab56ce59b"). InnerVolumeSpecName "kube-api-access-546rh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:38:33.380086 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:33.380072 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/63486e50-4059-425f-b04a-4adea538d427-istio-podinfo" pod "63486e50-4059-425f-b04a-4adea538d427" (UID: "63486e50-4059-425f-b04a-4adea538d427"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 04:38:33.476938 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:33.476904 4376 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70120ad8-48aa-44b4-b396-64eab56ce59b-istio-token\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:38:33.476938 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:33.476931 4376 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63486e50-4059-425f-b04a-4adea538d427-istio-data\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:38:33.476938 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:33.476940 4376 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70120ad8-48aa-44b4-b396-64eab56ce59b-istio-podinfo\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:38:33.476938 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:33.476948 4376 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63486e50-4059-425f-b04a-4adea538d427-istio-envoy\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:38:33.477255 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:33.476957 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-546rh\" (UniqueName: \"kubernetes.io/projected/70120ad8-48aa-44b4-b396-64eab56ce59b-kube-api-access-546rh\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:38:33.477255 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:33.476966 4376 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63486e50-4059-425f-b04a-4adea538d427-istio-podinfo\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:38:33.477255 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:33.476975 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvgkj\" (UniqueName: \"kubernetes.io/projected/63486e50-4059-425f-b04a-4adea538d427-kube-api-access-nvgkj\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:38:33.477255 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:33.476990 4376 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63486e50-4059-425f-b04a-4adea538d427-istio-token\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:38:33.477255 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:33.477001 4376 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70120ad8-48aa-44b4-b396-64eab56ce59b-istio-envoy\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:38:33.683382 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:33.683340 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7108-0d3be9" event={"ID":"7701398e-558d-48c9-8e08-1010af929cc7","Type":"ContainerStarted","Data":"d7f06af13f6fe9b1167f97eae8ad9f3328a03e80cf8722b8f1f2975b50b382f0"} Oct 31 04:38:33.683561 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:33.683393 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7108-0d3be9" event={"ID":"7701398e-558d-48c9-8e08-1010af929cc7","Type":"ContainerStarted","Data":"d543e67af38ceb9087e0668576ecf00fa53674be89937b13cd4356385cdbd5dd"} Oct 31 04:38:33.684688 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:33.684661 4376 generic.go:356] "Generic (PLEG): container finished" podID="63486e50-4059-425f-b04a-4adea538d427" containerID="ed822ccdccebd79101c8df29b99ac5823650b406625a22bb6e931795f33bada2" exitCode=0 Oct 31 04:38:33.684809 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:33.684701 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-crjjl" Oct 31 04:38:33.684809 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:33.684747 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-crjjl" event={"ID":"63486e50-4059-425f-b04a-4adea538d427","Type":"ContainerDied","Data":"ed822ccdccebd79101c8df29b99ac5823650b406625a22bb6e931795f33bada2"} Oct 31 04:38:33.684809 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:33.684801 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-crjjl" event={"ID":"63486e50-4059-425f-b04a-4adea538d427","Type":"ContainerDied","Data":"b5499f9070ec575618ec370fad01787adbbb585085137ae1ff2d5b0951e65cee"} Oct 31 04:38:33.684947 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:33.684830 4376 scope.go:122] "RemoveContainer" containerID="ed822ccdccebd79101c8df29b99ac5823650b406625a22bb6e931795f33bada2" Oct 31 04:38:33.690643 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:33.690620 4376 generic.go:356] "Generic (PLEG): container finished" podID="70120ad8-48aa-44b4-b396-64eab56ce59b" containerID="1dea307cb0f07118a743766d464d20933d0bf91533edea2b93501e78a8f198ca" exitCode=0 Oct 31 04:38:33.690786 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:33.690672 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mrx9s" Oct 31 04:38:33.690786 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:33.690707 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mrx9s" event={"ID":"70120ad8-48aa-44b4-b396-64eab56ce59b","Type":"ContainerDied","Data":"1dea307cb0f07118a743766d464d20933d0bf91533edea2b93501e78a8f198ca"} Oct 31 04:38:33.690786 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:33.690753 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mrx9s" event={"ID":"70120ad8-48aa-44b4-b396-64eab56ce59b","Type":"ContainerDied","Data":"52baa228b292f8319077488cc313e975602e3df8af32592d3e27433971eeab2d"} Oct 31 04:38:33.700031 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:33.700015 4376 scope.go:122] "RemoveContainer" containerID="ed822ccdccebd79101c8df29b99ac5823650b406625a22bb6e931795f33bada2" Oct 31 04:38:33.700295 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:38:33.700275 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed822ccdccebd79101c8df29b99ac5823650b406625a22bb6e931795f33bada2\": container with ID starting with ed822ccdccebd79101c8df29b99ac5823650b406625a22bb6e931795f33bada2 not found: ID does not exist" containerID="ed822ccdccebd79101c8df29b99ac5823650b406625a22bb6e931795f33bada2" Oct 31 04:38:33.700407 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:33.700303 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed822ccdccebd79101c8df29b99ac5823650b406625a22bb6e931795f33bada2"} err="failed to get container status \"ed822ccdccebd79101c8df29b99ac5823650b406625a22bb6e931795f33bada2\": rpc error: code = NotFound desc = could not find container \"ed822ccdccebd79101c8df29b99ac5823650b406625a22bb6e931795f33bada2\": container with ID starting with ed822ccdccebd79101c8df29b99ac5823650b406625a22bb6e931795f33bada2 not found: ID does not exist" Oct 31 04:38:33.700407 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:33.700320 4376 scope.go:122] "RemoveContainer" containerID="1dea307cb0f07118a743766d464d20933d0bf91533edea2b93501e78a8f198ca" Oct 31 04:38:33.709648 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:33.709633 4376 scope.go:122] "RemoveContainer" containerID="1dea307cb0f07118a743766d464d20933d0bf91533edea2b93501e78a8f198ca" Oct 31 04:38:33.709912 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:38:33.709895 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1dea307cb0f07118a743766d464d20933d0bf91533edea2b93501e78a8f198ca\": container with ID starting with 1dea307cb0f07118a743766d464d20933d0bf91533edea2b93501e78a8f198ca not found: ID does not exist" containerID="1dea307cb0f07118a743766d464d20933d0bf91533edea2b93501e78a8f198ca" Oct 31 04:38:33.709953 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:33.709918 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1dea307cb0f07118a743766d464d20933d0bf91533edea2b93501e78a8f198ca"} err="failed to get container status \"1dea307cb0f07118a743766d464d20933d0bf91533edea2b93501e78a8f198ca\": rpc error: code = NotFound desc = could not find container \"1dea307cb0f07118a743766d464d20933d0bf91533edea2b93501e78a8f198ca\": container with ID starting with 1dea307cb0f07118a743766d464d20933d0bf91533edea2b93501e78a8f198ca not found: ID does not exist" Oct 31 04:38:33.722948 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:33.722912 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7108-0d3be9" podStartSLOduration=1.722901604 podStartE2EDuration="1.722901604s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:38:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:38:33.721225171 +0000 UTC m=+2061.435559788" watchObservedRunningTime="2026-10-31 04:38:33.722901604 +0000 UTC m=+2061.437236220" Oct 31 04:38:33.762528 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:33.762497 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-crjjl"] Oct 31 04:38:33.766848 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:33.766822 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-crjjl"] Oct 31 04:38:33.799608 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:33.799579 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mrx9s"] Oct 31 04:38:33.807105 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:33.807077 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mrx9s"] Oct 31 04:38:34.444989 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:34.444957 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17ea2397-450e-4505-9a48-0cf93f76065e" path="/var/lib/kubelet/pods/17ea2397-450e-4505-9a48-0cf93f76065e/volumes" Oct 31 04:38:34.445500 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:34.445486 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="594f2d93-2f76-4632-904d-402fa126740c" path="/var/lib/kubelet/pods/594f2d93-2f76-4632-904d-402fa126740c/volumes" Oct 31 04:38:34.445904 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:34.445891 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6213a9f9-302f-4896-a1ce-dc0c7323082c" path="/var/lib/kubelet/pods/6213a9f9-302f-4896-a1ce-dc0c7323082c/volumes" Oct 31 04:38:34.446720 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:34.446697 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63486e50-4059-425f-b04a-4adea538d427" path="/var/lib/kubelet/pods/63486e50-4059-425f-b04a-4adea538d427/volumes" Oct 31 04:38:34.447148 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:34.447131 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70120ad8-48aa-44b4-b396-64eab56ce59b" path="/var/lib/kubelet/pods/70120ad8-48aa-44b4-b396-64eab56ce59b/volumes" Oct 31 04:38:34.447568 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:34.447553 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ead2084a-70a7-47d2-a757-f340d971bfb6" path="/var/lib/kubelet/pods/ead2084a-70a7-47d2-a757-f340d971bfb6/volumes" Oct 31 04:38:34.659914 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:34.659877 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7108-0d3be9"] Oct 31 04:38:34.666383 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:34.666351 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7108-0d3be9"] Oct 31 04:38:34.826274 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:34.826198 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7108-e075d1"] Oct 31 04:38:34.826841 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:34.826827 4376 cpu_manager.go:414] "containerMap: removing container" podUID="6213a9f9-302f-4896-a1ce-dc0c7323082c" containerName="istio-proxy" Oct 31 04:38:34.826882 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:34.826844 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6213a9f9-302f-4896-a1ce-dc0c7323082c" containerName="istio-proxy" Oct 31 04:38:34.826882 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:34.826863 4376 cpu_manager.go:414] "containerMap: removing container" podUID="17ea2397-450e-4505-9a48-0cf93f76065e" containerName="istio-proxy" Oct 31 04:38:34.826882 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:34.826869 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17ea2397-450e-4505-9a48-0cf93f76065e" containerName="istio-proxy" Oct 31 04:38:34.826882 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:34.826879 4376 cpu_manager.go:414] "containerMap: removing container" podUID="594f2d93-2f76-4632-904d-402fa126740c" containerName="istio-proxy" Oct 31 04:38:34.827006 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:34.826884 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="594f2d93-2f76-4632-904d-402fa126740c" containerName="istio-proxy" Oct 31 04:38:34.827006 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:34.826892 4376 cpu_manager.go:414] "containerMap: removing container" podUID="ead2084a-70a7-47d2-a757-f340d971bfb6" containerName="istio-proxy" Oct 31 04:38:34.827006 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:34.826897 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ead2084a-70a7-47d2-a757-f340d971bfb6" containerName="istio-proxy" Oct 31 04:38:34.827006 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:34.826905 4376 cpu_manager.go:414] "containerMap: removing container" podUID="70120ad8-48aa-44b4-b396-64eab56ce59b" containerName="istio-proxy" Oct 31 04:38:34.827006 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:34.826909 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70120ad8-48aa-44b4-b396-64eab56ce59b" containerName="istio-proxy" Oct 31 04:38:34.827006 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:34.826922 4376 cpu_manager.go:414] "containerMap: removing container" podUID="7701398e-558d-48c9-8e08-1010af929cc7" containerName="test-curl-7108-0d3be9" Oct 31 04:38:34.827006 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:34.826929 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7701398e-558d-48c9-8e08-1010af929cc7" containerName="test-curl-7108-0d3be9" Oct 31 04:38:34.827006 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:34.826940 4376 cpu_manager.go:414] "containerMap: removing container" podUID="63486e50-4059-425f-b04a-4adea538d427" containerName="istio-proxy" Oct 31 04:38:34.827006 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:34.826944 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63486e50-4059-425f-b04a-4adea538d427" containerName="istio-proxy" Oct 31 04:38:34.827273 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:34.827037 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="17ea2397-450e-4505-9a48-0cf93f76065e" containerName="istio-proxy" Oct 31 04:38:34.827273 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:34.827045 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="594f2d93-2f76-4632-904d-402fa126740c" containerName="istio-proxy" Oct 31 04:38:34.827273 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:34.827051 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="ead2084a-70a7-47d2-a757-f340d971bfb6" containerName="istio-proxy" Oct 31 04:38:34.827273 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:34.827057 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="70120ad8-48aa-44b4-b396-64eab56ce59b" containerName="istio-proxy" Oct 31 04:38:34.827273 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:34.827064 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="7701398e-558d-48c9-8e08-1010af929cc7" containerName="test-curl-7108-0d3be9" Oct 31 04:38:34.827273 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:34.827072 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="63486e50-4059-425f-b04a-4adea538d427" containerName="istio-proxy" Oct 31 04:38:34.827273 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:34.827081 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="6213a9f9-302f-4896-a1ce-dc0c7323082c" containerName="istio-proxy" Oct 31 04:38:34.832298 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:34.832281 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7108-e075d1" Oct 31 04:38:34.849200 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:34.849177 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7108-e075d1"] Oct 31 04:38:34.997529 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:34.997501 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lrqq\" (UniqueName: \"kubernetes.io/projected/bb1c5477-0a48-4118-98e1-4127357fd27c-kube-api-access-7lrqq\") pod \"test-curl-7108-e075d1\" (UID: \"bb1c5477-0a48-4118-98e1-4127357fd27c\") " pod="opendatahub/test-curl-7108-e075d1" Oct 31 04:38:35.098498 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:35.098405 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lrqq\" (UniqueName: \"kubernetes.io/projected/bb1c5477-0a48-4118-98e1-4127357fd27c-kube-api-access-7lrqq\") pod \"test-curl-7108-e075d1\" (UID: \"bb1c5477-0a48-4118-98e1-4127357fd27c\") " pod="opendatahub/test-curl-7108-e075d1" Oct 31 04:38:35.125657 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:35.125622 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lrqq\" (UniqueName: \"kubernetes.io/projected/bb1c5477-0a48-4118-98e1-4127357fd27c-kube-api-access-7lrqq\") pod \"test-curl-7108-e075d1\" (UID: \"bb1c5477-0a48-4118-98e1-4127357fd27c\") " pod="opendatahub/test-curl-7108-e075d1" Oct 31 04:38:35.145309 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:35.145272 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7108-e075d1" Oct 31 04:38:35.455772 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:35.455731 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7108-e075d1"] Oct 31 04:38:35.460230 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:38:35.460200 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbb1c5477_0a48_4118_98e1_4127357fd27c.slice/crio-116cc0d71d86b13014cc37b24e3783bd4b04c011e711ad59ad17ecd39dce477b WatchSource:0}: Error finding container 116cc0d71d86b13014cc37b24e3783bd4b04c011e711ad59ad17ecd39dce477b: Status 404 returned error can't find the container with id 116cc0d71d86b13014cc37b24e3783bd4b04c011e711ad59ad17ecd39dce477b Oct 31 04:38:35.762808 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:35.762703 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7108-0d3be9" podUID="7701398e-558d-48c9-8e08-1010af929cc7" containerName="test-curl-7108-0d3be9" containerID="cri-o://d7f06af13f6fe9b1167f97eae8ad9f3328a03e80cf8722b8f1f2975b50b382f0" gracePeriod=2 Oct 31 04:38:35.767354 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:35.763792 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7108-e075d1" event={"ID":"bb1c5477-0a48-4118-98e1-4127357fd27c","Type":"ContainerStarted","Data":"eb930bed74408ea070d12fac330fdddb87dd4de312aea79b14e8f5448b7d30bb"} Oct 31 04:38:35.767354 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:35.763835 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7108-e075d1" event={"ID":"bb1c5477-0a48-4118-98e1-4127357fd27c","Type":"ContainerStarted","Data":"116cc0d71d86b13014cc37b24e3783bd4b04c011e711ad59ad17ecd39dce477b"} Oct 31 04:38:35.808279 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:35.808222 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7108-e075d1" podStartSLOduration=1.808206902 podStartE2EDuration="1.808206902s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:38:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:38:35.805022366 +0000 UTC m=+2063.519357012" watchObservedRunningTime="2026-10-31 04:38:35.808206902 +0000 UTC m=+2063.522541519" Oct 31 04:38:35.818799 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:38:35.818760 4376 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7108-0d3be9\" is forbidden: User \"system:node:ip-10-0-134-155.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-155.ec2.internal' and this object" podUID="7701398e-558d-48c9-8e08-1010af929cc7" pod="opendatahub/test-curl-7108-0d3be9" Oct 31 04:38:35.832673 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:38:35.832634 4376 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7108-0d3be9\" is forbidden: User \"system:node:ip-10-0-134-155.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-155.ec2.internal' and this object" podUID="7701398e-558d-48c9-8e08-1010af929cc7" pod="opendatahub/test-curl-7108-0d3be9" Oct 31 04:38:36.704173 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:36.704138 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7108-e075d1"] Oct 31 04:38:36.709275 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:36.709245 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7108-e075d1"] Oct 31 04:38:36.870768 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:36.870738 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7108-535368"] Oct 31 04:38:36.871394 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:36.871379 4376 cpu_manager.go:414] "containerMap: removing container" podUID="bb1c5477-0a48-4118-98e1-4127357fd27c" containerName="test-curl-7108-e075d1" Oct 31 04:38:36.871457 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:36.871395 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb1c5477-0a48-4118-98e1-4127357fd27c" containerName="test-curl-7108-e075d1" Oct 31 04:38:36.871495 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:36.871484 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb1c5477-0a48-4118-98e1-4127357fd27c" containerName="test-curl-7108-e075d1" Oct 31 04:38:36.875019 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:36.875002 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7108-535368" Oct 31 04:38:36.891760 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:36.891729 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7108-535368"] Oct 31 04:38:36.929829 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:36.929793 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-nggns"] Oct 31 04:38:36.930024 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:36.929987 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-nggns" podUID="8d001c91-1967-4fb1-94f4-8f8a8db7216b" containerName="istio-proxy" containerID="cri-o://6896af4706d0f2de28de8772f390946c6406e8b80f07603c9ccc5934a88c144f" gracePeriod=30 Oct 31 04:38:36.941426 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:36.941313 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g82l6\" (UniqueName: \"kubernetes.io/projected/ae170649-4d7e-4863-b994-6ae750b1ab43-kube-api-access-g82l6\") pod \"test-curl-7108-535368\" (UID: \"ae170649-4d7e-4863-b994-6ae750b1ab43\") " pod="opendatahub/test-curl-7108-535368" Oct 31 04:38:37.044845 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:37.044684 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g82l6\" (UniqueName: \"kubernetes.io/projected/ae170649-4d7e-4863-b994-6ae750b1ab43-kube-api-access-g82l6\") pod \"test-curl-7108-535368\" (UID: \"ae170649-4d7e-4863-b994-6ae750b1ab43\") " pod="opendatahub/test-curl-7108-535368" Oct 31 04:38:37.068493 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:37.068460 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g82l6\" (UniqueName: \"kubernetes.io/projected/ae170649-4d7e-4863-b994-6ae750b1ab43-kube-api-access-g82l6\") pod \"test-curl-7108-535368\" (UID: \"ae170649-4d7e-4863-b994-6ae750b1ab43\") " pod="opendatahub/test-curl-7108-535368" Oct 31 04:38:37.187688 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:37.187653 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7108-535368" Oct 31 04:38:37.491089 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:38:37.491053 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podae170649_4d7e_4863_b994_6ae750b1ab43.slice/crio-e600be2ece31ca9ba6b97987613910280c8e098685661407753362d957f29114 WatchSource:0}: Error finding container e600be2ece31ca9ba6b97987613910280c8e098685661407753362d957f29114: Status 404 returned error can't find the container with id e600be2ece31ca9ba6b97987613910280c8e098685661407753362d957f29114 Oct 31 04:38:37.491659 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:37.491598 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7108-535368"] Oct 31 04:38:37.784906 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:37.784860 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7108-e075d1" podUID="bb1c5477-0a48-4118-98e1-4127357fd27c" containerName="test-curl-7108-e075d1" containerID="cri-o://eb930bed74408ea070d12fac330fdddb87dd4de312aea79b14e8f5448b7d30bb" gracePeriod=2 Oct 31 04:38:37.785775 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:37.785579 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7108-535368" event={"ID":"ae170649-4d7e-4863-b994-6ae750b1ab43","Type":"ContainerStarted","Data":"b1636b3408284eaa61e52a7856c46c9bd2844ecaa729927cebb038743793df0f"} Oct 31 04:38:37.785775 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:37.785627 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7108-535368" event={"ID":"ae170649-4d7e-4863-b994-6ae750b1ab43","Type":"ContainerStarted","Data":"e600be2ece31ca9ba6b97987613910280c8e098685661407753362d957f29114"} Oct 31 04:38:37.799403 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:38:37.799366 4376 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7108-e075d1\" is forbidden: User \"system:node:ip-10-0-134-155.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-155.ec2.internal' and this object" podUID="bb1c5477-0a48-4118-98e1-4127357fd27c" pod="opendatahub/test-curl-7108-e075d1" Oct 31 04:38:37.813407 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:38:37.813372 4376 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7108-e075d1\" is forbidden: User \"system:node:ip-10-0-134-155.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-155.ec2.internal' and this object" podUID="bb1c5477-0a48-4118-98e1-4127357fd27c" pod="opendatahub/test-curl-7108-e075d1" Oct 31 04:38:37.969566 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:37.969534 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7108-0d3be9" Oct 31 04:38:37.984162 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:38:37.984127 4376 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7108-e075d1\" is forbidden: User \"system:node:ip-10-0-134-155.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-155.ec2.internal' and this object" podUID="bb1c5477-0a48-4118-98e1-4127357fd27c" pod="opendatahub/test-curl-7108-e075d1" Oct 31 04:38:38.067552 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:38.067308 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7701398e-558d-48c9-8e08-1010af929cc7-kube-api-access-ngjkv\" (UniqueName: \"kubernetes.io/projected/7701398e-558d-48c9-8e08-1010af929cc7-kube-api-access-ngjkv\") pod \"7701398e-558d-48c9-8e08-1010af929cc7\" (UID: \"7701398e-558d-48c9-8e08-1010af929cc7\") " Oct 31 04:38:38.070995 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:38.070965 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7701398e-558d-48c9-8e08-1010af929cc7-kube-api-access-ngjkv" pod "7701398e-558d-48c9-8e08-1010af929cc7" (UID: "7701398e-558d-48c9-8e08-1010af929cc7"). InnerVolumeSpecName "kube-api-access-ngjkv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:38:38.170197 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:38.170160 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ngjkv\" (UniqueName: \"kubernetes.io/projected/7701398e-558d-48c9-8e08-1010af929cc7-kube-api-access-ngjkv\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:38:38.444277 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:38.444244 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7701398e-558d-48c9-8e08-1010af929cc7" path="/var/lib/kubelet/pods/7701398e-558d-48c9-8e08-1010af929cc7/volumes" Oct 31 04:38:38.571775 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:38.571737 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w4-57dc6d/auth-test-model-57dc6d-kserve-577cb5ffd5-2wk4v" podUID="1b9cb676-3050-4ea5-b38b-3d32d10ae997" containerName="main" containerID="cri-o://4c4ba863c4a2081f438ce295c6954539ad3cf932c40a67ddc686eb7138f22386" gracePeriod=45 Oct 31 04:38:38.773954 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:38.773900 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7108-535368" podStartSLOduration=2.773885366 podStartE2EDuration="2.773885366s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:38:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:38:37.853398768 +0000 UTC m=+2065.567733404" watchObservedRunningTime="2026-10-31 04:38:38.773885366 +0000 UTC m=+2066.488219983" Oct 31 04:38:38.777999 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:38.777160 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7108-535368"] Oct 31 04:38:38.790362 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:38.788479 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7108-535368"] Oct 31 04:38:38.796459 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:38.796361 4376 generic.go:356] "Generic (PLEG): container finished" podID="7701398e-558d-48c9-8e08-1010af929cc7" containerID="d7f06af13f6fe9b1167f97eae8ad9f3328a03e80cf8722b8f1f2975b50b382f0" exitCode=137 Oct 31 04:38:38.796459 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:38.796433 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7108-0d3be9" Oct 31 04:38:38.797558 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:38.797519 4376 scope.go:122] "RemoveContainer" containerID="d7f06af13f6fe9b1167f97eae8ad9f3328a03e80cf8722b8f1f2975b50b382f0" Oct 31 04:38:38.800768 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:38.800730 4376 generic.go:356] "Generic (PLEG): container finished" podID="1b9cb676-3050-4ea5-b38b-3d32d10ae997" containerID="4c4ba863c4a2081f438ce295c6954539ad3cf932c40a67ddc686eb7138f22386" exitCode=0 Oct 31 04:38:38.801040 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:38.800820 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-57dc6d/auth-test-model-57dc6d-kserve-577cb5ffd5-2wk4v" event={"ID":"1b9cb676-3050-4ea5-b38b-3d32d10ae997","Type":"ContainerDied","Data":"4c4ba863c4a2081f438ce295c6954539ad3cf932c40a67ddc686eb7138f22386"} Oct 31 04:38:38.809738 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:38.809713 4376 scope.go:122] "RemoveContainer" containerID="d7f06af13f6fe9b1167f97eae8ad9f3328a03e80cf8722b8f1f2975b50b382f0" Oct 31 04:38:38.810084 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:38:38.810053 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7f06af13f6fe9b1167f97eae8ad9f3328a03e80cf8722b8f1f2975b50b382f0\": container with ID starting with d7f06af13f6fe9b1167f97eae8ad9f3328a03e80cf8722b8f1f2975b50b382f0 not found: ID does not exist" containerID="d7f06af13f6fe9b1167f97eae8ad9f3328a03e80cf8722b8f1f2975b50b382f0" Oct 31 04:38:38.810189 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:38.810096 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7f06af13f6fe9b1167f97eae8ad9f3328a03e80cf8722b8f1f2975b50b382f0"} err="failed to get container status \"d7f06af13f6fe9b1167f97eae8ad9f3328a03e80cf8722b8f1f2975b50b382f0\": rpc error: code = NotFound desc = could not find container \"d7f06af13f6fe9b1167f97eae8ad9f3328a03e80cf8722b8f1f2975b50b382f0\": container with ID starting with d7f06af13f6fe9b1167f97eae8ad9f3328a03e80cf8722b8f1f2975b50b382f0 not found: ID does not exist" Oct 31 04:38:38.873858 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:38.873831 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-57dc6d/auth-test-model-57dc6d-kserve-577cb5ffd5-2wk4v" Oct 31 04:38:38.981005 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:38.980967 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b9cb676-3050-4ea5-b38b-3d32d10ae997-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b9cb676-3050-4ea5-b38b-3d32d10ae997-model-cache\") pod \"1b9cb676-3050-4ea5-b38b-3d32d10ae997\" (UID: \"1b9cb676-3050-4ea5-b38b-3d32d10ae997\") " Oct 31 04:38:38.981187 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:38.981051 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b9cb676-3050-4ea5-b38b-3d32d10ae997-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b9cb676-3050-4ea5-b38b-3d32d10ae997-tmp-dir\") pod \"1b9cb676-3050-4ea5-b38b-3d32d10ae997\" (UID: \"1b9cb676-3050-4ea5-b38b-3d32d10ae997\") " Oct 31 04:38:38.981187 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:38.981167 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b9cb676-3050-4ea5-b38b-3d32d10ae997-tls-certs\" (UniqueName: \"kubernetes.io/secret/1b9cb676-3050-4ea5-b38b-3d32d10ae997-tls-certs\") pod \"1b9cb676-3050-4ea5-b38b-3d32d10ae997\" (UID: \"1b9cb676-3050-4ea5-b38b-3d32d10ae997\") " Oct 31 04:38:38.981272 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:38.981201 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b9cb676-3050-4ea5-b38b-3d32d10ae997-home\" (UniqueName: \"kubernetes.io/empty-dir/1b9cb676-3050-4ea5-b38b-3d32d10ae997-home\") pod \"1b9cb676-3050-4ea5-b38b-3d32d10ae997\" (UID: \"1b9cb676-3050-4ea5-b38b-3d32d10ae997\") " Oct 31 04:38:38.981306 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:38.981281 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b9cb676-3050-4ea5-b38b-3d32d10ae997-model-cache" pod "1b9cb676-3050-4ea5-b38b-3d32d10ae997" (UID: "1b9cb676-3050-4ea5-b38b-3d32d10ae997"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:38:38.981375 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:38.981291 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b9cb676-3050-4ea5-b38b-3d32d10ae997-tmp-dir" pod "1b9cb676-3050-4ea5-b38b-3d32d10ae997" (UID: "1b9cb676-3050-4ea5-b38b-3d32d10ae997"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:38:38.981375 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:38.981318 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b9cb676-3050-4ea5-b38b-3d32d10ae997-kube-api-access-r7rdg\" (UniqueName: \"kubernetes.io/projected/1b9cb676-3050-4ea5-b38b-3d32d10ae997-kube-api-access-r7rdg\") pod \"1b9cb676-3050-4ea5-b38b-3d32d10ae997\" (UID: \"1b9cb676-3050-4ea5-b38b-3d32d10ae997\") " Oct 31 04:38:38.981375 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:38.981369 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b9cb676-3050-4ea5-b38b-3d32d10ae997-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b9cb676-3050-4ea5-b38b-3d32d10ae997-dshm\") pod \"1b9cb676-3050-4ea5-b38b-3d32d10ae997\" (UID: \"1b9cb676-3050-4ea5-b38b-3d32d10ae997\") " Oct 31 04:38:38.981543 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:38.981503 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b9cb676-3050-4ea5-b38b-3d32d10ae997-home" pod "1b9cb676-3050-4ea5-b38b-3d32d10ae997" (UID: "1b9cb676-3050-4ea5-b38b-3d32d10ae997"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:38:38.981931 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:38.981906 4376 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b9cb676-3050-4ea5-b38b-3d32d10ae997-model-cache\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:38:38.982019 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:38.981934 4376 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b9cb676-3050-4ea5-b38b-3d32d10ae997-tmp-dir\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:38:38.982019 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:38.981947 4376 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b9cb676-3050-4ea5-b38b-3d32d10ae997-home\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:38:38.983759 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:38.983735 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b9cb676-3050-4ea5-b38b-3d32d10ae997-kube-api-access-r7rdg" pod "1b9cb676-3050-4ea5-b38b-3d32d10ae997" (UID: "1b9cb676-3050-4ea5-b38b-3d32d10ae997"). InnerVolumeSpecName "kube-api-access-r7rdg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:38:38.983867 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:38.983848 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b9cb676-3050-4ea5-b38b-3d32d10ae997-tls-certs" pod "1b9cb676-3050-4ea5-b38b-3d32d10ae997" (UID: "1b9cb676-3050-4ea5-b38b-3d32d10ae997"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:38:38.983906 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:38.983853 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b9cb676-3050-4ea5-b38b-3d32d10ae997-dshm" pod "1b9cb676-3050-4ea5-b38b-3d32d10ae997" (UID: "1b9cb676-3050-4ea5-b38b-3d32d10ae997"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:38:39.083227 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:39.083196 4376 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b9cb676-3050-4ea5-b38b-3d32d10ae997-dshm\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:38:39.083227 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:39.083227 4376 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b9cb676-3050-4ea5-b38b-3d32d10ae997-tls-certs\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:38:39.083440 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:39.083238 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7rdg\" (UniqueName: \"kubernetes.io/projected/1b9cb676-3050-4ea5-b38b-3d32d10ae997-kube-api-access-r7rdg\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:38:39.811949 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:39.811921 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-57dc6d/auth-test-model-57dc6d-kserve-577cb5ffd5-2wk4v" Oct 31 04:38:39.812311 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:39.811961 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-57dc6d/auth-test-model-57dc6d-kserve-577cb5ffd5-2wk4v" event={"ID":"1b9cb676-3050-4ea5-b38b-3d32d10ae997","Type":"ContainerDied","Data":"939dcfa6431b8031a10cda52b3e8dce2dc37c444c17f75902e10e55bdc7fd0c2"} Oct 31 04:38:39.812311 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:39.812017 4376 scope.go:122] "RemoveContainer" containerID="4c4ba863c4a2081f438ce295c6954539ad3cf932c40a67ddc686eb7138f22386" Oct 31 04:38:39.813379 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:39.813324 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7108-535368" podUID="ae170649-4d7e-4863-b994-6ae750b1ab43" containerName="test-curl-7108-535368" containerID="cri-o://b1636b3408284eaa61e52a7856c46c9bd2844ecaa729927cebb038743793df0f" gracePeriod=2 Oct 31 04:38:39.845515 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:38:39.845482 4376 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7108-535368\" is forbidden: User \"system:node:ip-10-0-134-155.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-155.ec2.internal' and this object" podUID="ae170649-4d7e-4863-b994-6ae750b1ab43" pod="opendatahub/test-curl-7108-535368" Oct 31 04:38:39.904185 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:39.904146 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-57dc6d/auth-test-model-57dc6d-kserve-577cb5ffd5-2wk4v"] Oct 31 04:38:39.910652 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:39.910624 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w4-57dc6d/auth-test-model-57dc6d-kserve-577cb5ffd5-2wk4v"] Oct 31 04:38:39.921165 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:38:39.921121 4376 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7108-535368\" is forbidden: User \"system:node:ip-10-0-134-155.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-155.ec2.internal' and this object" podUID="ae170649-4d7e-4863-b994-6ae750b1ab43" pod="opendatahub/test-curl-7108-535368" Oct 31 04:38:40.018195 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:40.018169 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7108-e075d1" Oct 31 04:38:40.032214 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:38:40.032179 4376 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7108-535368\" is forbidden: User \"system:node:ip-10-0-134-155.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-155.ec2.internal' and this object" podUID="ae170649-4d7e-4863-b994-6ae750b1ab43" pod="opendatahub/test-curl-7108-535368" Oct 31 04:38:40.102509 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:40.102414 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb1c5477-0a48-4118-98e1-4127357fd27c-kube-api-access-7lrqq\" (UniqueName: \"kubernetes.io/projected/bb1c5477-0a48-4118-98e1-4127357fd27c-kube-api-access-7lrqq\") pod \"bb1c5477-0a48-4118-98e1-4127357fd27c\" (UID: \"bb1c5477-0a48-4118-98e1-4127357fd27c\") " Oct 31 04:38:40.104667 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:40.104640 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb1c5477-0a48-4118-98e1-4127357fd27c-kube-api-access-7lrqq" pod "bb1c5477-0a48-4118-98e1-4127357fd27c" (UID: "bb1c5477-0a48-4118-98e1-4127357fd27c"). InnerVolumeSpecName "kube-api-access-7lrqq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:38:40.204216 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:40.204188 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7lrqq\" (UniqueName: \"kubernetes.io/projected/bb1c5477-0a48-4118-98e1-4127357fd27c-kube-api-access-7lrqq\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:38:40.444740 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:40.444708 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b9cb676-3050-4ea5-b38b-3d32d10ae997" path="/var/lib/kubelet/pods/1b9cb676-3050-4ea5-b38b-3d32d10ae997/volumes" Oct 31 04:38:40.445137 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:40.445123 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb1c5477-0a48-4118-98e1-4127357fd27c" path="/var/lib/kubelet/pods/bb1c5477-0a48-4118-98e1-4127357fd27c/volumes" Oct 31 04:38:40.818872 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:40.818785 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-654741-openshift-default-8449896d9c-cf4pt"] Oct 31 04:38:40.820400 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:40.820375 4376 cpu_manager.go:414] "containerMap: removing container" podUID="ae170649-4d7e-4863-b994-6ae750b1ab43" containerName="test-curl-7108-535368" Oct 31 04:38:40.820545 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:40.820532 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae170649-4d7e-4863-b994-6ae750b1ab43" containerName="test-curl-7108-535368" Oct 31 04:38:40.820680 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:40.820670 4376 cpu_manager.go:414] "containerMap: removing container" podUID="1b9cb676-3050-4ea5-b38b-3d32d10ae997" containerName="main" Oct 31 04:38:40.820758 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:40.820748 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b9cb676-3050-4ea5-b38b-3d32d10ae997" containerName="main" Oct 31 04:38:40.821014 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:40.820998 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae170649-4d7e-4863-b994-6ae750b1ab43" containerName="test-curl-7108-535368" Oct 31 04:38:40.821122 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:40.821112 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b9cb676-3050-4ea5-b38b-3d32d10ae997" containerName="main" Oct 31 04:38:40.838389 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:40.836935 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-654741-openshift-default-8449896d9c-cf4pt" Oct 31 04:38:40.863824 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:40.861666 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-654741-openshift-default-dockercfg-w5m5s\"" Oct 31 04:38:40.879448 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:40.878973 4376 generic.go:356] "Generic (PLEG): container finished" podID="bb1c5477-0a48-4118-98e1-4127357fd27c" containerID="eb930bed74408ea070d12fac330fdddb87dd4de312aea79b14e8f5448b7d30bb" exitCode=137 Oct 31 04:38:40.879448 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:40.879099 4376 scope.go:122] "RemoveContainer" containerID="eb930bed74408ea070d12fac330fdddb87dd4de312aea79b14e8f5448b7d30bb" Oct 31 04:38:40.879448 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:40.879249 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7108-e075d1" Oct 31 04:38:40.892466 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:40.891385 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-654741-openshift-default-8449896d9c-cf4pt"] Oct 31 04:38:40.895084 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:40.895058 4376 scope.go:122] "RemoveContainer" containerID="eb930bed74408ea070d12fac330fdddb87dd4de312aea79b14e8f5448b7d30bb" Oct 31 04:38:40.895574 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:38:40.895545 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb930bed74408ea070d12fac330fdddb87dd4de312aea79b14e8f5448b7d30bb\": container with ID starting with eb930bed74408ea070d12fac330fdddb87dd4de312aea79b14e8f5448b7d30bb not found: ID does not exist" containerID="eb930bed74408ea070d12fac330fdddb87dd4de312aea79b14e8f5448b7d30bb" Oct 31 04:38:40.895749 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:40.895714 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb930bed74408ea070d12fac330fdddb87dd4de312aea79b14e8f5448b7d30bb"} err="failed to get container status \"eb930bed74408ea070d12fac330fdddb87dd4de312aea79b14e8f5448b7d30bb\": rpc error: code = NotFound desc = could not find container \"eb930bed74408ea070d12fac330fdddb87dd4de312aea79b14e8f5448b7d30bb\": container with ID starting with eb930bed74408ea070d12fac330fdddb87dd4de312aea79b14e8f5448b7d30bb not found: ID does not exist" Oct 31 04:38:40.913404 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:40.912789 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32954f51-d716-449a-ba69-ca46851a9ce2-istio-token\") pod \"e2e-extmod-a-654741-openshift-default-8449896d9c-cf4pt\" (UID: \"32954f51-d716-449a-ba69-ca46851a9ce2\") " pod="openshift-ingress/e2e-extmod-a-654741-openshift-default-8449896d9c-cf4pt" Oct 31 04:38:40.913404 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:40.912839 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32954f51-d716-449a-ba69-ca46851a9ce2-workload-socket\") pod \"e2e-extmod-a-654741-openshift-default-8449896d9c-cf4pt\" (UID: \"32954f51-d716-449a-ba69-ca46851a9ce2\") " pod="openshift-ingress/e2e-extmod-a-654741-openshift-default-8449896d9c-cf4pt" Oct 31 04:38:40.913404 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:40.912943 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32954f51-d716-449a-ba69-ca46851a9ce2-istio-envoy\") pod \"e2e-extmod-a-654741-openshift-default-8449896d9c-cf4pt\" (UID: \"32954f51-d716-449a-ba69-ca46851a9ce2\") " pod="openshift-ingress/e2e-extmod-a-654741-openshift-default-8449896d9c-cf4pt" Oct 31 04:38:40.913404 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:40.912980 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwp2l\" (UniqueName: \"kubernetes.io/projected/32954f51-d716-449a-ba69-ca46851a9ce2-kube-api-access-jwp2l\") pod \"e2e-extmod-a-654741-openshift-default-8449896d9c-cf4pt\" (UID: \"32954f51-d716-449a-ba69-ca46851a9ce2\") " pod="openshift-ingress/e2e-extmod-a-654741-openshift-default-8449896d9c-cf4pt" Oct 31 04:38:40.913404 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:40.913021 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32954f51-d716-449a-ba69-ca46851a9ce2-credential-socket\") pod \"e2e-extmod-a-654741-openshift-default-8449896d9c-cf4pt\" (UID: \"32954f51-d716-449a-ba69-ca46851a9ce2\") " pod="openshift-ingress/e2e-extmod-a-654741-openshift-default-8449896d9c-cf4pt" Oct 31 04:38:40.913404 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:40.913080 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32954f51-d716-449a-ba69-ca46851a9ce2-istio-podinfo\") pod \"e2e-extmod-a-654741-openshift-default-8449896d9c-cf4pt\" (UID: \"32954f51-d716-449a-ba69-ca46851a9ce2\") " pod="openshift-ingress/e2e-extmod-a-654741-openshift-default-8449896d9c-cf4pt" Oct 31 04:38:40.913404 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:40.913127 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32954f51-d716-449a-ba69-ca46851a9ce2-istiod-ca-cert\") pod \"e2e-extmod-a-654741-openshift-default-8449896d9c-cf4pt\" (UID: \"32954f51-d716-449a-ba69-ca46851a9ce2\") " pod="openshift-ingress/e2e-extmod-a-654741-openshift-default-8449896d9c-cf4pt" Oct 31 04:38:40.913404 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:40.913209 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32954f51-d716-449a-ba69-ca46851a9ce2-workload-certs\") pod \"e2e-extmod-a-654741-openshift-default-8449896d9c-cf4pt\" (UID: \"32954f51-d716-449a-ba69-ca46851a9ce2\") " pod="openshift-ingress/e2e-extmod-a-654741-openshift-default-8449896d9c-cf4pt" Oct 31 04:38:40.913404 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:40.913297 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32954f51-d716-449a-ba69-ca46851a9ce2-istio-data\") pod \"e2e-extmod-a-654741-openshift-default-8449896d9c-cf4pt\" (UID: \"32954f51-d716-449a-ba69-ca46851a9ce2\") " pod="openshift-ingress/e2e-extmod-a-654741-openshift-default-8449896d9c-cf4pt" Oct 31 04:38:41.041356 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:41.038535 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32954f51-d716-449a-ba69-ca46851a9ce2-credential-socket\") pod \"e2e-extmod-a-654741-openshift-default-8449896d9c-cf4pt\" (UID: \"32954f51-d716-449a-ba69-ca46851a9ce2\") " pod="openshift-ingress/e2e-extmod-a-654741-openshift-default-8449896d9c-cf4pt" Oct 31 04:38:41.041356 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:41.038619 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32954f51-d716-449a-ba69-ca46851a9ce2-istio-podinfo\") pod \"e2e-extmod-a-654741-openshift-default-8449896d9c-cf4pt\" (UID: \"32954f51-d716-449a-ba69-ca46851a9ce2\") " pod="openshift-ingress/e2e-extmod-a-654741-openshift-default-8449896d9c-cf4pt" Oct 31 04:38:41.041356 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:41.038665 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32954f51-d716-449a-ba69-ca46851a9ce2-istiod-ca-cert\") pod \"e2e-extmod-a-654741-openshift-default-8449896d9c-cf4pt\" (UID: \"32954f51-d716-449a-ba69-ca46851a9ce2\") " pod="openshift-ingress/e2e-extmod-a-654741-openshift-default-8449896d9c-cf4pt" Oct 31 04:38:41.041356 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:41.038744 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32954f51-d716-449a-ba69-ca46851a9ce2-workload-certs\") pod \"e2e-extmod-a-654741-openshift-default-8449896d9c-cf4pt\" (UID: \"32954f51-d716-449a-ba69-ca46851a9ce2\") " pod="openshift-ingress/e2e-extmod-a-654741-openshift-default-8449896d9c-cf4pt" Oct 31 04:38:41.041356 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:41.038828 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32954f51-d716-449a-ba69-ca46851a9ce2-istio-data\") pod \"e2e-extmod-a-654741-openshift-default-8449896d9c-cf4pt\" (UID: \"32954f51-d716-449a-ba69-ca46851a9ce2\") " pod="openshift-ingress/e2e-extmod-a-654741-openshift-default-8449896d9c-cf4pt" Oct 31 04:38:41.041356 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:41.038937 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32954f51-d716-449a-ba69-ca46851a9ce2-istio-token\") pod \"e2e-extmod-a-654741-openshift-default-8449896d9c-cf4pt\" (UID: \"32954f51-d716-449a-ba69-ca46851a9ce2\") " pod="openshift-ingress/e2e-extmod-a-654741-openshift-default-8449896d9c-cf4pt" Oct 31 04:38:41.041356 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:41.038959 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32954f51-d716-449a-ba69-ca46851a9ce2-workload-socket\") pod \"e2e-extmod-a-654741-openshift-default-8449896d9c-cf4pt\" (UID: \"32954f51-d716-449a-ba69-ca46851a9ce2\") " pod="openshift-ingress/e2e-extmod-a-654741-openshift-default-8449896d9c-cf4pt" Oct 31 04:38:41.041356 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:41.039042 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32954f51-d716-449a-ba69-ca46851a9ce2-istio-envoy\") pod \"e2e-extmod-a-654741-openshift-default-8449896d9c-cf4pt\" (UID: \"32954f51-d716-449a-ba69-ca46851a9ce2\") " pod="openshift-ingress/e2e-extmod-a-654741-openshift-default-8449896d9c-cf4pt" Oct 31 04:38:41.041356 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:41.039078 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwp2l\" (UniqueName: \"kubernetes.io/projected/32954f51-d716-449a-ba69-ca46851a9ce2-kube-api-access-jwp2l\") pod \"e2e-extmod-a-654741-openshift-default-8449896d9c-cf4pt\" (UID: \"32954f51-d716-449a-ba69-ca46851a9ce2\") " pod="openshift-ingress/e2e-extmod-a-654741-openshift-default-8449896d9c-cf4pt" Oct 31 04:38:41.041356 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:41.039798 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32954f51-d716-449a-ba69-ca46851a9ce2-credential-socket\") pod \"e2e-extmod-a-654741-openshift-default-8449896d9c-cf4pt\" (UID: \"32954f51-d716-449a-ba69-ca46851a9ce2\") " pod="openshift-ingress/e2e-extmod-a-654741-openshift-default-8449896d9c-cf4pt" Oct 31 04:38:41.050352 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:41.042225 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32954f51-d716-449a-ba69-ca46851a9ce2-istio-data\") pod \"e2e-extmod-a-654741-openshift-default-8449896d9c-cf4pt\" (UID: \"32954f51-d716-449a-ba69-ca46851a9ce2\") " pod="openshift-ingress/e2e-extmod-a-654741-openshift-default-8449896d9c-cf4pt" Oct 31 04:38:41.050617 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:41.047158 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32954f51-d716-449a-ba69-ca46851a9ce2-workload-socket\") pod \"e2e-extmod-a-654741-openshift-default-8449896d9c-cf4pt\" (UID: \"32954f51-d716-449a-ba69-ca46851a9ce2\") " pod="openshift-ingress/e2e-extmod-a-654741-openshift-default-8449896d9c-cf4pt" Oct 31 04:38:41.050733 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:41.049026 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32954f51-d716-449a-ba69-ca46851a9ce2-istiod-ca-cert\") pod \"e2e-extmod-a-654741-openshift-default-8449896d9c-cf4pt\" (UID: \"32954f51-d716-449a-ba69-ca46851a9ce2\") " pod="openshift-ingress/e2e-extmod-a-654741-openshift-default-8449896d9c-cf4pt" Oct 31 04:38:41.050839 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:41.049422 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32954f51-d716-449a-ba69-ca46851a9ce2-istio-podinfo\") pod \"e2e-extmod-a-654741-openshift-default-8449896d9c-cf4pt\" (UID: \"32954f51-d716-449a-ba69-ca46851a9ce2\") " pod="openshift-ingress/e2e-extmod-a-654741-openshift-default-8449896d9c-cf4pt" Oct 31 04:38:41.050929 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:41.049632 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32954f51-d716-449a-ba69-ca46851a9ce2-workload-certs\") pod \"e2e-extmod-a-654741-openshift-default-8449896d9c-cf4pt\" (UID: \"32954f51-d716-449a-ba69-ca46851a9ce2\") " pod="openshift-ingress/e2e-extmod-a-654741-openshift-default-8449896d9c-cf4pt" Oct 31 04:38:41.076880 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:41.061293 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32954f51-d716-449a-ba69-ca46851a9ce2-istio-envoy\") pod \"e2e-extmod-a-654741-openshift-default-8449896d9c-cf4pt\" (UID: \"32954f51-d716-449a-ba69-ca46851a9ce2\") " pod="openshift-ingress/e2e-extmod-a-654741-openshift-default-8449896d9c-cf4pt" Oct 31 04:38:41.077084 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:41.077051 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwp2l\" (UniqueName: \"kubernetes.io/projected/32954f51-d716-449a-ba69-ca46851a9ce2-kube-api-access-jwp2l\") pod \"e2e-extmod-a-654741-openshift-default-8449896d9c-cf4pt\" (UID: \"32954f51-d716-449a-ba69-ca46851a9ce2\") " pod="openshift-ingress/e2e-extmod-a-654741-openshift-default-8449896d9c-cf4pt" Oct 31 04:38:41.119425 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:41.104060 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32954f51-d716-449a-ba69-ca46851a9ce2-istio-token\") pod \"e2e-extmod-a-654741-openshift-default-8449896d9c-cf4pt\" (UID: \"32954f51-d716-449a-ba69-ca46851a9ce2\") " pod="openshift-ingress/e2e-extmod-a-654741-openshift-default-8449896d9c-cf4pt" Oct 31 04:38:41.175348 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:41.174724 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-654741-openshift-default-8449896d9c-cf4pt" Oct 31 04:38:41.187357 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:38:41.185174 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 04:38:41.523018 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:38:41.522986 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32954f51_d716_449a_ba69_ca46851a9ce2.slice/crio-a09d1cb50a03a82f070b9fce636cadf84121549c9828689296b92a9c0a09d32a WatchSource:0}: Error finding container a09d1cb50a03a82f070b9fce636cadf84121549c9828689296b92a9c0a09d32a: Status 404 returned error can't find the container with id a09d1cb50a03a82f070b9fce636cadf84121549c9828689296b92a9c0a09d32a Oct 31 04:38:41.525177 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:41.525153 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-654741-openshift-default-8449896d9c-cf4pt"] Oct 31 04:38:41.526380 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:41.526318 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:38:41.526456 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:41.526429 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:38:41.526499 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:41.526473 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:38:41.891151 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:41.891114 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-654741-openshift-default-8449896d9c-cf4pt" event={"ID":"32954f51-d716-449a-ba69-ca46851a9ce2","Type":"ContainerStarted","Data":"1e419808a87c9def4ea0621cb542832602a03d85745a696cd6428c97d9fa8b62"} Oct 31 04:38:41.891667 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:41.891183 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-654741-openshift-default-8449896d9c-cf4pt" event={"ID":"32954f51-d716-449a-ba69-ca46851a9ce2","Type":"ContainerStarted","Data":"a09d1cb50a03a82f070b9fce636cadf84121549c9828689296b92a9c0a09d32a"} Oct 31 04:38:41.893072 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:41.893048 4376 generic.go:356] "Generic (PLEG): container finished" podID="ae170649-4d7e-4863-b994-6ae750b1ab43" containerID="b1636b3408284eaa61e52a7856c46c9bd2844ecaa729927cebb038743793df0f" exitCode=137 Oct 31 04:38:41.968317 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:41.968250 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-654741-openshift-default-8449896d9c-cf4pt" podStartSLOduration=1.968226906 podStartE2EDuration="1.968226906s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:38:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:38:41.956387047 +0000 UTC m=+2069.670721676" watchObservedRunningTime="2026-10-31 04:38:41.968226906 +0000 UTC m=+2069.682561524" Oct 31 04:38:42.043288 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:42.043260 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7108-535368" Oct 31 04:38:42.173775 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:42.173752 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae170649-4d7e-4863-b994-6ae750b1ab43-kube-api-access-g82l6\" (UniqueName: \"kubernetes.io/projected/ae170649-4d7e-4863-b994-6ae750b1ab43-kube-api-access-g82l6\") pod \"ae170649-4d7e-4863-b994-6ae750b1ab43\" (UID: \"ae170649-4d7e-4863-b994-6ae750b1ab43\") " Oct 31 04:38:42.176673 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:42.176597 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-654741-openshift-default-8449896d9c-cf4pt" Oct 31 04:38:42.178846 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:42.178805 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae170649-4d7e-4863-b994-6ae750b1ab43-kube-api-access-g82l6" pod "ae170649-4d7e-4863-b994-6ae750b1ab43" (UID: "ae170649-4d7e-4863-b994-6ae750b1ab43"). InnerVolumeSpecName "kube-api-access-g82l6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:38:42.184603 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:42.184445 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-654741-openshift-default-8449896d9c-cf4pt" Oct 31 04:38:42.274788 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:42.274759 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-nggns" Oct 31 04:38:42.275803 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:42.275772 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g82l6\" (UniqueName: \"kubernetes.io/projected/ae170649-4d7e-4863-b994-6ae750b1ab43-kube-api-access-g82l6\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:38:42.376185 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:42.376147 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d001c91-1967-4fb1-94f4-8f8a8db7216b-istio-token\" (UniqueName: \"kubernetes.io/projected/8d001c91-1967-4fb1-94f4-8f8a8db7216b-istio-token\") pod \"8d001c91-1967-4fb1-94f4-8f8a8db7216b\" (UID: \"8d001c91-1967-4fb1-94f4-8f8a8db7216b\") " Oct 31 04:38:42.376480 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:42.376316 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d001c91-1967-4fb1-94f4-8f8a8db7216b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d001c91-1967-4fb1-94f4-8f8a8db7216b-istiod-ca-cert\") pod \"8d001c91-1967-4fb1-94f4-8f8a8db7216b\" (UID: \"8d001c91-1967-4fb1-94f4-8f8a8db7216b\") " Oct 31 04:38:42.376480 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:42.376413 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d001c91-1967-4fb1-94f4-8f8a8db7216b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d001c91-1967-4fb1-94f4-8f8a8db7216b-credential-socket\") pod \"8d001c91-1967-4fb1-94f4-8f8a8db7216b\" (UID: \"8d001c91-1967-4fb1-94f4-8f8a8db7216b\") " Oct 31 04:38:42.376611 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:42.376510 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d001c91-1967-4fb1-94f4-8f8a8db7216b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d001c91-1967-4fb1-94f4-8f8a8db7216b-istio-envoy\") pod \"8d001c91-1967-4fb1-94f4-8f8a8db7216b\" (UID: \"8d001c91-1967-4fb1-94f4-8f8a8db7216b\") " Oct 31 04:38:42.376611 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:42.376563 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d001c91-1967-4fb1-94f4-8f8a8db7216b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d001c91-1967-4fb1-94f4-8f8a8db7216b-istio-data\") pod \"8d001c91-1967-4fb1-94f4-8f8a8db7216b\" (UID: \"8d001c91-1967-4fb1-94f4-8f8a8db7216b\") " Oct 31 04:38:42.376710 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:42.376683 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d001c91-1967-4fb1-94f4-8f8a8db7216b-kube-api-access-662b5\" (UniqueName: \"kubernetes.io/projected/8d001c91-1967-4fb1-94f4-8f8a8db7216b-kube-api-access-662b5\") pod \"8d001c91-1967-4fb1-94f4-8f8a8db7216b\" (UID: \"8d001c91-1967-4fb1-94f4-8f8a8db7216b\") " Oct 31 04:38:42.376769 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:42.376716 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d001c91-1967-4fb1-94f4-8f8a8db7216b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d001c91-1967-4fb1-94f4-8f8a8db7216b-workload-certs\") pod \"8d001c91-1967-4fb1-94f4-8f8a8db7216b\" (UID: \"8d001c91-1967-4fb1-94f4-8f8a8db7216b\") " Oct 31 04:38:42.376769 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:42.376725 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d001c91-1967-4fb1-94f4-8f8a8db7216b-credential-socket" pod "8d001c91-1967-4fb1-94f4-8f8a8db7216b" (UID: "8d001c91-1967-4fb1-94f4-8f8a8db7216b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:38:42.376769 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:42.376756 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8d001c91-1967-4fb1-94f4-8f8a8db7216b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d001c91-1967-4fb1-94f4-8f8a8db7216b-istio-podinfo\") pod \"8d001c91-1967-4fb1-94f4-8f8a8db7216b\" (UID: \"8d001c91-1967-4fb1-94f4-8f8a8db7216b\") " Oct 31 04:38:42.376936 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:42.376817 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d001c91-1967-4fb1-94f4-8f8a8db7216b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d001c91-1967-4fb1-94f4-8f8a8db7216b-workload-socket\") pod \"8d001c91-1967-4fb1-94f4-8f8a8db7216b\" (UID: \"8d001c91-1967-4fb1-94f4-8f8a8db7216b\") " Oct 31 04:38:42.377495 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:42.376758 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d001c91-1967-4fb1-94f4-8f8a8db7216b-istiod-ca-cert" pod "8d001c91-1967-4fb1-94f4-8f8a8db7216b" (UID: "8d001c91-1967-4fb1-94f4-8f8a8db7216b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:38:42.377495 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:42.376957 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d001c91-1967-4fb1-94f4-8f8a8db7216b-istio-data" pod "8d001c91-1967-4fb1-94f4-8f8a8db7216b" (UID: "8d001c91-1967-4fb1-94f4-8f8a8db7216b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:38:42.377495 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:42.376968 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d001c91-1967-4fb1-94f4-8f8a8db7216b-workload-certs" pod "8d001c91-1967-4fb1-94f4-8f8a8db7216b" (UID: "8d001c91-1967-4fb1-94f4-8f8a8db7216b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:38:42.377716 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:42.377540 4376 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d001c91-1967-4fb1-94f4-8f8a8db7216b-istiod-ca-cert\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:38:42.377716 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:42.377569 4376 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d001c91-1967-4fb1-94f4-8f8a8db7216b-credential-socket\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:38:42.377716 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:42.377571 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d001c91-1967-4fb1-94f4-8f8a8db7216b-workload-socket" pod "8d001c91-1967-4fb1-94f4-8f8a8db7216b" (UID: "8d001c91-1967-4fb1-94f4-8f8a8db7216b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:38:42.379566 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:42.379537 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d001c91-1967-4fb1-94f4-8f8a8db7216b-istio-envoy" pod "8d001c91-1967-4fb1-94f4-8f8a8db7216b" (UID: "8d001c91-1967-4fb1-94f4-8f8a8db7216b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:38:42.379687 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:42.379623 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d001c91-1967-4fb1-94f4-8f8a8db7216b-kube-api-access-662b5" pod "8d001c91-1967-4fb1-94f4-8f8a8db7216b" (UID: "8d001c91-1967-4fb1-94f4-8f8a8db7216b"). InnerVolumeSpecName "kube-api-access-662b5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:38:42.379754 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:42.379722 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d001c91-1967-4fb1-94f4-8f8a8db7216b-istio-token" pod "8d001c91-1967-4fb1-94f4-8f8a8db7216b" (UID: "8d001c91-1967-4fb1-94f4-8f8a8db7216b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:38:42.379882 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:42.379861 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8d001c91-1967-4fb1-94f4-8f8a8db7216b-istio-podinfo" pod "8d001c91-1967-4fb1-94f4-8f8a8db7216b" (UID: "8d001c91-1967-4fb1-94f4-8f8a8db7216b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 04:38:42.448904 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:42.448860 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae170649-4d7e-4863-b994-6ae750b1ab43" path="/var/lib/kubelet/pods/ae170649-4d7e-4863-b994-6ae750b1ab43/volumes" Oct 31 04:38:42.479610 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:42.479548 4376 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d001c91-1967-4fb1-94f4-8f8a8db7216b-istio-envoy\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:38:42.479610 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:42.479580 4376 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d001c91-1967-4fb1-94f4-8f8a8db7216b-istio-data\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:38:42.479610 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:42.479596 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-662b5\" (UniqueName: \"kubernetes.io/projected/8d001c91-1967-4fb1-94f4-8f8a8db7216b-kube-api-access-662b5\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:38:42.479610 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:42.479609 4376 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d001c91-1967-4fb1-94f4-8f8a8db7216b-workload-certs\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:38:42.479610 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:42.479622 4376 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d001c91-1967-4fb1-94f4-8f8a8db7216b-istio-podinfo\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:38:42.479610 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:42.479633 4376 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d001c91-1967-4fb1-94f4-8f8a8db7216b-workload-socket\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:38:42.479951 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:42.479642 4376 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d001c91-1967-4fb1-94f4-8f8a8db7216b-istio-token\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:38:42.900886 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:42.900855 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7108-535368" Oct 31 04:38:42.901374 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:42.900912 4376 scope.go:122] "RemoveContainer" containerID="b1636b3408284eaa61e52a7856c46c9bd2844ecaa729927cebb038743793df0f" Oct 31 04:38:42.902712 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:42.902686 4376 generic.go:356] "Generic (PLEG): container finished" podID="8d001c91-1967-4fb1-94f4-8f8a8db7216b" containerID="6896af4706d0f2de28de8772f390946c6406e8b80f07603c9ccc5934a88c144f" exitCode=0 Oct 31 04:38:42.902853 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:42.902747 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-nggns" Oct 31 04:38:42.902853 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:42.902770 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-nggns" event={"ID":"8d001c91-1967-4fb1-94f4-8f8a8db7216b","Type":"ContainerDied","Data":"6896af4706d0f2de28de8772f390946c6406e8b80f07603c9ccc5934a88c144f"} Oct 31 04:38:42.902853 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:42.902832 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-nggns" event={"ID":"8d001c91-1967-4fb1-94f4-8f8a8db7216b","Type":"ContainerDied","Data":"6820bb766a0abdae176ff0b224dfe8b652c9a2ae69ddc99c484006f16d3dafdd"} Oct 31 04:38:42.903383 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:42.903363 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-654741-openshift-default-8449896d9c-cf4pt" Oct 31 04:38:42.904381 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:42.904362 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-654741-openshift-default-8449896d9c-cf4pt" Oct 31 04:38:42.911139 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:42.911114 4376 scope.go:122] "RemoveContainer" containerID="6896af4706d0f2de28de8772f390946c6406e8b80f07603c9ccc5934a88c144f" Oct 31 04:38:42.920878 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:42.920858 4376 scope.go:122] "RemoveContainer" containerID="6896af4706d0f2de28de8772f390946c6406e8b80f07603c9ccc5934a88c144f" Oct 31 04:38:42.921226 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:38:42.921208 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6896af4706d0f2de28de8772f390946c6406e8b80f07603c9ccc5934a88c144f\": container with ID starting with 6896af4706d0f2de28de8772f390946c6406e8b80f07603c9ccc5934a88c144f not found: ID does not exist" containerID="6896af4706d0f2de28de8772f390946c6406e8b80f07603c9ccc5934a88c144f" Oct 31 04:38:42.921272 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:42.921236 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6896af4706d0f2de28de8772f390946c6406e8b80f07603c9ccc5934a88c144f"} err="failed to get container status \"6896af4706d0f2de28de8772f390946c6406e8b80f07603c9ccc5934a88c144f\": rpc error: code = NotFound desc = could not find container \"6896af4706d0f2de28de8772f390946c6406e8b80f07603c9ccc5934a88c144f\": container with ID starting with 6896af4706d0f2de28de8772f390946c6406e8b80f07603c9ccc5934a88c144f not found: ID does not exist" Oct 31 04:38:42.970153 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:42.970119 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-nggns"] Oct 31 04:38:42.983811 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:42.983781 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-nggns"] Oct 31 04:38:43.037258 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:43.037216 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w4-9633f0/auth-test-model-9633f0-kserve-6cbd55ffd9-z9rjx" podUID="5b72272f-1227-401e-bec7-56182f8c1676" containerName="main" containerID="cri-o://242a852c6e0a38931e92511cfb8ede33e98a65859b0dfc9faa3b6515218f81d3" gracePeriod=45 Oct 31 04:38:43.325874 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:43.325840 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-9633f0/auth-test-model-9633f0-kserve-6cbd55ffd9-z9rjx" Oct 31 04:38:43.392186 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:43.392157 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5b72272f-1227-401e-bec7-56182f8c1676-tls-certs\" (UniqueName: \"kubernetes.io/secret/5b72272f-1227-401e-bec7-56182f8c1676-tls-certs\") pod \"5b72272f-1227-401e-bec7-56182f8c1676\" (UID: \"5b72272f-1227-401e-bec7-56182f8c1676\") " Oct 31 04:38:43.392350 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:43.392247 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b72272f-1227-401e-bec7-56182f8c1676-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b72272f-1227-401e-bec7-56182f8c1676-dshm\") pod \"5b72272f-1227-401e-bec7-56182f8c1676\" (UID: \"5b72272f-1227-401e-bec7-56182f8c1676\") " Oct 31 04:38:43.392350 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:43.392314 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b72272f-1227-401e-bec7-56182f8c1676-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b72272f-1227-401e-bec7-56182f8c1676-model-cache\") pod \"5b72272f-1227-401e-bec7-56182f8c1676\" (UID: \"5b72272f-1227-401e-bec7-56182f8c1676\") " Oct 31 04:38:43.392436 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:43.392379 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b72272f-1227-401e-bec7-56182f8c1676-kube-api-access-64j2q\" (UniqueName: \"kubernetes.io/projected/5b72272f-1227-401e-bec7-56182f8c1676-kube-api-access-64j2q\") pod \"5b72272f-1227-401e-bec7-56182f8c1676\" (UID: \"5b72272f-1227-401e-bec7-56182f8c1676\") " Oct 31 04:38:43.392436 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:43.392413 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b72272f-1227-401e-bec7-56182f8c1676-home\" (UniqueName: \"kubernetes.io/empty-dir/5b72272f-1227-401e-bec7-56182f8c1676-home\") pod \"5b72272f-1227-401e-bec7-56182f8c1676\" (UID: \"5b72272f-1227-401e-bec7-56182f8c1676\") " Oct 31 04:38:43.392557 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:43.392496 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b72272f-1227-401e-bec7-56182f8c1676-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b72272f-1227-401e-bec7-56182f8c1676-tmp-dir\") pod \"5b72272f-1227-401e-bec7-56182f8c1676\" (UID: \"5b72272f-1227-401e-bec7-56182f8c1676\") " Oct 31 04:38:43.392718 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:43.392665 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b72272f-1227-401e-bec7-56182f8c1676-model-cache" pod "5b72272f-1227-401e-bec7-56182f8c1676" (UID: "5b72272f-1227-401e-bec7-56182f8c1676"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:38:43.392848 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:43.392767 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b72272f-1227-401e-bec7-56182f8c1676-home" pod "5b72272f-1227-401e-bec7-56182f8c1676" (UID: "5b72272f-1227-401e-bec7-56182f8c1676"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:38:43.392985 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:43.392960 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b72272f-1227-401e-bec7-56182f8c1676-tmp-dir" pod "5b72272f-1227-401e-bec7-56182f8c1676" (UID: "5b72272f-1227-401e-bec7-56182f8c1676"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:38:43.393269 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:43.393248 4376 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b72272f-1227-401e-bec7-56182f8c1676-model-cache\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:38:43.393369 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:43.393275 4376 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b72272f-1227-401e-bec7-56182f8c1676-home\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:38:43.393369 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:43.393289 4376 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b72272f-1227-401e-bec7-56182f8c1676-tmp-dir\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:38:43.394741 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:43.394713 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b72272f-1227-401e-bec7-56182f8c1676-tls-certs" pod "5b72272f-1227-401e-bec7-56182f8c1676" (UID: "5b72272f-1227-401e-bec7-56182f8c1676"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:38:43.394892 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:43.394870 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b72272f-1227-401e-bec7-56182f8c1676-dshm" pod "5b72272f-1227-401e-bec7-56182f8c1676" (UID: "5b72272f-1227-401e-bec7-56182f8c1676"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:38:43.395233 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:43.395201 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b72272f-1227-401e-bec7-56182f8c1676-kube-api-access-64j2q" pod "5b72272f-1227-401e-bec7-56182f8c1676" (UID: "5b72272f-1227-401e-bec7-56182f8c1676"). InnerVolumeSpecName "kube-api-access-64j2q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:38:43.494748 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:43.494670 4376 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b72272f-1227-401e-bec7-56182f8c1676-dshm\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:38:43.494748 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:43.494698 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64j2q\" (UniqueName: \"kubernetes.io/projected/5b72272f-1227-401e-bec7-56182f8c1676-kube-api-access-64j2q\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:38:43.494748 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:43.494708 4376 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b72272f-1227-401e-bec7-56182f8c1676-tls-certs\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:38:43.958927 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:43.958891 4376 generic.go:356] "Generic (PLEG): container finished" podID="5b72272f-1227-401e-bec7-56182f8c1676" containerID="242a852c6e0a38931e92511cfb8ede33e98a65859b0dfc9faa3b6515218f81d3" exitCode=0 Oct 31 04:38:43.964350 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:43.960493 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-9633f0/auth-test-model-9633f0-kserve-6cbd55ffd9-z9rjx" Oct 31 04:38:43.986401 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:43.964592 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-9633f0/auth-test-model-9633f0-kserve-6cbd55ffd9-z9rjx" event={"ID":"5b72272f-1227-401e-bec7-56182f8c1676","Type":"ContainerDied","Data":"242a852c6e0a38931e92511cfb8ede33e98a65859b0dfc9faa3b6515218f81d3"} Oct 31 04:38:43.986401 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:43.964782 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-9633f0/auth-test-model-9633f0-kserve-6cbd55ffd9-z9rjx" event={"ID":"5b72272f-1227-401e-bec7-56182f8c1676","Type":"ContainerDied","Data":"83f97ef109af6888444a7318d395e4f49dd59628d9b4e2ebe728a06295730d9f"} Oct 31 04:38:43.986401 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:43.964815 4376 scope.go:122] "RemoveContainer" containerID="242a852c6e0a38931e92511cfb8ede33e98a65859b0dfc9faa3b6515218f81d3" Oct 31 04:38:44.019386 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:44.016861 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 04:38:44.028665 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:44.028101 4376 scope.go:122] "RemoveContainer" containerID="242a852c6e0a38931e92511cfb8ede33e98a65859b0dfc9faa3b6515218f81d3" Oct 31 04:38:44.028747 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:38:44.028671 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"242a852c6e0a38931e92511cfb8ede33e98a65859b0dfc9faa3b6515218f81d3\": container with ID starting with 242a852c6e0a38931e92511cfb8ede33e98a65859b0dfc9faa3b6515218f81d3 not found: ID does not exist" containerID="242a852c6e0a38931e92511cfb8ede33e98a65859b0dfc9faa3b6515218f81d3" Oct 31 04:38:44.028747 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:44.028703 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"242a852c6e0a38931e92511cfb8ede33e98a65859b0dfc9faa3b6515218f81d3"} err="failed to get container status \"242a852c6e0a38931e92511cfb8ede33e98a65859b0dfc9faa3b6515218f81d3\": rpc error: code = NotFound desc = could not find container \"242a852c6e0a38931e92511cfb8ede33e98a65859b0dfc9faa3b6515218f81d3\": container with ID starting with 242a852c6e0a38931e92511cfb8ede33e98a65859b0dfc9faa3b6515218f81d3 not found: ID does not exist" Oct 31 04:38:44.072417 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:44.072387 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-9633f0/auth-test-model-9633f0-kserve-6cbd55ffd9-z9rjx"] Oct 31 04:38:44.077710 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:44.077680 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w4-9633f0/auth-test-model-9633f0-kserve-6cbd55ffd9-z9rjx"] Oct 31 04:38:44.444474 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:44.444444 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b72272f-1227-401e-bec7-56182f8c1676" path="/var/lib/kubelet/pods/5b72272f-1227-401e-bec7-56182f8c1676/volumes" Oct 31 04:38:44.444966 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:44.444951 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d001c91-1967-4fb1-94f4-8f8a8db7216b" path="/var/lib/kubelet/pods/8d001c91-1967-4fb1-94f4-8f8a8db7216b/volumes" Oct 31 04:38:47.941753 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:47.941715 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-654741-5dbd945cf9-wrd4x"] Oct 31 04:38:47.942394 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:47.942378 4376 cpu_manager.go:414] "containerMap: removing container" podUID="5b72272f-1227-401e-bec7-56182f8c1676" containerName="main" Oct 31 04:38:47.942461 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:47.942397 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b72272f-1227-401e-bec7-56182f8c1676" containerName="main" Oct 31 04:38:47.942461 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:47.942435 4376 cpu_manager.go:414] "containerMap: removing container" podUID="8d001c91-1967-4fb1-94f4-8f8a8db7216b" containerName="istio-proxy" Oct 31 04:38:47.942461 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:47.942441 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d001c91-1967-4fb1-94f4-8f8a8db7216b" containerName="istio-proxy" Oct 31 04:38:47.942557 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:47.942532 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d001c91-1967-4fb1-94f4-8f8a8db7216b" containerName="istio-proxy" Oct 31 04:38:47.942557 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:47.942544 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b72272f-1227-401e-bec7-56182f8c1676" containerName="main" Oct 31 04:38:47.947577 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:47.947561 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-654741-5dbd945cf9-wrd4x" Oct 31 04:38:47.975951 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:47.975902 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-654741-dockercfg-wx762\"" Oct 31 04:38:47.976131 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:47.976012 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-654741\"" Oct 31 04:38:47.991318 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:47.981310 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-654741-5dbd945cf9-wrd4x"] Oct 31 04:38:48.012353 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:48.009534 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3b9ba37-7625-40d7-be02-793a54367260-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-654741-5dbd945cf9-wrd4x\" (UID: \"f3b9ba37-7625-40d7-be02-793a54367260\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-654741-5dbd945cf9-wrd4x" Oct 31 04:38:48.023349 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:48.015591 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txmq6\" (UniqueName: \"kubernetes.io/projected/f3b9ba37-7625-40d7-be02-793a54367260-kube-api-access-txmq6\") pod \"payload-pre-processing-e2e-extmod-a-654741-5dbd945cf9-wrd4x\" (UID: \"f3b9ba37-7625-40d7-be02-793a54367260\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-654741-5dbd945cf9-wrd4x" Oct 31 04:38:48.145353 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:48.141487 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txmq6\" (UniqueName: \"kubernetes.io/projected/f3b9ba37-7625-40d7-be02-793a54367260-kube-api-access-txmq6\") pod \"payload-pre-processing-e2e-extmod-a-654741-5dbd945cf9-wrd4x\" (UID: \"f3b9ba37-7625-40d7-be02-793a54367260\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-654741-5dbd945cf9-wrd4x" Oct 31 04:38:48.145353 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:48.141617 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3b9ba37-7625-40d7-be02-793a54367260-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-654741-5dbd945cf9-wrd4x\" (UID: \"f3b9ba37-7625-40d7-be02-793a54367260\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-654741-5dbd945cf9-wrd4x" Oct 31 04:38:48.195651 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:48.183085 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3b9ba37-7625-40d7-be02-793a54367260-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-654741-5dbd945cf9-wrd4x\" (UID: \"f3b9ba37-7625-40d7-be02-793a54367260\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-654741-5dbd945cf9-wrd4x" Oct 31 04:38:48.215401 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:48.214132 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-txmq6\" (UniqueName: \"kubernetes.io/projected/f3b9ba37-7625-40d7-be02-793a54367260-kube-api-access-txmq6\") pod \"payload-pre-processing-e2e-extmod-a-654741-5dbd945cf9-wrd4x\" (UID: \"f3b9ba37-7625-40d7-be02-793a54367260\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-654741-5dbd945cf9-wrd4x" Oct 31 04:38:48.216354 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:48.215605 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-654741-785568f87c-8k8tt"] Oct 31 04:38:48.237988 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:48.236785 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-654741-785568f87c-8k8tt"] Oct 31 04:38:48.237988 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:48.236894 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-654741-785568f87c-8k8tt" Oct 31 04:38:48.262465 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:48.262434 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-654741-5dbd945cf9-wrd4x" Oct 31 04:38:48.344401 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:48.344363 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4562\" (UniqueName: \"kubernetes.io/projected/2755051f-49ce-4ffe-8c54-0ae241bd45c2-kube-api-access-w4562\") pod \"payload-processing-e2e-extmod-a-654741-785568f87c-8k8tt\" (UID: \"2755051f-49ce-4ffe-8c54-0ae241bd45c2\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-654741-785568f87c-8k8tt" Oct 31 04:38:48.344576 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:48.344411 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2755051f-49ce-4ffe-8c54-0ae241bd45c2-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-654741-785568f87c-8k8tt\" (UID: \"2755051f-49ce-4ffe-8c54-0ae241bd45c2\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-654741-785568f87c-8k8tt" Oct 31 04:38:48.445871 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:48.445790 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4562\" (UniqueName: \"kubernetes.io/projected/2755051f-49ce-4ffe-8c54-0ae241bd45c2-kube-api-access-w4562\") pod \"payload-processing-e2e-extmod-a-654741-785568f87c-8k8tt\" (UID: \"2755051f-49ce-4ffe-8c54-0ae241bd45c2\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-654741-785568f87c-8k8tt" Oct 31 04:38:48.445871 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:48.445836 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2755051f-49ce-4ffe-8c54-0ae241bd45c2-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-654741-785568f87c-8k8tt\" (UID: \"2755051f-49ce-4ffe-8c54-0ae241bd45c2\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-654741-785568f87c-8k8tt" Oct 31 04:38:48.446566 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:48.446548 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2755051f-49ce-4ffe-8c54-0ae241bd45c2-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-654741-785568f87c-8k8tt\" (UID: \"2755051f-49ce-4ffe-8c54-0ae241bd45c2\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-654741-785568f87c-8k8tt" Oct 31 04:38:48.469569 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:48.469538 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4562\" (UniqueName: \"kubernetes.io/projected/2755051f-49ce-4ffe-8c54-0ae241bd45c2-kube-api-access-w4562\") pod \"payload-processing-e2e-extmod-a-654741-785568f87c-8k8tt\" (UID: \"2755051f-49ce-4ffe-8c54-0ae241bd45c2\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-654741-785568f87c-8k8tt" Oct 31 04:38:48.555498 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:48.555465 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-654741-785568f87c-8k8tt" Oct 31 04:38:48.566636 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:48.566607 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-654741-5dbd945cf9-wrd4x"] Oct 31 04:38:48.567776 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:38:48.567743 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3b9ba37_7625_40d7_be02_793a54367260.slice/crio-0c01a94ed567d681859d37e1578877064d6b1141c12b81ebea90670ce0d9c5bd WatchSource:0}: Error finding container 0c01a94ed567d681859d37e1578877064d6b1141c12b81ebea90670ce0d9c5bd: Status 404 returned error can't find the container with id 0c01a94ed567d681859d37e1578877064d6b1141c12b81ebea90670ce0d9c5bd Oct 31 04:38:48.917942 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:38:48.917906 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2755051f_49ce_4ffe_8c54_0ae241bd45c2.slice/crio-0beafa35131a65a4398a5360463dd34e96658bbd4c3ffa529317fc41f5d4e111 WatchSource:0}: Error finding container 0beafa35131a65a4398a5360463dd34e96658bbd4c3ffa529317fc41f5d4e111: Status 404 returned error can't find the container with id 0beafa35131a65a4398a5360463dd34e96658bbd4c3ffa529317fc41f5d4e111 Oct 31 04:38:48.918269 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:48.918237 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-654741-785568f87c-8k8tt"] Oct 31 04:38:49.224386 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:49.224322 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-654741-785568f87c-8k8tt" event={"ID":"2755051f-49ce-4ffe-8c54-0ae241bd45c2","Type":"ContainerStarted","Data":"ec39282578e488f88ddb62e156380467b18ca902f9262892d32aa84ec924ca21"} Oct 31 04:38:49.224843 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:49.224410 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-654741-785568f87c-8k8tt" Oct 31 04:38:49.224843 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:49.224434 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-654741-785568f87c-8k8tt" event={"ID":"2755051f-49ce-4ffe-8c54-0ae241bd45c2","Type":"ContainerStarted","Data":"0beafa35131a65a4398a5360463dd34e96658bbd4c3ffa529317fc41f5d4e111"} Oct 31 04:38:49.226096 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:49.226061 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-654741-5dbd945cf9-wrd4x" event={"ID":"f3b9ba37-7625-40d7-be02-793a54367260","Type":"ContainerStarted","Data":"a9ba12a8c50a9828306739d887f380ca0a746cae93dca61421acb423f3ed4041"} Oct 31 04:38:49.226208 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:49.226123 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-654741-5dbd945cf9-wrd4x" Oct 31 04:38:49.226208 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:49.226145 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-654741-5dbd945cf9-wrd4x" event={"ID":"f3b9ba37-7625-40d7-be02-793a54367260","Type":"ContainerStarted","Data":"0c01a94ed567d681859d37e1578877064d6b1141c12b81ebea90670ce0d9c5bd"} Oct 31 04:38:49.268449 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:49.268389 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-654741-785568f87c-8k8tt" podStartSLOduration=2.26836962 podStartE2EDuration="2.26836962s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:38:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:38:49.267612104 +0000 UTC m=+2076.981946745" watchObservedRunningTime="2026-10-31 04:38:49.26836962 +0000 UTC m=+2076.982704243" Oct 31 04:38:49.311993 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:49.311936 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-654741-5dbd945cf9-wrd4x" podStartSLOduration=2.311922186 podStartE2EDuration="2.311922186s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:38:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:38:49.308242532 +0000 UTC m=+2077.022577149" watchObservedRunningTime="2026-10-31 04:38:49.311922186 +0000 UTC m=+2077.026256803" Oct 31 04:38:52.624850 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:52.624820 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-afc519-openshift-default-dfdbf9b97-qvdkl"] Oct 31 04:38:52.629754 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:52.629726 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-afc519-openshift-default-dfdbf9b97-qvdkl" Oct 31 04:38:52.676880 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:52.676837 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-afc519-openshift-default-dfdbf9b97-qvdkl"] Oct 31 04:38:52.684803 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:52.684771 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-afc519-openshift-default-dockercfg-n5g5r\"" Oct 31 04:38:52.837358 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:52.822316 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80cdb0ed-7002-415c-9916-7e1650f0481f-istio-podinfo\") pod \"e2e-extmod-a-afc519-openshift-default-dfdbf9b97-qvdkl\" (UID: \"80cdb0ed-7002-415c-9916-7e1650f0481f\") " pod="openshift-ingress/e2e-extmod-a-afc519-openshift-default-dfdbf9b97-qvdkl" Oct 31 04:38:52.837358 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:52.822436 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80cdb0ed-7002-415c-9916-7e1650f0481f-istio-token\") pod \"e2e-extmod-a-afc519-openshift-default-dfdbf9b97-qvdkl\" (UID: \"80cdb0ed-7002-415c-9916-7e1650f0481f\") " pod="openshift-ingress/e2e-extmod-a-afc519-openshift-default-dfdbf9b97-qvdkl" Oct 31 04:38:52.837358 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:52.822477 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80cdb0ed-7002-415c-9916-7e1650f0481f-credential-socket\") pod \"e2e-extmod-a-afc519-openshift-default-dfdbf9b97-qvdkl\" (UID: \"80cdb0ed-7002-415c-9916-7e1650f0481f\") " pod="openshift-ingress/e2e-extmod-a-afc519-openshift-default-dfdbf9b97-qvdkl" Oct 31 04:38:52.837358 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:52.822536 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80cdb0ed-7002-415c-9916-7e1650f0481f-istiod-ca-cert\") pod \"e2e-extmod-a-afc519-openshift-default-dfdbf9b97-qvdkl\" (UID: \"80cdb0ed-7002-415c-9916-7e1650f0481f\") " pod="openshift-ingress/e2e-extmod-a-afc519-openshift-default-dfdbf9b97-qvdkl" Oct 31 04:38:52.837358 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:52.822567 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80cdb0ed-7002-415c-9916-7e1650f0481f-istio-data\") pod \"e2e-extmod-a-afc519-openshift-default-dfdbf9b97-qvdkl\" (UID: \"80cdb0ed-7002-415c-9916-7e1650f0481f\") " pod="openshift-ingress/e2e-extmod-a-afc519-openshift-default-dfdbf9b97-qvdkl" Oct 31 04:38:52.837358 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:52.822677 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80cdb0ed-7002-415c-9916-7e1650f0481f-workload-certs\") pod \"e2e-extmod-a-afc519-openshift-default-dfdbf9b97-qvdkl\" (UID: \"80cdb0ed-7002-415c-9916-7e1650f0481f\") " pod="openshift-ingress/e2e-extmod-a-afc519-openshift-default-dfdbf9b97-qvdkl" Oct 31 04:38:52.837358 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:52.822733 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80cdb0ed-7002-415c-9916-7e1650f0481f-istio-envoy\") pod \"e2e-extmod-a-afc519-openshift-default-dfdbf9b97-qvdkl\" (UID: \"80cdb0ed-7002-415c-9916-7e1650f0481f\") " pod="openshift-ingress/e2e-extmod-a-afc519-openshift-default-dfdbf9b97-qvdkl" Oct 31 04:38:52.837358 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:52.822766 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80cdb0ed-7002-415c-9916-7e1650f0481f-workload-socket\") pod \"e2e-extmod-a-afc519-openshift-default-dfdbf9b97-qvdkl\" (UID: \"80cdb0ed-7002-415c-9916-7e1650f0481f\") " pod="openshift-ingress/e2e-extmod-a-afc519-openshift-default-dfdbf9b97-qvdkl" Oct 31 04:38:52.837358 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:52.822805 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqdqk\" (UniqueName: \"kubernetes.io/projected/80cdb0ed-7002-415c-9916-7e1650f0481f-kube-api-access-jqdqk\") pod \"e2e-extmod-a-afc519-openshift-default-dfdbf9b97-qvdkl\" (UID: \"80cdb0ed-7002-415c-9916-7e1650f0481f\") " pod="openshift-ingress/e2e-extmod-a-afc519-openshift-default-dfdbf9b97-qvdkl" Oct 31 04:38:52.927512 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:52.927402 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80cdb0ed-7002-415c-9916-7e1650f0481f-istio-envoy\") pod \"e2e-extmod-a-afc519-openshift-default-dfdbf9b97-qvdkl\" (UID: \"80cdb0ed-7002-415c-9916-7e1650f0481f\") " pod="openshift-ingress/e2e-extmod-a-afc519-openshift-default-dfdbf9b97-qvdkl" Oct 31 04:38:52.927693 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:52.927549 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80cdb0ed-7002-415c-9916-7e1650f0481f-workload-socket\") pod \"e2e-extmod-a-afc519-openshift-default-dfdbf9b97-qvdkl\" (UID: \"80cdb0ed-7002-415c-9916-7e1650f0481f\") " pod="openshift-ingress/e2e-extmod-a-afc519-openshift-default-dfdbf9b97-qvdkl" Oct 31 04:38:52.927693 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:52.927599 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqdqk\" (UniqueName: \"kubernetes.io/projected/80cdb0ed-7002-415c-9916-7e1650f0481f-kube-api-access-jqdqk\") pod \"e2e-extmod-a-afc519-openshift-default-dfdbf9b97-qvdkl\" (UID: \"80cdb0ed-7002-415c-9916-7e1650f0481f\") " pod="openshift-ingress/e2e-extmod-a-afc519-openshift-default-dfdbf9b97-qvdkl" Oct 31 04:38:52.927825 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:52.927756 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80cdb0ed-7002-415c-9916-7e1650f0481f-istio-podinfo\") pod \"e2e-extmod-a-afc519-openshift-default-dfdbf9b97-qvdkl\" (UID: \"80cdb0ed-7002-415c-9916-7e1650f0481f\") " pod="openshift-ingress/e2e-extmod-a-afc519-openshift-default-dfdbf9b97-qvdkl" Oct 31 04:38:52.927883 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:52.927830 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80cdb0ed-7002-415c-9916-7e1650f0481f-istio-token\") pod \"e2e-extmod-a-afc519-openshift-default-dfdbf9b97-qvdkl\" (UID: \"80cdb0ed-7002-415c-9916-7e1650f0481f\") " pod="openshift-ingress/e2e-extmod-a-afc519-openshift-default-dfdbf9b97-qvdkl" Oct 31 04:38:52.927883 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:52.927866 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80cdb0ed-7002-415c-9916-7e1650f0481f-credential-socket\") pod \"e2e-extmod-a-afc519-openshift-default-dfdbf9b97-qvdkl\" (UID: \"80cdb0ed-7002-415c-9916-7e1650f0481f\") " pod="openshift-ingress/e2e-extmod-a-afc519-openshift-default-dfdbf9b97-qvdkl" Oct 31 04:38:52.927995 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:52.927922 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80cdb0ed-7002-415c-9916-7e1650f0481f-istiod-ca-cert\") pod \"e2e-extmod-a-afc519-openshift-default-dfdbf9b97-qvdkl\" (UID: \"80cdb0ed-7002-415c-9916-7e1650f0481f\") " pod="openshift-ingress/e2e-extmod-a-afc519-openshift-default-dfdbf9b97-qvdkl" Oct 31 04:38:52.927995 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:52.927951 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80cdb0ed-7002-415c-9916-7e1650f0481f-istio-data\") pod \"e2e-extmod-a-afc519-openshift-default-dfdbf9b97-qvdkl\" (UID: \"80cdb0ed-7002-415c-9916-7e1650f0481f\") " pod="openshift-ingress/e2e-extmod-a-afc519-openshift-default-dfdbf9b97-qvdkl" Oct 31 04:38:52.928107 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:52.928066 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80cdb0ed-7002-415c-9916-7e1650f0481f-workload-certs\") pod \"e2e-extmod-a-afc519-openshift-default-dfdbf9b97-qvdkl\" (UID: \"80cdb0ed-7002-415c-9916-7e1650f0481f\") " pod="openshift-ingress/e2e-extmod-a-afc519-openshift-default-dfdbf9b97-qvdkl" Oct 31 04:38:52.928655 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:52.928624 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80cdb0ed-7002-415c-9916-7e1650f0481f-workload-certs\") pod \"e2e-extmod-a-afc519-openshift-default-dfdbf9b97-qvdkl\" (UID: \"80cdb0ed-7002-415c-9916-7e1650f0481f\") " pod="openshift-ingress/e2e-extmod-a-afc519-openshift-default-dfdbf9b97-qvdkl" Oct 31 04:38:52.929604 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:52.929577 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80cdb0ed-7002-415c-9916-7e1650f0481f-workload-socket\") pod \"e2e-extmod-a-afc519-openshift-default-dfdbf9b97-qvdkl\" (UID: \"80cdb0ed-7002-415c-9916-7e1650f0481f\") " pod="openshift-ingress/e2e-extmod-a-afc519-openshift-default-dfdbf9b97-qvdkl" Oct 31 04:38:52.929766 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:52.929738 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80cdb0ed-7002-415c-9916-7e1650f0481f-credential-socket\") pod \"e2e-extmod-a-afc519-openshift-default-dfdbf9b97-qvdkl\" (UID: \"80cdb0ed-7002-415c-9916-7e1650f0481f\") " pod="openshift-ingress/e2e-extmod-a-afc519-openshift-default-dfdbf9b97-qvdkl" Oct 31 04:38:52.932993 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:52.930596 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80cdb0ed-7002-415c-9916-7e1650f0481f-istio-data\") pod \"e2e-extmod-a-afc519-openshift-default-dfdbf9b97-qvdkl\" (UID: \"80cdb0ed-7002-415c-9916-7e1650f0481f\") " pod="openshift-ingress/e2e-extmod-a-afc519-openshift-default-dfdbf9b97-qvdkl" Oct 31 04:38:52.932993 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:52.931576 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80cdb0ed-7002-415c-9916-7e1650f0481f-istiod-ca-cert\") pod \"e2e-extmod-a-afc519-openshift-default-dfdbf9b97-qvdkl\" (UID: \"80cdb0ed-7002-415c-9916-7e1650f0481f\") " pod="openshift-ingress/e2e-extmod-a-afc519-openshift-default-dfdbf9b97-qvdkl" Oct 31 04:38:52.937369 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:52.937094 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80cdb0ed-7002-415c-9916-7e1650f0481f-istio-envoy\") pod \"e2e-extmod-a-afc519-openshift-default-dfdbf9b97-qvdkl\" (UID: \"80cdb0ed-7002-415c-9916-7e1650f0481f\") " pod="openshift-ingress/e2e-extmod-a-afc519-openshift-default-dfdbf9b97-qvdkl" Oct 31 04:38:52.943285 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:52.943223 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80cdb0ed-7002-415c-9916-7e1650f0481f-istio-podinfo\") pod \"e2e-extmod-a-afc519-openshift-default-dfdbf9b97-qvdkl\" (UID: \"80cdb0ed-7002-415c-9916-7e1650f0481f\") " pod="openshift-ingress/e2e-extmod-a-afc519-openshift-default-dfdbf9b97-qvdkl" Oct 31 04:38:52.972372 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:52.970224 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqdqk\" (UniqueName: \"kubernetes.io/projected/80cdb0ed-7002-415c-9916-7e1650f0481f-kube-api-access-jqdqk\") pod \"e2e-extmod-a-afc519-openshift-default-dfdbf9b97-qvdkl\" (UID: \"80cdb0ed-7002-415c-9916-7e1650f0481f\") " pod="openshift-ingress/e2e-extmod-a-afc519-openshift-default-dfdbf9b97-qvdkl" Oct 31 04:38:52.979733 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:52.978655 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80cdb0ed-7002-415c-9916-7e1650f0481f-istio-token\") pod \"e2e-extmod-a-afc519-openshift-default-dfdbf9b97-qvdkl\" (UID: \"80cdb0ed-7002-415c-9916-7e1650f0481f\") " pod="openshift-ingress/e2e-extmod-a-afc519-openshift-default-dfdbf9b97-qvdkl" Oct 31 04:38:53.246687 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:53.246598 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-afc519-openshift-default-dfdbf9b97-qvdkl" Oct 31 04:38:53.567300 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:38:53.567267 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80cdb0ed_7002_415c_9916_7e1650f0481f.slice/crio-7231d9c2596b6d0665d5ee72a2280c2236a915adb081f2a04af30e3a21654cb3 WatchSource:0}: Error finding container 7231d9c2596b6d0665d5ee72a2280c2236a915adb081f2a04af30e3a21654cb3: Status 404 returned error can't find the container with id 7231d9c2596b6d0665d5ee72a2280c2236a915adb081f2a04af30e3a21654cb3 Oct 31 04:38:53.569152 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:53.569122 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-afc519-openshift-default-dfdbf9b97-qvdkl"] Oct 31 04:38:53.569996 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:53.569959 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:38:53.570111 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:53.570036 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:38:53.570111 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:53.570086 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:38:53.881028 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:53.880994 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-np274"] Oct 31 04:38:53.885668 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:53.885640 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-np274" Oct 31 04:38:53.903886 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:53.903844 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-57dc6d-openshift-default-dockercfg-4gkzk\"" Oct 31 04:38:53.919640 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:53.919601 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-np274"] Oct 31 04:38:53.950909 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:53.940621 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0de4ffc1-c510-4022-b7be-3f8f13360f76-workload-certs\") pod \"e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-np274\" (UID: \"0de4ffc1-c510-4022-b7be-3f8f13360f76\") " pod="openshift-ingress/e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-np274" Oct 31 04:38:53.950909 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:53.940711 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0de4ffc1-c510-4022-b7be-3f8f13360f76-credential-socket\") pod \"e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-np274\" (UID: \"0de4ffc1-c510-4022-b7be-3f8f13360f76\") " pod="openshift-ingress/e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-np274" Oct 31 04:38:53.950909 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:53.940740 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xf8b\" (UniqueName: \"kubernetes.io/projected/0de4ffc1-c510-4022-b7be-3f8f13360f76-kube-api-access-7xf8b\") pod \"e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-np274\" (UID: \"0de4ffc1-c510-4022-b7be-3f8f13360f76\") " pod="openshift-ingress/e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-np274" Oct 31 04:38:53.950909 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:53.940794 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0de4ffc1-c510-4022-b7be-3f8f13360f76-istiod-ca-cert\") pod \"e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-np274\" (UID: \"0de4ffc1-c510-4022-b7be-3f8f13360f76\") " pod="openshift-ingress/e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-np274" Oct 31 04:38:53.950909 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:53.940952 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0de4ffc1-c510-4022-b7be-3f8f13360f76-istio-token\") pod \"e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-np274\" (UID: \"0de4ffc1-c510-4022-b7be-3f8f13360f76\") " pod="openshift-ingress/e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-np274" Oct 31 04:38:53.950909 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:53.940991 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0de4ffc1-c510-4022-b7be-3f8f13360f76-istio-envoy\") pod \"e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-np274\" (UID: \"0de4ffc1-c510-4022-b7be-3f8f13360f76\") " pod="openshift-ingress/e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-np274" Oct 31 04:38:53.950909 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:53.941049 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0de4ffc1-c510-4022-b7be-3f8f13360f76-istio-data\") pod \"e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-np274\" (UID: \"0de4ffc1-c510-4022-b7be-3f8f13360f76\") " pod="openshift-ingress/e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-np274" Oct 31 04:38:53.950909 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:53.941094 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0de4ffc1-c510-4022-b7be-3f8f13360f76-istio-podinfo\") pod \"e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-np274\" (UID: \"0de4ffc1-c510-4022-b7be-3f8f13360f76\") " pod="openshift-ingress/e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-np274" Oct 31 04:38:53.950909 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:53.941128 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0de4ffc1-c510-4022-b7be-3f8f13360f76-workload-socket\") pod \"e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-np274\" (UID: \"0de4ffc1-c510-4022-b7be-3f8f13360f76\") " pod="openshift-ingress/e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-np274" Oct 31 04:38:54.042854 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:54.042817 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0de4ffc1-c510-4022-b7be-3f8f13360f76-istio-token\") pod \"e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-np274\" (UID: \"0de4ffc1-c510-4022-b7be-3f8f13360f76\") " pod="openshift-ingress/e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-np274" Oct 31 04:38:54.042854 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:54.042867 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0de4ffc1-c510-4022-b7be-3f8f13360f76-istio-envoy\") pod \"e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-np274\" (UID: \"0de4ffc1-c510-4022-b7be-3f8f13360f76\") " pod="openshift-ingress/e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-np274" Oct 31 04:38:54.043113 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:54.042904 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0de4ffc1-c510-4022-b7be-3f8f13360f76-istio-data\") pod \"e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-np274\" (UID: \"0de4ffc1-c510-4022-b7be-3f8f13360f76\") " pod="openshift-ingress/e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-np274" Oct 31 04:38:54.043113 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:54.042920 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0de4ffc1-c510-4022-b7be-3f8f13360f76-istio-podinfo\") pod \"e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-np274\" (UID: \"0de4ffc1-c510-4022-b7be-3f8f13360f76\") " pod="openshift-ingress/e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-np274" Oct 31 04:38:54.043113 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:54.042940 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0de4ffc1-c510-4022-b7be-3f8f13360f76-workload-socket\") pod \"e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-np274\" (UID: \"0de4ffc1-c510-4022-b7be-3f8f13360f76\") " pod="openshift-ingress/e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-np274" Oct 31 04:38:54.043113 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:54.042993 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0de4ffc1-c510-4022-b7be-3f8f13360f76-workload-certs\") pod \"e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-np274\" (UID: \"0de4ffc1-c510-4022-b7be-3f8f13360f76\") " pod="openshift-ingress/e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-np274" Oct 31 04:38:54.043113 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:54.043061 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0de4ffc1-c510-4022-b7be-3f8f13360f76-credential-socket\") pod \"e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-np274\" (UID: \"0de4ffc1-c510-4022-b7be-3f8f13360f76\") " pod="openshift-ingress/e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-np274" Oct 31 04:38:54.043113 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:54.043089 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xf8b\" (UniqueName: \"kubernetes.io/projected/0de4ffc1-c510-4022-b7be-3f8f13360f76-kube-api-access-7xf8b\") pod \"e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-np274\" (UID: \"0de4ffc1-c510-4022-b7be-3f8f13360f76\") " pod="openshift-ingress/e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-np274" Oct 31 04:38:54.043465 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:54.043129 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0de4ffc1-c510-4022-b7be-3f8f13360f76-istiod-ca-cert\") pod \"e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-np274\" (UID: \"0de4ffc1-c510-4022-b7be-3f8f13360f76\") " pod="openshift-ingress/e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-np274" Oct 31 04:38:54.043465 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:54.043412 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0de4ffc1-c510-4022-b7be-3f8f13360f76-istio-data\") pod \"e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-np274\" (UID: \"0de4ffc1-c510-4022-b7be-3f8f13360f76\") " pod="openshift-ingress/e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-np274" Oct 31 04:38:54.043595 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:54.043465 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0de4ffc1-c510-4022-b7be-3f8f13360f76-workload-socket\") pod \"e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-np274\" (UID: \"0de4ffc1-c510-4022-b7be-3f8f13360f76\") " pod="openshift-ingress/e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-np274" Oct 31 04:38:54.043709 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:54.043685 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0de4ffc1-c510-4022-b7be-3f8f13360f76-workload-certs\") pod \"e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-np274\" (UID: \"0de4ffc1-c510-4022-b7be-3f8f13360f76\") " pod="openshift-ingress/e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-np274" Oct 31 04:38:54.043773 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:54.043705 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0de4ffc1-c510-4022-b7be-3f8f13360f76-credential-socket\") pod \"e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-np274\" (UID: \"0de4ffc1-c510-4022-b7be-3f8f13360f76\") " pod="openshift-ingress/e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-np274" Oct 31 04:38:54.043813 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:54.043776 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0de4ffc1-c510-4022-b7be-3f8f13360f76-istiod-ca-cert\") pod \"e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-np274\" (UID: \"0de4ffc1-c510-4022-b7be-3f8f13360f76\") " pod="openshift-ingress/e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-np274" Oct 31 04:38:54.045938 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:54.045911 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0de4ffc1-c510-4022-b7be-3f8f13360f76-istio-envoy\") pod \"e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-np274\" (UID: \"0de4ffc1-c510-4022-b7be-3f8f13360f76\") " pod="openshift-ingress/e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-np274" Oct 31 04:38:54.046124 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:54.046108 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0de4ffc1-c510-4022-b7be-3f8f13360f76-istio-podinfo\") pod \"e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-np274\" (UID: \"0de4ffc1-c510-4022-b7be-3f8f13360f76\") " pod="openshift-ingress/e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-np274" Oct 31 04:38:54.067082 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:54.067052 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0de4ffc1-c510-4022-b7be-3f8f13360f76-istio-token\") pod \"e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-np274\" (UID: \"0de4ffc1-c510-4022-b7be-3f8f13360f76\") " pod="openshift-ingress/e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-np274" Oct 31 04:38:54.067232 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:54.067210 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xf8b\" (UniqueName: \"kubernetes.io/projected/0de4ffc1-c510-4022-b7be-3f8f13360f76-kube-api-access-7xf8b\") pod \"e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-np274\" (UID: \"0de4ffc1-c510-4022-b7be-3f8f13360f76\") " pod="openshift-ingress/e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-np274" Oct 31 04:38:54.202898 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:54.202813 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-np274" Oct 31 04:38:54.344320 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:54.338096 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-afc519-openshift-default-dfdbf9b97-qvdkl" event={"ID":"80cdb0ed-7002-415c-9916-7e1650f0481f","Type":"ContainerStarted","Data":"f7752764b4fabf3ad4cc5192c8e0217f044eed7de2108ea7b914eecd9885e170"} Oct 31 04:38:54.344320 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:54.338167 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-afc519-openshift-default-dfdbf9b97-qvdkl" event={"ID":"80cdb0ed-7002-415c-9916-7e1650f0481f","Type":"ContainerStarted","Data":"7231d9c2596b6d0665d5ee72a2280c2236a915adb081f2a04af30e3a21654cb3"} Oct 31 04:38:54.391221 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:54.391160 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-afc519-openshift-default-dfdbf9b97-qvdkl" podStartSLOduration=2.3911449 podStartE2EDuration="2.3911449s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:38:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:38:54.387433892 +0000 UTC m=+2082.101768530" watchObservedRunningTime="2026-10-31 04:38:54.3911449 +0000 UTC m=+2082.105479517" Oct 31 04:38:54.558801 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:38:54.558446 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0de4ffc1_c510_4022_b7be_3f8f13360f76.slice/crio-4d14ca7e5d16d3e77c692c07da3e3bd77db11f1f3b84e1460e1680b078dc5b22 WatchSource:0}: Error finding container 4d14ca7e5d16d3e77c692c07da3e3bd77db11f1f3b84e1460e1680b078dc5b22: Status 404 returned error can't find the container with id 4d14ca7e5d16d3e77c692c07da3e3bd77db11f1f3b84e1460e1680b078dc5b22 Oct 31 04:38:54.564192 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:54.564042 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-np274"] Oct 31 04:38:54.565155 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:54.565109 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:38:54.565286 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:54.565202 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:38:54.565286 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:54.565242 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:38:55.247508 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:55.247474 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-afc519-openshift-default-dfdbf9b97-qvdkl" Oct 31 04:38:55.253259 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:55.253232 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-afc519-openshift-default-dfdbf9b97-qvdkl" Oct 31 04:38:55.353419 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:55.353385 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-np274" event={"ID":"0de4ffc1-c510-4022-b7be-3f8f13360f76","Type":"ContainerStarted","Data":"218edb5fe0fbcf9808d1435b2dd347951d228dba57f508800d6e58852999807e"} Oct 31 04:38:55.353606 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:55.353447 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-afc519-openshift-default-dfdbf9b97-qvdkl" Oct 31 04:38:55.353606 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:55.353477 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-np274" event={"ID":"0de4ffc1-c510-4022-b7be-3f8f13360f76","Type":"ContainerStarted","Data":"4d14ca7e5d16d3e77c692c07da3e3bd77db11f1f3b84e1460e1680b078dc5b22"} Oct 31 04:38:55.354655 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:55.354628 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-afc519-openshift-default-dfdbf9b97-qvdkl" Oct 31 04:38:55.406512 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:55.406454 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-np274" podStartSLOduration=2.40643457 podStartE2EDuration="2.40643457s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:38:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:38:55.403641182 +0000 UTC m=+2083.117975800" watchObservedRunningTime="2026-10-31 04:38:55.40643457 +0000 UTC m=+2083.120769347" Oct 31 04:38:56.203232 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:56.203195 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-np274" Oct 31 04:38:56.204652 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:56.204623 4376 patch_prober.go:29] interesting pod/e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-np274 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" start-of-body= Oct 31 04:38:56.204798 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:56.204676 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-np274" podUID="0de4ffc1-c510-4022-b7be-3f8f13360f76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" Oct 31 04:38:57.204136 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:57.204099 4376 patch_prober.go:29] interesting pod/e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-np274 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" start-of-body= Oct 31 04:38:57.204577 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:57.204155 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-np274" podUID="0de4ffc1-c510-4022-b7be-3f8f13360f76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" Oct 31 04:38:58.204160 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:58.204125 4376 patch_prober.go:29] interesting pod/e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-np274 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" start-of-body= Oct 31 04:38:58.204666 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:58.204179 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-np274" podUID="0de4ffc1-c510-4022-b7be-3f8f13360f76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" Oct 31 04:38:58.772250 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:58.772214 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-afc519-69cbc98467-l7w68"] Oct 31 04:38:58.776830 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:58.776809 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-afc519-69cbc98467-l7w68" Oct 31 04:38:58.796577 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:58.796544 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-afc519-69cbc98467-l7w68"] Oct 31 04:38:58.804840 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:58.804813 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-afc519-dockercfg-d4k2j\"" Oct 31 04:38:58.805142 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:58.805124 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-afc519\"" Oct 31 04:38:58.833485 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:58.833448 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84d95c12-4dca-44d2-a2cb-fdbfe919135b-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-afc519-69cbc98467-l7w68\" (UID: \"84d95c12-4dca-44d2-a2cb-fdbfe919135b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-afc519-69cbc98467-l7w68" Oct 31 04:38:58.833706 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:58.833616 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpnvh\" (UniqueName: \"kubernetes.io/projected/84d95c12-4dca-44d2-a2cb-fdbfe919135b-kube-api-access-kpnvh\") pod \"payload-pre-processing-e2e-extmod-a-afc519-69cbc98467-l7w68\" (UID: \"84d95c12-4dca-44d2-a2cb-fdbfe919135b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-afc519-69cbc98467-l7w68" Oct 31 04:38:58.968352 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:58.956353 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpnvh\" (UniqueName: \"kubernetes.io/projected/84d95c12-4dca-44d2-a2cb-fdbfe919135b-kube-api-access-kpnvh\") pod \"payload-pre-processing-e2e-extmod-a-afc519-69cbc98467-l7w68\" (UID: \"84d95c12-4dca-44d2-a2cb-fdbfe919135b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-afc519-69cbc98467-l7w68" Oct 31 04:38:58.968352 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:58.956640 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84d95c12-4dca-44d2-a2cb-fdbfe919135b-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-afc519-69cbc98467-l7w68\" (UID: \"84d95c12-4dca-44d2-a2cb-fdbfe919135b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-afc519-69cbc98467-l7w68" Oct 31 04:38:58.968352 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:58.957634 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84d95c12-4dca-44d2-a2cb-fdbfe919135b-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-afc519-69cbc98467-l7w68\" (UID: \"84d95c12-4dca-44d2-a2cb-fdbfe919135b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-afc519-69cbc98467-l7w68" Oct 31 04:38:59.083464 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:59.082416 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-afc519-d746ff6cc-npmp9"] Oct 31 04:38:59.183561 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:59.183525 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpnvh\" (UniqueName: \"kubernetes.io/projected/84d95c12-4dca-44d2-a2cb-fdbfe919135b-kube-api-access-kpnvh\") pod \"payload-pre-processing-e2e-extmod-a-afc519-69cbc98467-l7w68\" (UID: \"84d95c12-4dca-44d2-a2cb-fdbfe919135b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-afc519-69cbc98467-l7w68" Oct 31 04:38:59.187988 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:59.187959 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-afc519-d746ff6cc-npmp9"] Oct 31 04:38:59.188141 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:59.188065 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-afc519-d746ff6cc-npmp9" Oct 31 04:38:59.204196 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:59.204162 4376 patch_prober.go:29] interesting pod/e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-np274 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" start-of-body= Oct 31 04:38:59.204569 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:59.204228 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-np274" podUID="0de4ffc1-c510-4022-b7be-3f8f13360f76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" Oct 31 04:38:59.296286 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:59.296252 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wf84q\" (UniqueName: \"kubernetes.io/projected/e90146f8-ed41-49b8-bf8b-c92bee5f8cdd-kube-api-access-wf84q\") pod \"payload-processing-e2e-extmod-a-afc519-d746ff6cc-npmp9\" (UID: \"e90146f8-ed41-49b8-bf8b-c92bee5f8cdd\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-afc519-d746ff6cc-npmp9" Oct 31 04:38:59.296496 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:59.296365 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e90146f8-ed41-49b8-bf8b-c92bee5f8cdd-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-afc519-d746ff6cc-npmp9\" (UID: \"e90146f8-ed41-49b8-bf8b-c92bee5f8cdd\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-afc519-d746ff6cc-npmp9" Oct 31 04:38:59.392613 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:59.392573 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-afc519-69cbc98467-l7w68" Oct 31 04:38:59.397793 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:59.397767 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wf84q\" (UniqueName: \"kubernetes.io/projected/e90146f8-ed41-49b8-bf8b-c92bee5f8cdd-kube-api-access-wf84q\") pod \"payload-processing-e2e-extmod-a-afc519-d746ff6cc-npmp9\" (UID: \"e90146f8-ed41-49b8-bf8b-c92bee5f8cdd\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-afc519-d746ff6cc-npmp9" Oct 31 04:38:59.397889 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:59.397869 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e90146f8-ed41-49b8-bf8b-c92bee5f8cdd-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-afc519-d746ff6cc-npmp9\" (UID: \"e90146f8-ed41-49b8-bf8b-c92bee5f8cdd\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-afc519-d746ff6cc-npmp9" Oct 31 04:38:59.398554 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:59.398531 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e90146f8-ed41-49b8-bf8b-c92bee5f8cdd-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-afc519-d746ff6cc-npmp9\" (UID: \"e90146f8-ed41-49b8-bf8b-c92bee5f8cdd\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-afc519-d746ff6cc-npmp9" Oct 31 04:38:59.422604 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:59.422541 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wf84q\" (UniqueName: \"kubernetes.io/projected/e90146f8-ed41-49b8-bf8b-c92bee5f8cdd-kube-api-access-wf84q\") pod \"payload-processing-e2e-extmod-a-afc519-d746ff6cc-npmp9\" (UID: \"e90146f8-ed41-49b8-bf8b-c92bee5f8cdd\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-afc519-d746ff6cc-npmp9" Oct 31 04:38:59.505001 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:59.504748 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-afc519-d746ff6cc-npmp9" Oct 31 04:38:59.722417 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:38:59.722380 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84d95c12_4dca_44d2_a2cb_fdbfe919135b.slice/crio-af8b3658eb011aae6b133d27fa92b5fbc7d8fa781c8c3e5502b79d50f67d3564 WatchSource:0}: Error finding container af8b3658eb011aae6b133d27fa92b5fbc7d8fa781c8c3e5502b79d50f67d3564: Status 404 returned error can't find the container with id af8b3658eb011aae6b133d27fa92b5fbc7d8fa781c8c3e5502b79d50f67d3564 Oct 31 04:38:59.723822 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:59.723795 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-afc519-69cbc98467-l7w68"] Oct 31 04:38:59.867531 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:38:59.867493 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-afc519-d746ff6cc-npmp9"] Oct 31 04:38:59.867791 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:38:59.867766 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode90146f8_ed41_49b8_bf8b_c92bee5f8cdd.slice/crio-08694f61e7057f1688d6b89f1a4dab93c4996799c941c7679a38fe83c72f29e7 WatchSource:0}: Error finding container 08694f61e7057f1688d6b89f1a4dab93c4996799c941c7679a38fe83c72f29e7: Status 404 returned error can't find the container with id 08694f61e7057f1688d6b89f1a4dab93c4996799c941c7679a38fe83c72f29e7 Oct 31 04:39:00.203839 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:00.203804 4376 patch_prober.go:29] interesting pod/e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-np274 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" start-of-body= Oct 31 04:39:00.204018 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:00.203856 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-np274" podUID="0de4ffc1-c510-4022-b7be-3f8f13360f76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" Oct 31 04:39:00.234738 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:00.234710 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-654741-5dbd945cf9-wrd4x" Oct 31 04:39:00.235118 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:00.234776 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-654741-785568f87c-8k8tt" Oct 31 04:39:00.531422 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:00.531177 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-afc519-69cbc98467-l7w68" event={"ID":"84d95c12-4dca-44d2-a2cb-fdbfe919135b","Type":"ContainerStarted","Data":"53c4edd38ab68b826ecbed62aa5bff0dd9e26fd76265fa486ad478d6b0967fa0"} Oct 31 04:39:00.531422 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:00.531264 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-afc519-69cbc98467-l7w68" Oct 31 04:39:00.531422 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:00.531292 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-afc519-d746ff6cc-npmp9" Oct 31 04:39:00.531422 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:00.531322 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-afc519-69cbc98467-l7w68" event={"ID":"84d95c12-4dca-44d2-a2cb-fdbfe919135b","Type":"ContainerStarted","Data":"af8b3658eb011aae6b133d27fa92b5fbc7d8fa781c8c3e5502b79d50f67d3564"} Oct 31 04:39:00.531422 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:00.531391 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-afc519-d746ff6cc-npmp9" event={"ID":"e90146f8-ed41-49b8-bf8b-c92bee5f8cdd","Type":"ContainerStarted","Data":"2a7a08c67ee09683a03dcda6277fc82fa6c5690a205555171d927bbb2ef54aa7"} Oct 31 04:39:00.531422 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:00.531415 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-afc519-d746ff6cc-npmp9" event={"ID":"e90146f8-ed41-49b8-bf8b-c92bee5f8cdd","Type":"ContainerStarted","Data":"08694f61e7057f1688d6b89f1a4dab93c4996799c941c7679a38fe83c72f29e7"} Oct 31 04:39:00.585545 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:00.585490 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-afc519-69cbc98467-l7w68" podStartSLOduration=2.58547672 podStartE2EDuration="2.58547672s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:38:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:39:00.585433771 +0000 UTC m=+2088.299768388" watchObservedRunningTime="2026-10-31 04:39:00.58547672 +0000 UTC m=+2088.299811338" Oct 31 04:39:00.627882 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:00.627828 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-afc519-d746ff6cc-npmp9" podStartSLOduration=2.627813657 podStartE2EDuration="2.627813657s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:38:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:39:00.625967537 +0000 UTC m=+2088.340302155" watchObservedRunningTime="2026-10-31 04:39:00.627813657 +0000 UTC m=+2088.342148275" Oct 31 04:39:01.207668 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:01.207637 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-np274" Oct 31 04:39:01.208141 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:01.208122 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-np274" Oct 31 04:39:01.208870 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:01.208851 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-np274" Oct 31 04:39:02.017749 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:02.015982 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-895sj"] Oct 31 04:39:02.017749 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:02.016605 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-895sj" podUID="bd748a1b-27f9-4e1e-9398-e3bff9c73ae7" containerName="istio-proxy" containerID="cri-o://5ccc783fb255eae038c19b151fd480cb4d0a34175fff95d454b54ee0030712e6" gracePeriod=30 Oct 31 04:39:02.045385 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:02.043382 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-2vsk6"] Oct 31 04:39:02.045385 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:02.043630 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-2vsk6" podUID="8682ee97-bdd1-4755-a5c6-9fa9d4120247" containerName="istio-proxy" containerID="cri-o://34ff2ca539c44439336625349c26eb6ef4a8e897cd67580a007655aa862c7fff" gracePeriod=30 Oct 31 04:39:02.078952 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:02.077746 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-zmth9"] Oct 31 04:39:02.078952 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:02.077812 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-bpgxq"] Oct 31 04:39:02.078952 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:02.077964 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-bpgxq" podUID="880c7780-4eac-4a67-b948-3d52a354c554" containerName="istio-proxy" containerID="cri-o://b7f0d327394e7f57beaa3798867e475cf9ee3d8ad80ac864847d019d4ed24535" gracePeriod=30 Oct 31 04:39:02.078952 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:02.078406 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-zmth9" podUID="5d825a50-865d-46c6-b88d-73f7170963c5" containerName="istio-proxy" containerID="cri-o://ef3065d9e6b828d32123237c43ff3949d01ad4aa59f2bd3080d3ca6fffe8548f" gracePeriod=30 Oct 31 04:39:03.084858 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:03.083986 4376 patch_prober.go:29] interesting pod/e2e-worker-w0-965f61-openshift-default-55854964c-zmth9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 04:39:03.084858 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:03.084049 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-zmth9" podUID="5d825a50-865d-46c6-b88d-73f7170963c5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 04:39:06.005811 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:06.005779 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-25f56e-openshift-default-6465df847-6j725"] Oct 31 04:39:06.011562 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:06.011541 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-25f56e-openshift-default-6465df847-6j725" Oct 31 04:39:06.029920 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:06.029861 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-25f56e-openshift-default-dockercfg-9tcc4\"" Oct 31 04:39:06.044144 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:06.044120 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-25f56e-openshift-default-6465df847-6j725"] Oct 31 04:39:06.120665 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:06.120633 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55vn7\" (UniqueName: \"kubernetes.io/projected/caf8772a-4894-4924-9f18-f2b191aadad7-kube-api-access-55vn7\") pod \"e2e-shared-b-w0-25f56e-openshift-default-6465df847-6j725\" (UID: \"caf8772a-4894-4924-9f18-f2b191aadad7\") " pod="openshift-ingress/e2e-shared-b-w0-25f56e-openshift-default-6465df847-6j725" Oct 31 04:39:06.120850 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:06.120768 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/caf8772a-4894-4924-9f18-f2b191aadad7-workload-certs\") pod \"e2e-shared-b-w0-25f56e-openshift-default-6465df847-6j725\" (UID: \"caf8772a-4894-4924-9f18-f2b191aadad7\") " pod="openshift-ingress/e2e-shared-b-w0-25f56e-openshift-default-6465df847-6j725" Oct 31 04:39:06.120850 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:06.120817 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/caf8772a-4894-4924-9f18-f2b191aadad7-istio-envoy\") pod \"e2e-shared-b-w0-25f56e-openshift-default-6465df847-6j725\" (UID: \"caf8772a-4894-4924-9f18-f2b191aadad7\") " pod="openshift-ingress/e2e-shared-b-w0-25f56e-openshift-default-6465df847-6j725" Oct 31 04:39:06.120955 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:06.120880 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/caf8772a-4894-4924-9f18-f2b191aadad7-istio-data\") pod \"e2e-shared-b-w0-25f56e-openshift-default-6465df847-6j725\" (UID: \"caf8772a-4894-4924-9f18-f2b191aadad7\") " pod="openshift-ingress/e2e-shared-b-w0-25f56e-openshift-default-6465df847-6j725" Oct 31 04:39:06.120955 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:06.120917 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/caf8772a-4894-4924-9f18-f2b191aadad7-istio-token\") pod \"e2e-shared-b-w0-25f56e-openshift-default-6465df847-6j725\" (UID: \"caf8772a-4894-4924-9f18-f2b191aadad7\") " pod="openshift-ingress/e2e-shared-b-w0-25f56e-openshift-default-6465df847-6j725" Oct 31 04:39:06.121047 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:06.121015 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/caf8772a-4894-4924-9f18-f2b191aadad7-istiod-ca-cert\") pod \"e2e-shared-b-w0-25f56e-openshift-default-6465df847-6j725\" (UID: \"caf8772a-4894-4924-9f18-f2b191aadad7\") " pod="openshift-ingress/e2e-shared-b-w0-25f56e-openshift-default-6465df847-6j725" Oct 31 04:39:06.121105 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:06.121079 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/caf8772a-4894-4924-9f18-f2b191aadad7-workload-socket\") pod \"e2e-shared-b-w0-25f56e-openshift-default-6465df847-6j725\" (UID: \"caf8772a-4894-4924-9f18-f2b191aadad7\") " pod="openshift-ingress/e2e-shared-b-w0-25f56e-openshift-default-6465df847-6j725" Oct 31 04:39:06.121157 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:06.121139 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/caf8772a-4894-4924-9f18-f2b191aadad7-credential-socket\") pod \"e2e-shared-b-w0-25f56e-openshift-default-6465df847-6j725\" (UID: \"caf8772a-4894-4924-9f18-f2b191aadad7\") " pod="openshift-ingress/e2e-shared-b-w0-25f56e-openshift-default-6465df847-6j725" Oct 31 04:39:06.121270 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:06.121254 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/caf8772a-4894-4924-9f18-f2b191aadad7-istio-podinfo\") pod \"e2e-shared-b-w0-25f56e-openshift-default-6465df847-6j725\" (UID: \"caf8772a-4894-4924-9f18-f2b191aadad7\") " pod="openshift-ingress/e2e-shared-b-w0-25f56e-openshift-default-6465df847-6j725" Oct 31 04:39:06.226275 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:06.226213 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55vn7\" (UniqueName: \"kubernetes.io/projected/caf8772a-4894-4924-9f18-f2b191aadad7-kube-api-access-55vn7\") pod \"e2e-shared-b-w0-25f56e-openshift-default-6465df847-6j725\" (UID: \"caf8772a-4894-4924-9f18-f2b191aadad7\") " pod="openshift-ingress/e2e-shared-b-w0-25f56e-openshift-default-6465df847-6j725" Oct 31 04:39:06.226685 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:06.226654 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/caf8772a-4894-4924-9f18-f2b191aadad7-workload-certs\") pod \"e2e-shared-b-w0-25f56e-openshift-default-6465df847-6j725\" (UID: \"caf8772a-4894-4924-9f18-f2b191aadad7\") " pod="openshift-ingress/e2e-shared-b-w0-25f56e-openshift-default-6465df847-6j725" Oct 31 04:39:06.226841 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:06.226822 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/caf8772a-4894-4924-9f18-f2b191aadad7-istio-envoy\") pod \"e2e-shared-b-w0-25f56e-openshift-default-6465df847-6j725\" (UID: \"caf8772a-4894-4924-9f18-f2b191aadad7\") " pod="openshift-ingress/e2e-shared-b-w0-25f56e-openshift-default-6465df847-6j725" Oct 31 04:39:06.226996 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:06.226979 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/caf8772a-4894-4924-9f18-f2b191aadad7-istio-data\") pod \"e2e-shared-b-w0-25f56e-openshift-default-6465df847-6j725\" (UID: \"caf8772a-4894-4924-9f18-f2b191aadad7\") " pod="openshift-ingress/e2e-shared-b-w0-25f56e-openshift-default-6465df847-6j725" Oct 31 04:39:06.243491 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:06.243446 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/caf8772a-4894-4924-9f18-f2b191aadad7-istio-token\") pod \"e2e-shared-b-w0-25f56e-openshift-default-6465df847-6j725\" (UID: \"caf8772a-4894-4924-9f18-f2b191aadad7\") " pod="openshift-ingress/e2e-shared-b-w0-25f56e-openshift-default-6465df847-6j725" Oct 31 04:39:06.243887 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:06.243836 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/caf8772a-4894-4924-9f18-f2b191aadad7-istiod-ca-cert\") pod \"e2e-shared-b-w0-25f56e-openshift-default-6465df847-6j725\" (UID: \"caf8772a-4894-4924-9f18-f2b191aadad7\") " pod="openshift-ingress/e2e-shared-b-w0-25f56e-openshift-default-6465df847-6j725" Oct 31 04:39:06.244034 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:06.244017 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/caf8772a-4894-4924-9f18-f2b191aadad7-workload-socket\") pod \"e2e-shared-b-w0-25f56e-openshift-default-6465df847-6j725\" (UID: \"caf8772a-4894-4924-9f18-f2b191aadad7\") " pod="openshift-ingress/e2e-shared-b-w0-25f56e-openshift-default-6465df847-6j725" Oct 31 04:39:06.244186 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:06.244172 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/caf8772a-4894-4924-9f18-f2b191aadad7-credential-socket\") pod \"e2e-shared-b-w0-25f56e-openshift-default-6465df847-6j725\" (UID: \"caf8772a-4894-4924-9f18-f2b191aadad7\") " pod="openshift-ingress/e2e-shared-b-w0-25f56e-openshift-default-6465df847-6j725" Oct 31 04:39:06.244422 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:06.244391 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/caf8772a-4894-4924-9f18-f2b191aadad7-istio-podinfo\") pod \"e2e-shared-b-w0-25f56e-openshift-default-6465df847-6j725\" (UID: \"caf8772a-4894-4924-9f18-f2b191aadad7\") " pod="openshift-ingress/e2e-shared-b-w0-25f56e-openshift-default-6465df847-6j725" Oct 31 04:39:06.247546 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:06.247518 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/caf8772a-4894-4924-9f18-f2b191aadad7-istiod-ca-cert\") pod \"e2e-shared-b-w0-25f56e-openshift-default-6465df847-6j725\" (UID: \"caf8772a-4894-4924-9f18-f2b191aadad7\") " pod="openshift-ingress/e2e-shared-b-w0-25f56e-openshift-default-6465df847-6j725" Oct 31 04:39:06.248311 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:06.248287 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/caf8772a-4894-4924-9f18-f2b191aadad7-workload-socket\") pod \"e2e-shared-b-w0-25f56e-openshift-default-6465df847-6j725\" (UID: \"caf8772a-4894-4924-9f18-f2b191aadad7\") " pod="openshift-ingress/e2e-shared-b-w0-25f56e-openshift-default-6465df847-6j725" Oct 31 04:39:06.249317 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:06.249289 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/caf8772a-4894-4924-9f18-f2b191aadad7-workload-certs\") pod \"e2e-shared-b-w0-25f56e-openshift-default-6465df847-6j725\" (UID: \"caf8772a-4894-4924-9f18-f2b191aadad7\") " pod="openshift-ingress/e2e-shared-b-w0-25f56e-openshift-default-6465df847-6j725" Oct 31 04:39:06.254224 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:06.249762 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/caf8772a-4894-4924-9f18-f2b191aadad7-istio-data\") pod \"e2e-shared-b-w0-25f56e-openshift-default-6465df847-6j725\" (UID: \"caf8772a-4894-4924-9f18-f2b191aadad7\") " pod="openshift-ingress/e2e-shared-b-w0-25f56e-openshift-default-6465df847-6j725" Oct 31 04:39:06.255089 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:06.255065 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/caf8772a-4894-4924-9f18-f2b191aadad7-credential-socket\") pod \"e2e-shared-b-w0-25f56e-openshift-default-6465df847-6j725\" (UID: \"caf8772a-4894-4924-9f18-f2b191aadad7\") " pod="openshift-ingress/e2e-shared-b-w0-25f56e-openshift-default-6465df847-6j725" Oct 31 04:39:06.303388 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:06.282927 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55vn7\" (UniqueName: \"kubernetes.io/projected/caf8772a-4894-4924-9f18-f2b191aadad7-kube-api-access-55vn7\") pod \"e2e-shared-b-w0-25f56e-openshift-default-6465df847-6j725\" (UID: \"caf8772a-4894-4924-9f18-f2b191aadad7\") " pod="openshift-ingress/e2e-shared-b-w0-25f56e-openshift-default-6465df847-6j725" Oct 31 04:39:06.303388 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:06.287745 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/caf8772a-4894-4924-9f18-f2b191aadad7-istio-token\") pod \"e2e-shared-b-w0-25f56e-openshift-default-6465df847-6j725\" (UID: \"caf8772a-4894-4924-9f18-f2b191aadad7\") " pod="openshift-ingress/e2e-shared-b-w0-25f56e-openshift-default-6465df847-6j725" Oct 31 04:39:06.303388 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:06.291008 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/caf8772a-4894-4924-9f18-f2b191aadad7-istio-podinfo\") pod \"e2e-shared-b-w0-25f56e-openshift-default-6465df847-6j725\" (UID: \"caf8772a-4894-4924-9f18-f2b191aadad7\") " pod="openshift-ingress/e2e-shared-b-w0-25f56e-openshift-default-6465df847-6j725" Oct 31 04:39:06.303933 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:06.303877 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/caf8772a-4894-4924-9f18-f2b191aadad7-istio-envoy\") pod \"e2e-shared-b-w0-25f56e-openshift-default-6465df847-6j725\" (UID: \"caf8772a-4894-4924-9f18-f2b191aadad7\") " pod="openshift-ingress/e2e-shared-b-w0-25f56e-openshift-default-6465df847-6j725" Oct 31 04:39:06.329844 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:06.329587 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-25f56e-openshift-default-6465df847-6j725" Oct 31 04:39:06.731315 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:39:06.731282 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcaf8772a_4894_4924_9f18_f2b191aadad7.slice/crio-40b252fe9e99746909a575a9ee76452605feaee2da3a1e6df9b33da8548421fd WatchSource:0}: Error finding container 40b252fe9e99746909a575a9ee76452605feaee2da3a1e6df9b33da8548421fd: Status 404 returned error can't find the container with id 40b252fe9e99746909a575a9ee76452605feaee2da3a1e6df9b33da8548421fd Oct 31 04:39:06.734252 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:06.734231 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-25f56e-openshift-default-6465df847-6j725"] Oct 31 04:39:06.734421 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:06.734391 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:39:06.734466 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:06.734455 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:39:06.734520 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:06.734488 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:39:07.367692 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:07.367245 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-bpgxq" Oct 31 04:39:07.382389 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:07.382304 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/880c7780-4eac-4a67-b948-3d52a354c554-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/880c7780-4eac-4a67-b948-3d52a354c554-credential-socket\") pod \"880c7780-4eac-4a67-b948-3d52a354c554\" (UID: \"880c7780-4eac-4a67-b948-3d52a354c554\") " Oct 31 04:39:07.382522 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:07.382413 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/880c7780-4eac-4a67-b948-3d52a354c554-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/880c7780-4eac-4a67-b948-3d52a354c554-workload-socket\") pod \"880c7780-4eac-4a67-b948-3d52a354c554\" (UID: \"880c7780-4eac-4a67-b948-3d52a354c554\") " Oct 31 04:39:07.382522 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:07.382482 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/880c7780-4eac-4a67-b948-3d52a354c554-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/880c7780-4eac-4a67-b948-3d52a354c554-istiod-ca-cert\") pod \"880c7780-4eac-4a67-b948-3d52a354c554\" (UID: \"880c7780-4eac-4a67-b948-3d52a354c554\") " Oct 31 04:39:07.382645 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:07.382558 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/880c7780-4eac-4a67-b948-3d52a354c554-kube-api-access-j49nq\" (UniqueName: \"kubernetes.io/projected/880c7780-4eac-4a67-b948-3d52a354c554-kube-api-access-j49nq\") pod \"880c7780-4eac-4a67-b948-3d52a354c554\" (UID: \"880c7780-4eac-4a67-b948-3d52a354c554\") " Oct 31 04:39:07.382645 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:07.382600 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/880c7780-4eac-4a67-b948-3d52a354c554-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/880c7780-4eac-4a67-b948-3d52a354c554-workload-certs\") pod \"880c7780-4eac-4a67-b948-3d52a354c554\" (UID: \"880c7780-4eac-4a67-b948-3d52a354c554\") " Oct 31 04:39:07.382753 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:07.382665 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/880c7780-4eac-4a67-b948-3d52a354c554-istio-token\" (UniqueName: \"kubernetes.io/projected/880c7780-4eac-4a67-b948-3d52a354c554-istio-token\") pod \"880c7780-4eac-4a67-b948-3d52a354c554\" (UID: \"880c7780-4eac-4a67-b948-3d52a354c554\") " Oct 31 04:39:07.382753 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:07.382708 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/880c7780-4eac-4a67-b948-3d52a354c554-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/880c7780-4eac-4a67-b948-3d52a354c554-istio-envoy\") pod \"880c7780-4eac-4a67-b948-3d52a354c554\" (UID: \"880c7780-4eac-4a67-b948-3d52a354c554\") " Oct 31 04:39:07.384110 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:07.382905 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/880c7780-4eac-4a67-b948-3d52a354c554-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/880c7780-4eac-4a67-b948-3d52a354c554-istio-podinfo\") pod \"880c7780-4eac-4a67-b948-3d52a354c554\" (UID: \"880c7780-4eac-4a67-b948-3d52a354c554\") " Oct 31 04:39:07.384110 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:07.382972 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/880c7780-4eac-4a67-b948-3d52a354c554-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/880c7780-4eac-4a67-b948-3d52a354c554-istio-data\") pod \"880c7780-4eac-4a67-b948-3d52a354c554\" (UID: \"880c7780-4eac-4a67-b948-3d52a354c554\") " Oct 31 04:39:07.384274 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:07.384172 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/880c7780-4eac-4a67-b948-3d52a354c554-istio-data" pod "880c7780-4eac-4a67-b948-3d52a354c554" (UID: "880c7780-4eac-4a67-b948-3d52a354c554"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:39:07.385652 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:07.384422 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/880c7780-4eac-4a67-b948-3d52a354c554-workload-socket" pod "880c7780-4eac-4a67-b948-3d52a354c554" (UID: "880c7780-4eac-4a67-b948-3d52a354c554"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:39:07.385652 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:07.384661 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/880c7780-4eac-4a67-b948-3d52a354c554-credential-socket" pod "880c7780-4eac-4a67-b948-3d52a354c554" (UID: "880c7780-4eac-4a67-b948-3d52a354c554"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:39:07.388188 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:07.387207 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/880c7780-4eac-4a67-b948-3d52a354c554-workload-certs" pod "880c7780-4eac-4a67-b948-3d52a354c554" (UID: "880c7780-4eac-4a67-b948-3d52a354c554"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:39:07.388188 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:07.387378 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/880c7780-4eac-4a67-b948-3d52a354c554-istiod-ca-cert" pod "880c7780-4eac-4a67-b948-3d52a354c554" (UID: "880c7780-4eac-4a67-b948-3d52a354c554"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:39:07.392177 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:07.391508 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/880c7780-4eac-4a67-b948-3d52a354c554-istio-envoy" pod "880c7780-4eac-4a67-b948-3d52a354c554" (UID: "880c7780-4eac-4a67-b948-3d52a354c554"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:39:07.393732 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:07.392531 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/880c7780-4eac-4a67-b948-3d52a354c554-istio-podinfo" pod "880c7780-4eac-4a67-b948-3d52a354c554" (UID: "880c7780-4eac-4a67-b948-3d52a354c554"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 04:39:07.393732 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:07.392533 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/880c7780-4eac-4a67-b948-3d52a354c554-istio-token" pod "880c7780-4eac-4a67-b948-3d52a354c554" (UID: "880c7780-4eac-4a67-b948-3d52a354c554"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:39:07.393732 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:07.392685 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/880c7780-4eac-4a67-b948-3d52a354c554-kube-api-access-j49nq" pod "880c7780-4eac-4a67-b948-3d52a354c554" (UID: "880c7780-4eac-4a67-b948-3d52a354c554"). InnerVolumeSpecName "kube-api-access-j49nq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:39:07.404602 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:07.404581 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-2vsk6" Oct 31 04:39:07.492085 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:07.492047 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8682ee97-bdd1-4755-a5c6-9fa9d4120247-istio-token\" (UniqueName: \"kubernetes.io/projected/8682ee97-bdd1-4755-a5c6-9fa9d4120247-istio-token\") pod \"8682ee97-bdd1-4755-a5c6-9fa9d4120247\" (UID: \"8682ee97-bdd1-4755-a5c6-9fa9d4120247\") " Oct 31 04:39:07.492264 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:07.492099 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8682ee97-bdd1-4755-a5c6-9fa9d4120247-kube-api-access-j8whx\" (UniqueName: \"kubernetes.io/projected/8682ee97-bdd1-4755-a5c6-9fa9d4120247-kube-api-access-j8whx\") pod \"8682ee97-bdd1-4755-a5c6-9fa9d4120247\" (UID: \"8682ee97-bdd1-4755-a5c6-9fa9d4120247\") " Oct 31 04:39:07.492264 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:07.492158 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8682ee97-bdd1-4755-a5c6-9fa9d4120247-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8682ee97-bdd1-4755-a5c6-9fa9d4120247-istio-envoy\") pod \"8682ee97-bdd1-4755-a5c6-9fa9d4120247\" (UID: \"8682ee97-bdd1-4755-a5c6-9fa9d4120247\") " Oct 31 04:39:07.492428 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:07.492284 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8682ee97-bdd1-4755-a5c6-9fa9d4120247-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8682ee97-bdd1-4755-a5c6-9fa9d4120247-workload-socket\") pod \"8682ee97-bdd1-4755-a5c6-9fa9d4120247\" (UID: \"8682ee97-bdd1-4755-a5c6-9fa9d4120247\") " Oct 31 04:39:07.492428 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:07.492387 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8682ee97-bdd1-4755-a5c6-9fa9d4120247-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8682ee97-bdd1-4755-a5c6-9fa9d4120247-workload-certs\") pod \"8682ee97-bdd1-4755-a5c6-9fa9d4120247\" (UID: \"8682ee97-bdd1-4755-a5c6-9fa9d4120247\") " Oct 31 04:39:07.492670 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:07.492442 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8682ee97-bdd1-4755-a5c6-9fa9d4120247-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8682ee97-bdd1-4755-a5c6-9fa9d4120247-istio-podinfo\") pod \"8682ee97-bdd1-4755-a5c6-9fa9d4120247\" (UID: \"8682ee97-bdd1-4755-a5c6-9fa9d4120247\") " Oct 31 04:39:07.492670 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:07.492501 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8682ee97-bdd1-4755-a5c6-9fa9d4120247-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8682ee97-bdd1-4755-a5c6-9fa9d4120247-istiod-ca-cert\") pod \"8682ee97-bdd1-4755-a5c6-9fa9d4120247\" (UID: \"8682ee97-bdd1-4755-a5c6-9fa9d4120247\") " Oct 31 04:39:07.492670 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:07.492595 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8682ee97-bdd1-4755-a5c6-9fa9d4120247-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8682ee97-bdd1-4755-a5c6-9fa9d4120247-credential-socket\") pod \"8682ee97-bdd1-4755-a5c6-9fa9d4120247\" (UID: \"8682ee97-bdd1-4755-a5c6-9fa9d4120247\") " Oct 31 04:39:07.492670 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:07.492635 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8682ee97-bdd1-4755-a5c6-9fa9d4120247-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8682ee97-bdd1-4755-a5c6-9fa9d4120247-istio-data\") pod \"8682ee97-bdd1-4755-a5c6-9fa9d4120247\" (UID: \"8682ee97-bdd1-4755-a5c6-9fa9d4120247\") " Oct 31 04:39:07.493397 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:07.493352 4376 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/880c7780-4eac-4a67-b948-3d52a354c554-istio-podinfo\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:39:07.493397 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:07.493377 4376 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/880c7780-4eac-4a67-b948-3d52a354c554-istio-data\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:39:07.493397 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:07.493391 4376 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/880c7780-4eac-4a67-b948-3d52a354c554-credential-socket\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:39:07.493665 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:07.493403 4376 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/880c7780-4eac-4a67-b948-3d52a354c554-workload-socket\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:39:07.493665 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:07.493416 4376 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/880c7780-4eac-4a67-b948-3d52a354c554-istiod-ca-cert\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:39:07.493665 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:07.493429 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j49nq\" (UniqueName: \"kubernetes.io/projected/880c7780-4eac-4a67-b948-3d52a354c554-kube-api-access-j49nq\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:39:07.493665 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:07.493442 4376 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/880c7780-4eac-4a67-b948-3d52a354c554-workload-certs\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:39:07.493665 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:07.493454 4376 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/880c7780-4eac-4a67-b948-3d52a354c554-istio-token\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:39:07.493665 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:07.493465 4376 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/880c7780-4eac-4a67-b948-3d52a354c554-istio-envoy\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:39:07.494070 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:07.493984 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8682ee97-bdd1-4755-a5c6-9fa9d4120247-istiod-ca-cert" pod "8682ee97-bdd1-4755-a5c6-9fa9d4120247" (UID: "8682ee97-bdd1-4755-a5c6-9fa9d4120247"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:39:07.494241 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:07.494210 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8682ee97-bdd1-4755-a5c6-9fa9d4120247-credential-socket" pod "8682ee97-bdd1-4755-a5c6-9fa9d4120247" (UID: "8682ee97-bdd1-4755-a5c6-9fa9d4120247"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:39:07.494427 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:07.494406 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8682ee97-bdd1-4755-a5c6-9fa9d4120247-istio-data" pod "8682ee97-bdd1-4755-a5c6-9fa9d4120247" (UID: "8682ee97-bdd1-4755-a5c6-9fa9d4120247"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:39:07.495263 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:07.495238 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8682ee97-bdd1-4755-a5c6-9fa9d4120247-workload-socket" pod "8682ee97-bdd1-4755-a5c6-9fa9d4120247" (UID: "8682ee97-bdd1-4755-a5c6-9fa9d4120247"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:39:07.495471 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:07.495452 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8682ee97-bdd1-4755-a5c6-9fa9d4120247-workload-certs" pod "8682ee97-bdd1-4755-a5c6-9fa9d4120247" (UID: "8682ee97-bdd1-4755-a5c6-9fa9d4120247"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:39:07.496864 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:07.496830 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8682ee97-bdd1-4755-a5c6-9fa9d4120247-istio-podinfo" pod "8682ee97-bdd1-4755-a5c6-9fa9d4120247" (UID: "8682ee97-bdd1-4755-a5c6-9fa9d4120247"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 04:39:07.496864 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:07.496849 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8682ee97-bdd1-4755-a5c6-9fa9d4120247-istio-token" pod "8682ee97-bdd1-4755-a5c6-9fa9d4120247" (UID: "8682ee97-bdd1-4755-a5c6-9fa9d4120247"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:39:07.499085 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:07.499057 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8682ee97-bdd1-4755-a5c6-9fa9d4120247-kube-api-access-j8whx" pod "8682ee97-bdd1-4755-a5c6-9fa9d4120247" (UID: "8682ee97-bdd1-4755-a5c6-9fa9d4120247"). InnerVolumeSpecName "kube-api-access-j8whx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:39:07.500875 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:07.500854 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8682ee97-bdd1-4755-a5c6-9fa9d4120247-istio-envoy" pod "8682ee97-bdd1-4755-a5c6-9fa9d4120247" (UID: "8682ee97-bdd1-4755-a5c6-9fa9d4120247"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:39:07.515371 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:07.515315 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-zmth9" Oct 31 04:39:07.528421 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:07.528403 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-895sj" Oct 31 04:39:07.586071 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:07.585970 4376 generic.go:356] "Generic (PLEG): container finished" podID="5d825a50-865d-46c6-b88d-73f7170963c5" containerID="ef3065d9e6b828d32123237c43ff3949d01ad4aa59f2bd3080d3ca6fffe8548f" exitCode=0 Oct 31 04:39:07.586071 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:07.586067 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-zmth9" event={"ID":"5d825a50-865d-46c6-b88d-73f7170963c5","Type":"ContainerDied","Data":"ef3065d9e6b828d32123237c43ff3949d01ad4aa59f2bd3080d3ca6fffe8548f"} Oct 31 04:39:07.586375 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:07.586114 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-zmth9" event={"ID":"5d825a50-865d-46c6-b88d-73f7170963c5","Type":"ContainerDied","Data":"dcab1365bcd83b6138d68270ff4f2f214909b6070b689eb55c67d6c2400239b3"} Oct 31 04:39:07.586375 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:07.586144 4376 scope.go:122] "RemoveContainer" containerID="ef3065d9e6b828d32123237c43ff3949d01ad4aa59f2bd3080d3ca6fffe8548f" Oct 31 04:39:07.586375 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:07.586296 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-zmth9" Oct 31 04:39:07.589424 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:07.589398 4376 generic.go:356] "Generic (PLEG): container finished" podID="880c7780-4eac-4a67-b948-3d52a354c554" containerID="b7f0d327394e7f57beaa3798867e475cf9ee3d8ad80ac864847d019d4ed24535" exitCode=0 Oct 31 04:39:07.589546 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:07.589466 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-bpgxq" Oct 31 04:39:07.589546 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:07.589483 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-bpgxq" event={"ID":"880c7780-4eac-4a67-b948-3d52a354c554","Type":"ContainerDied","Data":"b7f0d327394e7f57beaa3798867e475cf9ee3d8ad80ac864847d019d4ed24535"} Oct 31 04:39:07.589546 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:07.589538 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-bpgxq" event={"ID":"880c7780-4eac-4a67-b948-3d52a354c554","Type":"ContainerDied","Data":"3400214d725e8b114ad65d8f4a84013eedcb778e49621f7ed883c50f73787ca2"} Oct 31 04:39:07.591020 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:07.591002 4376 generic.go:356] "Generic (PLEG): container finished" podID="bd748a1b-27f9-4e1e-9398-e3bff9c73ae7" containerID="5ccc783fb255eae038c19b151fd480cb4d0a34175fff95d454b54ee0030712e6" exitCode=0 Oct 31 04:39:07.591125 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:07.591051 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-895sj" Oct 31 04:39:07.591125 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:07.591091 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-895sj" event={"ID":"bd748a1b-27f9-4e1e-9398-e3bff9c73ae7","Type":"ContainerDied","Data":"5ccc783fb255eae038c19b151fd480cb4d0a34175fff95d454b54ee0030712e6"} Oct 31 04:39:07.591247 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:07.591147 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-895sj" event={"ID":"bd748a1b-27f9-4e1e-9398-e3bff9c73ae7","Type":"ContainerDied","Data":"f95863a5c065618edeaae98286ed0a0438cea0909b8853b5d708935b7642b682"} Oct 31 04:39:07.592640 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:07.592620 4376 generic.go:356] "Generic (PLEG): container finished" podID="8682ee97-bdd1-4755-a5c6-9fa9d4120247" containerID="34ff2ca539c44439336625349c26eb6ef4a8e897cd67580a007655aa862c7fff" exitCode=0 Oct 31 04:39:07.592747 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:07.592661 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-2vsk6" Oct 31 04:39:07.592747 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:07.592692 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-2vsk6" event={"ID":"8682ee97-bdd1-4755-a5c6-9fa9d4120247","Type":"ContainerDied","Data":"34ff2ca539c44439336625349c26eb6ef4a8e897cd67580a007655aa862c7fff"} Oct 31 04:39:07.592747 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:07.592729 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-2vsk6" event={"ID":"8682ee97-bdd1-4755-a5c6-9fa9d4120247","Type":"ContainerDied","Data":"76f6d6915d759e0323029805681e8394aca80c2eb4b50c495d4eca2e986ead15"} Oct 31 04:39:07.593909 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:07.593884 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d825a50-865d-46c6-b88d-73f7170963c5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d825a50-865d-46c6-b88d-73f7170963c5-credential-socket\") pod \"5d825a50-865d-46c6-b88d-73f7170963c5\" (UID: \"5d825a50-865d-46c6-b88d-73f7170963c5\") " Oct 31 04:39:07.594029 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:07.593936 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd748a1b-27f9-4e1e-9398-e3bff9c73ae7-kube-api-access-6nxp4\" (UniqueName: \"kubernetes.io/projected/bd748a1b-27f9-4e1e-9398-e3bff9c73ae7-kube-api-access-6nxp4\") pod \"bd748a1b-27f9-4e1e-9398-e3bff9c73ae7\" (UID: \"bd748a1b-27f9-4e1e-9398-e3bff9c73ae7\") " Oct 31 04:39:07.594029 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:07.593992 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d825a50-865d-46c6-b88d-73f7170963c5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d825a50-865d-46c6-b88d-73f7170963c5-workload-certs\") pod \"5d825a50-865d-46c6-b88d-73f7170963c5\" (UID: \"5d825a50-865d-46c6-b88d-73f7170963c5\") " Oct 31 04:39:07.594159 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:07.594030 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d825a50-865d-46c6-b88d-73f7170963c5-kube-api-access-84mr7\" (UniqueName: \"kubernetes.io/projected/5d825a50-865d-46c6-b88d-73f7170963c5-kube-api-access-84mr7\") pod \"5d825a50-865d-46c6-b88d-73f7170963c5\" (UID: \"5d825a50-865d-46c6-b88d-73f7170963c5\") " Oct 31 04:39:07.594159 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:07.594102 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d825a50-865d-46c6-b88d-73f7170963c5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d825a50-865d-46c6-b88d-73f7170963c5-istio-envoy\") pod \"5d825a50-865d-46c6-b88d-73f7170963c5\" (UID: \"5d825a50-865d-46c6-b88d-73f7170963c5\") " Oct 31 04:39:07.594159 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:07.594147 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d825a50-865d-46c6-b88d-73f7170963c5-credential-socket" pod "5d825a50-865d-46c6-b88d-73f7170963c5" (UID: "5d825a50-865d-46c6-b88d-73f7170963c5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:39:07.594343 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:07.594158 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd748a1b-27f9-4e1e-9398-e3bff9c73ae7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd748a1b-27f9-4e1e-9398-e3bff9c73ae7-credential-socket\") pod \"bd748a1b-27f9-4e1e-9398-e3bff9c73ae7\" (UID: \"bd748a1b-27f9-4e1e-9398-e3bff9c73ae7\") " Oct 31 04:39:07.594343 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:07.594208 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd748a1b-27f9-4e1e-9398-e3bff9c73ae7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd748a1b-27f9-4e1e-9398-e3bff9c73ae7-istiod-ca-cert\") pod \"bd748a1b-27f9-4e1e-9398-e3bff9c73ae7\" (UID: \"bd748a1b-27f9-4e1e-9398-e3bff9c73ae7\") " Oct 31 04:39:07.594343 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:07.594268 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d825a50-865d-46c6-b88d-73f7170963c5-istio-token\" (UniqueName: \"kubernetes.io/projected/5d825a50-865d-46c6-b88d-73f7170963c5-istio-token\") pod \"5d825a50-865d-46c6-b88d-73f7170963c5\" (UID: \"5d825a50-865d-46c6-b88d-73f7170963c5\") " Oct 31 04:39:07.594343 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:07.594303 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d825a50-865d-46c6-b88d-73f7170963c5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d825a50-865d-46c6-b88d-73f7170963c5-istiod-ca-cert\") pod \"5d825a50-865d-46c6-b88d-73f7170963c5\" (UID: \"5d825a50-865d-46c6-b88d-73f7170963c5\") " Oct 31 04:39:07.594583 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:07.594424 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd748a1b-27f9-4e1e-9398-e3bff9c73ae7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd748a1b-27f9-4e1e-9398-e3bff9c73ae7-istio-envoy\") pod \"bd748a1b-27f9-4e1e-9398-e3bff9c73ae7\" (UID: \"bd748a1b-27f9-4e1e-9398-e3bff9c73ae7\") " Oct 31 04:39:07.594583 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:07.594465 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd748a1b-27f9-4e1e-9398-e3bff9c73ae7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd748a1b-27f9-4e1e-9398-e3bff9c73ae7-workload-certs\") pod \"bd748a1b-27f9-4e1e-9398-e3bff9c73ae7\" (UID: \"bd748a1b-27f9-4e1e-9398-e3bff9c73ae7\") " Oct 31 04:39:07.594583 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:07.594504 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d825a50-865d-46c6-b88d-73f7170963c5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d825a50-865d-46c6-b88d-73f7170963c5-istio-data\") pod \"5d825a50-865d-46c6-b88d-73f7170963c5\" (UID: \"5d825a50-865d-46c6-b88d-73f7170963c5\") " Oct 31 04:39:07.594583 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:07.594541 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd748a1b-27f9-4e1e-9398-e3bff9c73ae7-istio-token\" (UniqueName: \"kubernetes.io/projected/bd748a1b-27f9-4e1e-9398-e3bff9c73ae7-istio-token\") pod \"bd748a1b-27f9-4e1e-9398-e3bff9c73ae7\" (UID: \"bd748a1b-27f9-4e1e-9398-e3bff9c73ae7\") " Oct 31 04:39:07.594789 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:07.594668 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd748a1b-27f9-4e1e-9398-e3bff9c73ae7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd748a1b-27f9-4e1e-9398-e3bff9c73ae7-istio-data\") pod \"bd748a1b-27f9-4e1e-9398-e3bff9c73ae7\" (UID: \"bd748a1b-27f9-4e1e-9398-e3bff9c73ae7\") " Oct 31 04:39:07.594789 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:07.594703 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5d825a50-865d-46c6-b88d-73f7170963c5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d825a50-865d-46c6-b88d-73f7170963c5-istio-podinfo\") pod \"5d825a50-865d-46c6-b88d-73f7170963c5\" (UID: \"5d825a50-865d-46c6-b88d-73f7170963c5\") " Oct 31 04:39:07.594789 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:07.594731 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bd748a1b-27f9-4e1e-9398-e3bff9c73ae7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd748a1b-27f9-4e1e-9398-e3bff9c73ae7-istio-podinfo\") pod \"bd748a1b-27f9-4e1e-9398-e3bff9c73ae7\" (UID: \"bd748a1b-27f9-4e1e-9398-e3bff9c73ae7\") " Oct 31 04:39:07.594789 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:07.594763 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d825a50-865d-46c6-b88d-73f7170963c5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d825a50-865d-46c6-b88d-73f7170963c5-workload-socket\") pod \"5d825a50-865d-46c6-b88d-73f7170963c5\" (UID: \"5d825a50-865d-46c6-b88d-73f7170963c5\") " Oct 31 04:39:07.595000 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:07.594813 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd748a1b-27f9-4e1e-9398-e3bff9c73ae7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd748a1b-27f9-4e1e-9398-e3bff9c73ae7-workload-socket\") pod \"bd748a1b-27f9-4e1e-9398-e3bff9c73ae7\" (UID: \"bd748a1b-27f9-4e1e-9398-e3bff9c73ae7\") " Oct 31 04:39:07.595321 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:07.595299 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd748a1b-27f9-4e1e-9398-e3bff9c73ae7-workload-certs" pod "bd748a1b-27f9-4e1e-9398-e3bff9c73ae7" (UID: "bd748a1b-27f9-4e1e-9398-e3bff9c73ae7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:39:07.595459 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:07.595434 4376 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8682ee97-bdd1-4755-a5c6-9fa9d4120247-workload-socket\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:39:07.595545 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:07.595468 4376 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8682ee97-bdd1-4755-a5c6-9fa9d4120247-workload-certs\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:39:07.595545 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:07.595483 4376 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8682ee97-bdd1-4755-a5c6-9fa9d4120247-istio-podinfo\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:39:07.595545 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:07.595497 4376 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8682ee97-bdd1-4755-a5c6-9fa9d4120247-istiod-ca-cert\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:39:07.595545 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:07.595511 4376 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8682ee97-bdd1-4755-a5c6-9fa9d4120247-credential-socket\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:39:07.595545 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:07.595527 4376 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8682ee97-bdd1-4755-a5c6-9fa9d4120247-istio-data\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:39:07.595545 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:07.595542 4376 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d825a50-865d-46c6-b88d-73f7170963c5-credential-socket\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:39:07.595969 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:07.595556 4376 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8682ee97-bdd1-4755-a5c6-9fa9d4120247-istio-token\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:39:07.595969 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:07.595570 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8whx\" (UniqueName: \"kubernetes.io/projected/8682ee97-bdd1-4755-a5c6-9fa9d4120247-kube-api-access-j8whx\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:39:07.595969 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:07.595584 4376 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8682ee97-bdd1-4755-a5c6-9fa9d4120247-istio-envoy\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:39:07.595969 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:07.595745 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-25f56e-openshift-default-6465df847-6j725" event={"ID":"caf8772a-4894-4924-9f18-f2b191aadad7","Type":"ContainerStarted","Data":"440604183b7edcc8cd815d9149aa46b04f0ad06c051c8897a66a7ef0bd0d4761"} Oct 31 04:39:07.595969 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:07.595800 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-25f56e-openshift-default-6465df847-6j725" event={"ID":"caf8772a-4894-4924-9f18-f2b191aadad7","Type":"ContainerStarted","Data":"40b252fe9e99746909a575a9ee76452605feaee2da3a1e6df9b33da8548421fd"} Oct 31 04:39:07.596378 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:07.596283 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd748a1b-27f9-4e1e-9398-e3bff9c73ae7-istio-data" pod "bd748a1b-27f9-4e1e-9398-e3bff9c73ae7" (UID: "bd748a1b-27f9-4e1e-9398-e3bff9c73ae7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:39:07.596537 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:07.596506 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d825a50-865d-46c6-b88d-73f7170963c5-istio-data" pod "5d825a50-865d-46c6-b88d-73f7170963c5" (UID: "5d825a50-865d-46c6-b88d-73f7170963c5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:39:07.596821 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:07.596800 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd748a1b-27f9-4e1e-9398-e3bff9c73ae7-workload-socket" pod "bd748a1b-27f9-4e1e-9398-e3bff9c73ae7" (UID: "bd748a1b-27f9-4e1e-9398-e3bff9c73ae7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:39:07.596916 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:07.596805 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d825a50-865d-46c6-b88d-73f7170963c5-kube-api-access-84mr7" pod "5d825a50-865d-46c6-b88d-73f7170963c5" (UID: "5d825a50-865d-46c6-b88d-73f7170963c5"). InnerVolumeSpecName "kube-api-access-84mr7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:39:07.597095 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:07.597072 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d825a50-865d-46c6-b88d-73f7170963c5-workload-socket" pod "5d825a50-865d-46c6-b88d-73f7170963c5" (UID: "5d825a50-865d-46c6-b88d-73f7170963c5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:39:07.599078 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:07.599051 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bd748a1b-27f9-4e1e-9398-e3bff9c73ae7-istio-podinfo" pod "bd748a1b-27f9-4e1e-9398-e3bff9c73ae7" (UID: "bd748a1b-27f9-4e1e-9398-e3bff9c73ae7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 04:39:07.599417 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:07.599394 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d825a50-865d-46c6-b88d-73f7170963c5-workload-certs" pod "5d825a50-865d-46c6-b88d-73f7170963c5" (UID: "5d825a50-865d-46c6-b88d-73f7170963c5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:39:07.599511 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:07.599431 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d825a50-865d-46c6-b88d-73f7170963c5-istiod-ca-cert" pod "5d825a50-865d-46c6-b88d-73f7170963c5" (UID: "5d825a50-865d-46c6-b88d-73f7170963c5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:39:07.599649 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:07.599624 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd748a1b-27f9-4e1e-9398-e3bff9c73ae7-credential-socket" pod "bd748a1b-27f9-4e1e-9398-e3bff9c73ae7" (UID: "bd748a1b-27f9-4e1e-9398-e3bff9c73ae7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:39:07.599877 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:07.599859 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd748a1b-27f9-4e1e-9398-e3bff9c73ae7-istiod-ca-cert" pod "bd748a1b-27f9-4e1e-9398-e3bff9c73ae7" (UID: "bd748a1b-27f9-4e1e-9398-e3bff9c73ae7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:39:07.601708 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:07.601684 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d825a50-865d-46c6-b88d-73f7170963c5-istio-envoy" pod "5d825a50-865d-46c6-b88d-73f7170963c5" (UID: "5d825a50-865d-46c6-b88d-73f7170963c5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:39:07.601811 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:07.601713 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd748a1b-27f9-4e1e-9398-e3bff9c73ae7-istio-envoy" pod "bd748a1b-27f9-4e1e-9398-e3bff9c73ae7" (UID: "bd748a1b-27f9-4e1e-9398-e3bff9c73ae7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:39:07.601811 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:07.601802 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd748a1b-27f9-4e1e-9398-e3bff9c73ae7-kube-api-access-6nxp4" pod "bd748a1b-27f9-4e1e-9398-e3bff9c73ae7" (UID: "bd748a1b-27f9-4e1e-9398-e3bff9c73ae7"). InnerVolumeSpecName "kube-api-access-6nxp4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:39:07.601969 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:07.601947 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d825a50-865d-46c6-b88d-73f7170963c5-istio-token" pod "5d825a50-865d-46c6-b88d-73f7170963c5" (UID: "5d825a50-865d-46c6-b88d-73f7170963c5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:39:07.602071 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:07.602052 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd748a1b-27f9-4e1e-9398-e3bff9c73ae7-istio-token" pod "bd748a1b-27f9-4e1e-9398-e3bff9c73ae7" (UID: "bd748a1b-27f9-4e1e-9398-e3bff9c73ae7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:39:07.603310 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:07.603212 4376 scope.go:122] "RemoveContainer" containerID="ba4b20e19ffef90a458a50db30fa6d872dfa311726d28f320c05c268046f4bac" Oct 31 04:39:07.603748 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:07.603716 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5d825a50-865d-46c6-b88d-73f7170963c5-istio-podinfo" pod "5d825a50-865d-46c6-b88d-73f7170963c5" (UID: "5d825a50-865d-46c6-b88d-73f7170963c5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 04:39:07.623997 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:07.623972 4376 scope.go:122] "RemoveContainer" containerID="ef3065d9e6b828d32123237c43ff3949d01ad4aa59f2bd3080d3ca6fffe8548f" Oct 31 04:39:07.625067 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:39:07.625041 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef3065d9e6b828d32123237c43ff3949d01ad4aa59f2bd3080d3ca6fffe8548f\": container with ID starting with ef3065d9e6b828d32123237c43ff3949d01ad4aa59f2bd3080d3ca6fffe8548f not found: ID does not exist" containerID="ef3065d9e6b828d32123237c43ff3949d01ad4aa59f2bd3080d3ca6fffe8548f" Oct 31 04:39:07.625168 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:07.625079 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef3065d9e6b828d32123237c43ff3949d01ad4aa59f2bd3080d3ca6fffe8548f"} err="failed to get container status \"ef3065d9e6b828d32123237c43ff3949d01ad4aa59f2bd3080d3ca6fffe8548f\": rpc error: code = NotFound desc = could not find container \"ef3065d9e6b828d32123237c43ff3949d01ad4aa59f2bd3080d3ca6fffe8548f\": container with ID starting with ef3065d9e6b828d32123237c43ff3949d01ad4aa59f2bd3080d3ca6fffe8548f not found: ID does not exist" Oct 31 04:39:07.625168 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:07.625103 4376 scope.go:122] "RemoveContainer" containerID="ba4b20e19ffef90a458a50db30fa6d872dfa311726d28f320c05c268046f4bac" Oct 31 04:39:07.625594 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:39:07.625571 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba4b20e19ffef90a458a50db30fa6d872dfa311726d28f320c05c268046f4bac\": container with ID starting with ba4b20e19ffef90a458a50db30fa6d872dfa311726d28f320c05c268046f4bac not found: ID does not exist" containerID="ba4b20e19ffef90a458a50db30fa6d872dfa311726d28f320c05c268046f4bac" Oct 31 04:39:07.625667 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:07.625604 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba4b20e19ffef90a458a50db30fa6d872dfa311726d28f320c05c268046f4bac"} err="failed to get container status \"ba4b20e19ffef90a458a50db30fa6d872dfa311726d28f320c05c268046f4bac\": rpc error: code = NotFound desc = could not find container \"ba4b20e19ffef90a458a50db30fa6d872dfa311726d28f320c05c268046f4bac\": container with ID starting with ba4b20e19ffef90a458a50db30fa6d872dfa311726d28f320c05c268046f4bac not found: ID does not exist" Oct 31 04:39:07.625667 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:07.625622 4376 scope.go:122] "RemoveContainer" containerID="b7f0d327394e7f57beaa3798867e475cf9ee3d8ad80ac864847d019d4ed24535" Oct 31 04:39:07.637370 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:07.637318 4376 scope.go:122] "RemoveContainer" containerID="b7f0d327394e7f57beaa3798867e475cf9ee3d8ad80ac864847d019d4ed24535" Oct 31 04:39:07.637682 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:39:07.637663 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7f0d327394e7f57beaa3798867e475cf9ee3d8ad80ac864847d019d4ed24535\": container with ID starting with b7f0d327394e7f57beaa3798867e475cf9ee3d8ad80ac864847d019d4ed24535 not found: ID does not exist" containerID="b7f0d327394e7f57beaa3798867e475cf9ee3d8ad80ac864847d019d4ed24535" Oct 31 04:39:07.637777 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:07.637700 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7f0d327394e7f57beaa3798867e475cf9ee3d8ad80ac864847d019d4ed24535"} err="failed to get container status \"b7f0d327394e7f57beaa3798867e475cf9ee3d8ad80ac864847d019d4ed24535\": rpc error: code = NotFound desc = could not find container \"b7f0d327394e7f57beaa3798867e475cf9ee3d8ad80ac864847d019d4ed24535\": container with ID starting with b7f0d327394e7f57beaa3798867e475cf9ee3d8ad80ac864847d019d4ed24535 not found: ID does not exist" Oct 31 04:39:07.637777 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:07.637722 4376 scope.go:122] "RemoveContainer" containerID="5ccc783fb255eae038c19b151fd480cb4d0a34175fff95d454b54ee0030712e6" Oct 31 04:39:07.649355 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:07.649290 4376 scope.go:122] "RemoveContainer" containerID="5ccc783fb255eae038c19b151fd480cb4d0a34175fff95d454b54ee0030712e6" Oct 31 04:39:07.649630 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:39:07.649600 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ccc783fb255eae038c19b151fd480cb4d0a34175fff95d454b54ee0030712e6\": container with ID starting with 5ccc783fb255eae038c19b151fd480cb4d0a34175fff95d454b54ee0030712e6 not found: ID does not exist" containerID="5ccc783fb255eae038c19b151fd480cb4d0a34175fff95d454b54ee0030712e6" Oct 31 04:39:07.649718 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:07.649639 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ccc783fb255eae038c19b151fd480cb4d0a34175fff95d454b54ee0030712e6"} err="failed to get container status \"5ccc783fb255eae038c19b151fd480cb4d0a34175fff95d454b54ee0030712e6\": rpc error: code = NotFound desc = could not find container \"5ccc783fb255eae038c19b151fd480cb4d0a34175fff95d454b54ee0030712e6\": container with ID starting with 5ccc783fb255eae038c19b151fd480cb4d0a34175fff95d454b54ee0030712e6 not found: ID does not exist" Oct 31 04:39:07.649718 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:07.649660 4376 scope.go:122] "RemoveContainer" containerID="34ff2ca539c44439336625349c26eb6ef4a8e897cd67580a007655aa862c7fff" Oct 31 04:39:07.660341 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:07.660276 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-25f56e-openshift-default-6465df847-6j725" podStartSLOduration=2.6602599700000003 podStartE2EDuration="2.66025997s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:39:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:39:07.655874618 +0000 UTC m=+2095.370209261" watchObservedRunningTime="2026-10-31 04:39:07.66025997 +0000 UTC m=+2095.374594591" Oct 31 04:39:07.662191 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:07.662176 4376 scope.go:122] "RemoveContainer" containerID="34ff2ca539c44439336625349c26eb6ef4a8e897cd67580a007655aa862c7fff" Oct 31 04:39:07.662493 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:39:07.662451 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34ff2ca539c44439336625349c26eb6ef4a8e897cd67580a007655aa862c7fff\": container with ID starting with 34ff2ca539c44439336625349c26eb6ef4a8e897cd67580a007655aa862c7fff not found: ID does not exist" containerID="34ff2ca539c44439336625349c26eb6ef4a8e897cd67580a007655aa862c7fff" Oct 31 04:39:07.662611 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:07.662497 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34ff2ca539c44439336625349c26eb6ef4a8e897cd67580a007655aa862c7fff"} err="failed to get container status \"34ff2ca539c44439336625349c26eb6ef4a8e897cd67580a007655aa862c7fff\": rpc error: code = NotFound desc = could not find container \"34ff2ca539c44439336625349c26eb6ef4a8e897cd67580a007655aa862c7fff\": container with ID starting with 34ff2ca539c44439336625349c26eb6ef4a8e897cd67580a007655aa862c7fff not found: ID does not exist" Oct 31 04:39:07.697000 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:07.696974 4376 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d825a50-865d-46c6-b88d-73f7170963c5-istio-envoy\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:39:07.697130 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:07.697004 4376 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd748a1b-27f9-4e1e-9398-e3bff9c73ae7-credential-socket\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:39:07.697130 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:07.697021 4376 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd748a1b-27f9-4e1e-9398-e3bff9c73ae7-istiod-ca-cert\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:39:07.697130 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:07.697036 4376 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d825a50-865d-46c6-b88d-73f7170963c5-istio-token\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:39:07.697130 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:07.697051 4376 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d825a50-865d-46c6-b88d-73f7170963c5-istiod-ca-cert\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:39:07.697130 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:07.697064 4376 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd748a1b-27f9-4e1e-9398-e3bff9c73ae7-istio-envoy\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:39:07.697130 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:07.697075 4376 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd748a1b-27f9-4e1e-9398-e3bff9c73ae7-workload-certs\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:39:07.697130 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:07.697087 4376 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d825a50-865d-46c6-b88d-73f7170963c5-istio-data\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:39:07.697130 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:07.697100 4376 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd748a1b-27f9-4e1e-9398-e3bff9c73ae7-istio-token\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:39:07.697130 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:07.697113 4376 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd748a1b-27f9-4e1e-9398-e3bff9c73ae7-istio-data\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:39:07.697130 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:07.697125 4376 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d825a50-865d-46c6-b88d-73f7170963c5-istio-podinfo\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:39:07.697524 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:07.697138 4376 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd748a1b-27f9-4e1e-9398-e3bff9c73ae7-istio-podinfo\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:39:07.697524 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:07.697154 4376 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d825a50-865d-46c6-b88d-73f7170963c5-workload-socket\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:39:07.697524 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:07.697184 4376 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd748a1b-27f9-4e1e-9398-e3bff9c73ae7-workload-socket\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:39:07.697524 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:07.697206 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6nxp4\" (UniqueName: \"kubernetes.io/projected/bd748a1b-27f9-4e1e-9398-e3bff9c73ae7-kube-api-access-6nxp4\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:39:07.697524 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:07.697221 4376 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d825a50-865d-46c6-b88d-73f7170963c5-workload-certs\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:39:07.697524 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:07.697237 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84mr7\" (UniqueName: \"kubernetes.io/projected/5d825a50-865d-46c6-b88d-73f7170963c5-kube-api-access-84mr7\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:39:07.744757 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:07.744724 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-2vsk6"] Oct 31 04:39:07.749835 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:07.749810 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-2vsk6"] Oct 31 04:39:07.783985 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:07.783957 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-bpgxq"] Oct 31 04:39:07.788034 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:07.788012 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-bpgxq"] Oct 31 04:39:07.940155 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:07.940124 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-zmth9"] Oct 31 04:39:07.946437 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:07.946414 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-zmth9"] Oct 31 04:39:07.982759 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:07.982733 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-895sj"] Oct 31 04:39:07.986278 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:07.986250 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-895sj"] Oct 31 04:39:08.330407 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:08.330373 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-25f56e-openshift-default-6465df847-6j725" Oct 31 04:39:08.336732 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:08.336709 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-25f56e-openshift-default-6465df847-6j725" Oct 31 04:39:08.444154 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:08.444120 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d825a50-865d-46c6-b88d-73f7170963c5" path="/var/lib/kubelet/pods/5d825a50-865d-46c6-b88d-73f7170963c5/volumes" Oct 31 04:39:08.444657 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:08.444641 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8682ee97-bdd1-4755-a5c6-9fa9d4120247" path="/var/lib/kubelet/pods/8682ee97-bdd1-4755-a5c6-9fa9d4120247/volumes" Oct 31 04:39:08.445083 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:08.445069 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="880c7780-4eac-4a67-b948-3d52a354c554" path="/var/lib/kubelet/pods/880c7780-4eac-4a67-b948-3d52a354c554/volumes" Oct 31 04:39:08.445852 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:08.445831 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd748a1b-27f9-4e1e-9398-e3bff9c73ae7" path="/var/lib/kubelet/pods/bd748a1b-27f9-4e1e-9398-e3bff9c73ae7/volumes" Oct 31 04:39:08.608763 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:08.608674 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-25f56e-openshift-default-6465df847-6j725" Oct 31 04:39:08.609506 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:08.609486 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-25f56e-openshift-default-6465df847-6j725" Oct 31 04:39:10.082951 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:10.082914 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-401761-f8dbb88dc-9ks76"] Oct 31 04:39:10.083605 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:10.083576 4376 cpu_manager.go:414] "containerMap: removing container" podUID="bd748a1b-27f9-4e1e-9398-e3bff9c73ae7" containerName="istio-proxy" Oct 31 04:39:10.083605 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:10.083607 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd748a1b-27f9-4e1e-9398-e3bff9c73ae7" containerName="istio-proxy" Oct 31 04:39:10.083706 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:10.083624 4376 cpu_manager.go:414] "containerMap: removing container" podUID="5d825a50-865d-46c6-b88d-73f7170963c5" containerName="istio-proxy" Oct 31 04:39:10.083706 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:10.083629 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d825a50-865d-46c6-b88d-73f7170963c5" containerName="istio-proxy" Oct 31 04:39:10.083706 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:10.083661 4376 cpu_manager.go:414] "containerMap: removing container" podUID="8682ee97-bdd1-4755-a5c6-9fa9d4120247" containerName="istio-proxy" Oct 31 04:39:10.083706 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:10.083667 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8682ee97-bdd1-4755-a5c6-9fa9d4120247" containerName="istio-proxy" Oct 31 04:39:10.083706 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:10.083679 4376 cpu_manager.go:414] "containerMap: removing container" podUID="5d825a50-865d-46c6-b88d-73f7170963c5" containerName="istio-proxy" Oct 31 04:39:10.083706 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:10.083685 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d825a50-865d-46c6-b88d-73f7170963c5" containerName="istio-proxy" Oct 31 04:39:10.083706 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:10.083693 4376 cpu_manager.go:414] "containerMap: removing container" podUID="880c7780-4eac-4a67-b948-3d52a354c554" containerName="istio-proxy" Oct 31 04:39:10.083706 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:10.083699 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="880c7780-4eac-4a67-b948-3d52a354c554" containerName="istio-proxy" Oct 31 04:39:10.083938 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:10.083785 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="8682ee97-bdd1-4755-a5c6-9fa9d4120247" containerName="istio-proxy" Oct 31 04:39:10.083938 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:10.083795 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d825a50-865d-46c6-b88d-73f7170963c5" containerName="istio-proxy" Oct 31 04:39:10.083938 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:10.083801 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="880c7780-4eac-4a67-b948-3d52a354c554" containerName="istio-proxy" Oct 31 04:39:10.083938 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:10.083810 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd748a1b-27f9-4e1e-9398-e3bff9c73ae7" containerName="istio-proxy" Oct 31 04:39:10.083938 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:10.083817 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d825a50-865d-46c6-b88d-73f7170963c5" containerName="istio-proxy" Oct 31 04:39:10.088828 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:10.088811 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-401761-f8dbb88dc-9ks76" Oct 31 04:39:10.109110 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:10.109088 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-401761\"" Oct 31 04:39:10.109365 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:10.109341 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-401761-f8dbb88dc-9ks76"] Oct 31 04:39:10.117724 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:10.117700 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-401761-dockercfg-f4chj\"" Oct 31 04:39:10.129356 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:10.129305 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rx2k7\" (UniqueName: \"kubernetes.io/projected/8ab573d4-4b87-40e0-9d16-ff7fdd3c3963-kube-api-access-rx2k7\") pod \"payload-pre-processing-e2e-extmod-a-401761-f8dbb88dc-9ks76\" (UID: \"8ab573d4-4b87-40e0-9d16-ff7fdd3c3963\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-401761-f8dbb88dc-9ks76" Oct 31 04:39:10.129601 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:10.129567 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ab573d4-4b87-40e0-9d16-ff7fdd3c3963-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-401761-f8dbb88dc-9ks76\" (UID: \"8ab573d4-4b87-40e0-9d16-ff7fdd3c3963\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-401761-f8dbb88dc-9ks76" Oct 31 04:39:10.174638 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:10.174608 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-401761-6ddf8dc4bf-fbwrw"] Oct 31 04:39:10.178849 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:10.178830 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-401761-6ddf8dc4bf-fbwrw" Oct 31 04:39:10.202469 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:10.202441 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-401761-6ddf8dc4bf-fbwrw"] Oct 31 04:39:10.231161 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:10.231117 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0551b77c-f034-48e4-b643-3361b903df31-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-401761-6ddf8dc4bf-fbwrw\" (UID: \"0551b77c-f034-48e4-b643-3361b903df31\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-401761-6ddf8dc4bf-fbwrw" Oct 31 04:39:10.231311 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:10.231240 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ab573d4-4b87-40e0-9d16-ff7fdd3c3963-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-401761-f8dbb88dc-9ks76\" (UID: \"8ab573d4-4b87-40e0-9d16-ff7fdd3c3963\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-401761-f8dbb88dc-9ks76" Oct 31 04:39:10.231311 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:10.231297 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9kkt\" (UniqueName: \"kubernetes.io/projected/0551b77c-f034-48e4-b643-3361b903df31-kube-api-access-f9kkt\") pod \"payload-processing-e2e-extmod-a-401761-6ddf8dc4bf-fbwrw\" (UID: \"0551b77c-f034-48e4-b643-3361b903df31\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-401761-6ddf8dc4bf-fbwrw" Oct 31 04:39:10.231484 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:10.231462 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rx2k7\" (UniqueName: \"kubernetes.io/projected/8ab573d4-4b87-40e0-9d16-ff7fdd3c3963-kube-api-access-rx2k7\") pod \"payload-pre-processing-e2e-extmod-a-401761-f8dbb88dc-9ks76\" (UID: \"8ab573d4-4b87-40e0-9d16-ff7fdd3c3963\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-401761-f8dbb88dc-9ks76" Oct 31 04:39:10.231935 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:10.231914 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ab573d4-4b87-40e0-9d16-ff7fdd3c3963-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-401761-f8dbb88dc-9ks76\" (UID: \"8ab573d4-4b87-40e0-9d16-ff7fdd3c3963\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-401761-f8dbb88dc-9ks76" Oct 31 04:39:10.254969 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:10.254950 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rx2k7\" (UniqueName: \"kubernetes.io/projected/8ab573d4-4b87-40e0-9d16-ff7fdd3c3963-kube-api-access-rx2k7\") pod \"payload-pre-processing-e2e-extmod-a-401761-f8dbb88dc-9ks76\" (UID: \"8ab573d4-4b87-40e0-9d16-ff7fdd3c3963\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-401761-f8dbb88dc-9ks76" Oct 31 04:39:10.338426 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:10.338281 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0551b77c-f034-48e4-b643-3361b903df31-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-401761-6ddf8dc4bf-fbwrw\" (UID: \"0551b77c-f034-48e4-b643-3361b903df31\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-401761-6ddf8dc4bf-fbwrw" Oct 31 04:39:10.338619 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:10.338533 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9kkt\" (UniqueName: \"kubernetes.io/projected/0551b77c-f034-48e4-b643-3361b903df31-kube-api-access-f9kkt\") pod \"payload-processing-e2e-extmod-a-401761-6ddf8dc4bf-fbwrw\" (UID: \"0551b77c-f034-48e4-b643-3361b903df31\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-401761-6ddf8dc4bf-fbwrw" Oct 31 04:39:10.347364 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:10.339852 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0551b77c-f034-48e4-b643-3361b903df31-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-401761-6ddf8dc4bf-fbwrw\" (UID: \"0551b77c-f034-48e4-b643-3361b903df31\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-401761-6ddf8dc4bf-fbwrw" Oct 31 04:39:10.383980 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:10.371502 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9kkt\" (UniqueName: \"kubernetes.io/projected/0551b77c-f034-48e4-b643-3361b903df31-kube-api-access-f9kkt\") pod \"payload-processing-e2e-extmod-a-401761-6ddf8dc4bf-fbwrw\" (UID: \"0551b77c-f034-48e4-b643-3361b903df31\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-401761-6ddf8dc4bf-fbwrw" Oct 31 04:39:10.411011 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:10.410811 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-401761-f8dbb88dc-9ks76" Oct 31 04:39:10.493866 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:10.493706 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-401761-6ddf8dc4bf-fbwrw" Oct 31 04:39:10.754644 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:39:10.754611 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ab573d4_4b87_40e0_9d16_ff7fdd3c3963.slice/crio-6bcf65fcf5fc1220b7191798e72f69ec092f87335b59a3a0fbc20ad042d538f4 WatchSource:0}: Error finding container 6bcf65fcf5fc1220b7191798e72f69ec092f87335b59a3a0fbc20ad042d538f4: Status 404 returned error can't find the container with id 6bcf65fcf5fc1220b7191798e72f69ec092f87335b59a3a0fbc20ad042d538f4 Oct 31 04:39:10.756519 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:10.756495 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-401761-f8dbb88dc-9ks76"] Oct 31 04:39:10.833209 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:39:10.833182 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0551b77c_f034_48e4_b643_3361b903df31.slice/crio-f9fd237b0531d196a75cf07efa66ca19ce6ebbb7b3a6c5cc713806c47e9a47d0 WatchSource:0}: Error finding container f9fd237b0531d196a75cf07efa66ca19ce6ebbb7b3a6c5cc713806c47e9a47d0: Status 404 returned error can't find the container with id f9fd237b0531d196a75cf07efa66ca19ce6ebbb7b3a6c5cc713806c47e9a47d0 Oct 31 04:39:10.834785 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:10.834753 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-401761-6ddf8dc4bf-fbwrw"] Oct 31 04:39:11.001117 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:39:11.001069 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 04:39:11.541917 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:11.541802 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-afc519-d746ff6cc-npmp9" Oct 31 04:39:11.637653 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:11.637614 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-401761-f8dbb88dc-9ks76" event={"ID":"8ab573d4-4b87-40e0-9d16-ff7fdd3c3963","Type":"ContainerStarted","Data":"9ce8304ec062a9b80afc6829cb5375563596b4665dc66b7a68a7f4f27fc85192"} Oct 31 04:39:11.637846 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:11.637682 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-401761-f8dbb88dc-9ks76" event={"ID":"8ab573d4-4b87-40e0-9d16-ff7fdd3c3963","Type":"ContainerStarted","Data":"6bcf65fcf5fc1220b7191798e72f69ec092f87335b59a3a0fbc20ad042d538f4"} Oct 31 04:39:11.637846 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:11.637714 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-401761-f8dbb88dc-9ks76" Oct 31 04:39:11.639428 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:11.639401 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-401761-6ddf8dc4bf-fbwrw" event={"ID":"0551b77c-f034-48e4-b643-3361b903df31","Type":"ContainerStarted","Data":"070a0fdc33137ca68093bd7a4a18a9dc4e197d6cf0a9546d9f26c217536715c9"} Oct 31 04:39:11.639544 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:11.639447 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-401761-6ddf8dc4bf-fbwrw" Oct 31 04:39:11.639544 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:11.639462 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-401761-6ddf8dc4bf-fbwrw" event={"ID":"0551b77c-f034-48e4-b643-3361b903df31","Type":"ContainerStarted","Data":"f9fd237b0531d196a75cf07efa66ca19ce6ebbb7b3a6c5cc713806c47e9a47d0"} Oct 31 04:39:11.681087 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:11.681028 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-401761-f8dbb88dc-9ks76" podStartSLOduration=1.681015734 podStartE2EDuration="1.681015734s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:39:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:39:11.679112582 +0000 UTC m=+2099.393447199" watchObservedRunningTime="2026-10-31 04:39:11.681015734 +0000 UTC m=+2099.395350388" Oct 31 04:39:11.720797 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:11.720731 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-401761-6ddf8dc4bf-fbwrw" podStartSLOduration=1.720715669 podStartE2EDuration="1.720715669s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:39:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:39:11.719915349 +0000 UTC m=+2099.434249968" watchObservedRunningTime="2026-10-31 04:39:11.720715669 +0000 UTC m=+2099.435050287" Oct 31 04:39:12.017913 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:12.017878 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-25f56e-69577647c5-wvb9t"] Oct 31 04:39:12.022225 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:12.022204 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-25f56e-69577647c5-wvb9t" Oct 31 04:39:12.050003 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:12.049981 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-25f56e\"" Oct 31 04:39:12.050523 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:12.050475 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-25f56e-dockercfg-4jdvs\"" Oct 31 04:39:12.053791 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:12.053767 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-25f56e-69577647c5-wvb9t"] Oct 31 04:39:12.061982 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:12.061953 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69d5f412-aaf9-4d2a-afb0-3f76f0861259-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-25f56e-69577647c5-wvb9t\" (UID: \"69d5f412-aaf9-4d2a-afb0-3f76f0861259\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-25f56e-69577647c5-wvb9t" Oct 31 04:39:12.062208 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:12.062184 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s99jg\" (UniqueName: \"kubernetes.io/projected/69d5f412-aaf9-4d2a-afb0-3f76f0861259-kube-api-access-s99jg\") pod \"payload-pre-processing-e2e-shared-b-w0-25f56e-69577647c5-wvb9t\" (UID: \"69d5f412-aaf9-4d2a-afb0-3f76f0861259\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-25f56e-69577647c5-wvb9t" Oct 31 04:39:12.184532 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:12.184489 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s99jg\" (UniqueName: \"kubernetes.io/projected/69d5f412-aaf9-4d2a-afb0-3f76f0861259-kube-api-access-s99jg\") pod \"payload-pre-processing-e2e-shared-b-w0-25f56e-69577647c5-wvb9t\" (UID: \"69d5f412-aaf9-4d2a-afb0-3f76f0861259\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-25f56e-69577647c5-wvb9t" Oct 31 04:39:12.190861 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:12.184936 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69d5f412-aaf9-4d2a-afb0-3f76f0861259-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-25f56e-69577647c5-wvb9t\" (UID: \"69d5f412-aaf9-4d2a-afb0-3f76f0861259\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-25f56e-69577647c5-wvb9t" Oct 31 04:39:12.214872 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:12.193213 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69d5f412-aaf9-4d2a-afb0-3f76f0861259-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-25f56e-69577647c5-wvb9t\" (UID: \"69d5f412-aaf9-4d2a-afb0-3f76f0861259\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-25f56e-69577647c5-wvb9t" Oct 31 04:39:12.249523 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:12.243393 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-25f56e-bd47bbbd9-r4fhf"] Oct 31 04:39:12.271620 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:12.271534 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-25f56e-bd47bbbd9-r4fhf"] Oct 31 04:39:12.271793 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:12.271677 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-25f56e-bd47bbbd9-r4fhf" Oct 31 04:39:12.291473 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:12.290602 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s99jg\" (UniqueName: \"kubernetes.io/projected/69d5f412-aaf9-4d2a-afb0-3f76f0861259-kube-api-access-s99jg\") pod \"payload-pre-processing-e2e-shared-b-w0-25f56e-69577647c5-wvb9t\" (UID: \"69d5f412-aaf9-4d2a-afb0-3f76f0861259\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-25f56e-69577647c5-wvb9t" Oct 31 04:39:12.336658 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:12.336627 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-25f56e-69577647c5-wvb9t" Oct 31 04:39:12.460270 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:12.418459 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43d89e12-3149-41df-825f-40801de7576a-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-25f56e-bd47bbbd9-r4fhf\" (UID: \"43d89e12-3149-41df-825f-40801de7576a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-25f56e-bd47bbbd9-r4fhf" Oct 31 04:39:12.460270 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:12.418512 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8tvs\" (UniqueName: \"kubernetes.io/projected/43d89e12-3149-41df-825f-40801de7576a-kube-api-access-d8tvs\") pod \"payload-processing-e2e-shared-b-w0-25f56e-bd47bbbd9-r4fhf\" (UID: \"43d89e12-3149-41df-825f-40801de7576a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-25f56e-bd47bbbd9-r4fhf" Oct 31 04:39:12.520322 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:12.520291 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43d89e12-3149-41df-825f-40801de7576a-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-25f56e-bd47bbbd9-r4fhf\" (UID: \"43d89e12-3149-41df-825f-40801de7576a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-25f56e-bd47bbbd9-r4fhf" Oct 31 04:39:12.520524 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:12.520354 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8tvs\" (UniqueName: \"kubernetes.io/projected/43d89e12-3149-41df-825f-40801de7576a-kube-api-access-d8tvs\") pod \"payload-processing-e2e-shared-b-w0-25f56e-bd47bbbd9-r4fhf\" (UID: \"43d89e12-3149-41df-825f-40801de7576a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-25f56e-bd47bbbd9-r4fhf" Oct 31 04:39:12.521524 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:12.521499 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43d89e12-3149-41df-825f-40801de7576a-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-25f56e-bd47bbbd9-r4fhf\" (UID: \"43d89e12-3149-41df-825f-40801de7576a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-25f56e-bd47bbbd9-r4fhf" Oct 31 04:39:12.544151 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:12.544093 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8tvs\" (UniqueName: \"kubernetes.io/projected/43d89e12-3149-41df-825f-40801de7576a-kube-api-access-d8tvs\") pod \"payload-processing-e2e-shared-b-w0-25f56e-bd47bbbd9-r4fhf\" (UID: \"43d89e12-3149-41df-825f-40801de7576a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-25f56e-bd47bbbd9-r4fhf" Oct 31 04:39:12.550231 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:12.550211 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-afc519-69cbc98467-l7w68" Oct 31 04:39:12.608178 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:12.608155 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-25f56e-bd47bbbd9-r4fhf" Oct 31 04:39:12.642005 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:39:12.641975 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69d5f412_aaf9_4d2a_afb0_3f76f0861259.slice/crio-9f7233d559ca8f4166d1295101f33b7a64337d2b75aa911322467ce167e184ad WatchSource:0}: Error finding container 9f7233d559ca8f4166d1295101f33b7a64337d2b75aa911322467ce167e184ad: Status 404 returned error can't find the container with id 9f7233d559ca8f4166d1295101f33b7a64337d2b75aa911322467ce167e184ad Oct 31 04:39:12.644111 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:12.644078 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-25f56e-69577647c5-wvb9t"] Oct 31 04:39:12.932471 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:39:12.932441 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43d89e12_3149_41df_825f_40801de7576a.slice/crio-0af22bd332e4a66b41d2baee11349ed366462ceae9504e004025c0dfbf0ca70d WatchSource:0}: Error finding container 0af22bd332e4a66b41d2baee11349ed366462ceae9504e004025c0dfbf0ca70d: Status 404 returned error can't find the container with id 0af22bd332e4a66b41d2baee11349ed366462ceae9504e004025c0dfbf0ca70d Oct 31 04:39:12.933949 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:12.933928 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-25f56e-bd47bbbd9-r4fhf"] Oct 31 04:39:13.662486 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:13.662447 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-25f56e-69577647c5-wvb9t" event={"ID":"69d5f412-aaf9-4d2a-afb0-3f76f0861259","Type":"ContainerStarted","Data":"90b51dfd6d3297a2dbbc7bfc2f535824c07608cd387cb1587293c431c18c9dc0"} Oct 31 04:39:13.662915 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:13.662516 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-25f56e-69577647c5-wvb9t" Oct 31 04:39:13.662915 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:13.662538 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-25f56e-69577647c5-wvb9t" event={"ID":"69d5f412-aaf9-4d2a-afb0-3f76f0861259","Type":"ContainerStarted","Data":"9f7233d559ca8f4166d1295101f33b7a64337d2b75aa911322467ce167e184ad"} Oct 31 04:39:13.664255 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:13.664229 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-25f56e-bd47bbbd9-r4fhf" event={"ID":"43d89e12-3149-41df-825f-40801de7576a","Type":"ContainerStarted","Data":"9aa705e067a14e38ce68e66c9b17454a1612d5d2566fd7ca585063485bbec729"} Oct 31 04:39:13.664413 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:13.664271 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-25f56e-bd47bbbd9-r4fhf" event={"ID":"43d89e12-3149-41df-825f-40801de7576a","Type":"ContainerStarted","Data":"0af22bd332e4a66b41d2baee11349ed366462ceae9504e004025c0dfbf0ca70d"} Oct 31 04:39:13.664413 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:13.664354 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-25f56e-bd47bbbd9-r4fhf" Oct 31 04:39:13.707785 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:13.707738 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-25f56e-69577647c5-wvb9t" podStartSLOduration=2.70772157 podStartE2EDuration="2.70772157s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:39:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:39:13.706661937 +0000 UTC m=+2101.420996555" watchObservedRunningTime="2026-10-31 04:39:13.70772157 +0000 UTC m=+2101.422056186" Oct 31 04:39:13.750412 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:13.750359 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-25f56e-bd47bbbd9-r4fhf" podStartSLOduration=1.750342577 podStartE2EDuration="1.750342577s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:39:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:39:13.747300605 +0000 UTC m=+2101.461635223" watchObservedRunningTime="2026-10-31 04:39:13.750342577 +0000 UTC m=+2101.464677194" Oct 31 04:39:14.102781 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:14.102704 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-654741-6d9cc779bc-wxdnw"] Oct 31 04:39:14.108051 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:14.108032 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-654741-6d9cc779bc-wxdnw" Oct 31 04:39:14.125825 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:14.125799 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-654741-serving-cert\"" Oct 31 04:39:14.141197 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:14.141170 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-654741-6d9cc779bc-wxdnw"] Oct 31 04:39:14.145873 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:14.145848 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6f4q\" (UniqueName: \"kubernetes.io/projected/84000b34-2ffa-4914-97f3-73bafc243ec5-kube-api-access-t6f4q\") pod \"maas-api-e2e-extmod-a-654741-6d9cc779bc-wxdnw\" (UID: \"84000b34-2ffa-4914-97f3-73bafc243ec5\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-654741-6d9cc779bc-wxdnw" Oct 31 04:39:14.146016 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:14.145913 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84000b34-2ffa-4914-97f3-73bafc243ec5-metrics-certs\") pod \"maas-api-e2e-extmod-a-654741-6d9cc779bc-wxdnw\" (UID: \"84000b34-2ffa-4914-97f3-73bafc243ec5\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-654741-6d9cc779bc-wxdnw" Oct 31 04:39:14.146016 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:14.145965 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84000b34-2ffa-4914-97f3-73bafc243ec5-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-654741-6d9cc779bc-wxdnw\" (UID: \"84000b34-2ffa-4914-97f3-73bafc243ec5\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-654741-6d9cc779bc-wxdnw" Oct 31 04:39:14.146187 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:14.146163 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84000b34-2ffa-4914-97f3-73bafc243ec5-maas-api-tls\") pod \"maas-api-e2e-extmod-a-654741-6d9cc779bc-wxdnw\" (UID: \"84000b34-2ffa-4914-97f3-73bafc243ec5\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-654741-6d9cc779bc-wxdnw" Oct 31 04:39:14.247523 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:14.247490 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6f4q\" (UniqueName: \"kubernetes.io/projected/84000b34-2ffa-4914-97f3-73bafc243ec5-kube-api-access-t6f4q\") pod \"maas-api-e2e-extmod-a-654741-6d9cc779bc-wxdnw\" (UID: \"84000b34-2ffa-4914-97f3-73bafc243ec5\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-654741-6d9cc779bc-wxdnw" Oct 31 04:39:14.247691 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:14.247553 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84000b34-2ffa-4914-97f3-73bafc243ec5-metrics-certs\") pod \"maas-api-e2e-extmod-a-654741-6d9cc779bc-wxdnw\" (UID: \"84000b34-2ffa-4914-97f3-73bafc243ec5\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-654741-6d9cc779bc-wxdnw" Oct 31 04:39:14.247691 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:14.247570 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84000b34-2ffa-4914-97f3-73bafc243ec5-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-654741-6d9cc779bc-wxdnw\" (UID: \"84000b34-2ffa-4914-97f3-73bafc243ec5\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-654741-6d9cc779bc-wxdnw" Oct 31 04:39:14.247691 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:14.247622 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84000b34-2ffa-4914-97f3-73bafc243ec5-maas-api-tls\") pod \"maas-api-e2e-extmod-a-654741-6d9cc779bc-wxdnw\" (UID: \"84000b34-2ffa-4914-97f3-73bafc243ec5\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-654741-6d9cc779bc-wxdnw" Oct 31 04:39:14.247855 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:39:14.247788 4376 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-654741-serving-cert: secret "maas-api-e2e-extmod-a-654741-serving-cert" not found Oct 31 04:39:14.247910 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:39:14.247864 4376 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84000b34-2ffa-4914-97f3-73bafc243ec5-maas-api-tls podName:84000b34-2ffa-4914-97f3-73bafc243ec5 nodeName:}" failed. No retries permitted until 2026-10-31 04:39:14.747842327 +0000 UTC m=+2102.462176936 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/84000b34-2ffa-4914-97f3-73bafc243ec5-maas-api-tls") pod "maas-api-e2e-extmod-a-654741-6d9cc779bc-wxdnw" (UID: "84000b34-2ffa-4914-97f3-73bafc243ec5") : secret "maas-api-e2e-extmod-a-654741-serving-cert" not found Oct 31 04:39:14.248289 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:14.248269 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84000b34-2ffa-4914-97f3-73bafc243ec5-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-654741-6d9cc779bc-wxdnw\" (UID: \"84000b34-2ffa-4914-97f3-73bafc243ec5\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-654741-6d9cc779bc-wxdnw" Oct 31 04:39:14.250089 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:14.250065 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84000b34-2ffa-4914-97f3-73bafc243ec5-metrics-certs\") pod \"maas-api-e2e-extmod-a-654741-6d9cc779bc-wxdnw\" (UID: \"84000b34-2ffa-4914-97f3-73bafc243ec5\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-654741-6d9cc779bc-wxdnw" Oct 31 04:39:14.273718 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:14.273691 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6f4q\" (UniqueName: \"kubernetes.io/projected/84000b34-2ffa-4914-97f3-73bafc243ec5-kube-api-access-t6f4q\") pod \"maas-api-e2e-extmod-a-654741-6d9cc779bc-wxdnw\" (UID: \"84000b34-2ffa-4914-97f3-73bafc243ec5\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-654741-6d9cc779bc-wxdnw" Oct 31 04:39:14.754699 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:14.754665 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84000b34-2ffa-4914-97f3-73bafc243ec5-maas-api-tls\") pod \"maas-api-e2e-extmod-a-654741-6d9cc779bc-wxdnw\" (UID: \"84000b34-2ffa-4914-97f3-73bafc243ec5\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-654741-6d9cc779bc-wxdnw" Oct 31 04:39:14.757244 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:14.757212 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84000b34-2ffa-4914-97f3-73bafc243ec5-maas-api-tls\") pod \"maas-api-e2e-extmod-a-654741-6d9cc779bc-wxdnw\" (UID: \"84000b34-2ffa-4914-97f3-73bafc243ec5\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-654741-6d9cc779bc-wxdnw" Oct 31 04:39:15.022446 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:15.022362 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-654741-6d9cc779bc-wxdnw" Oct 31 04:39:15.348583 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:39:15.348553 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84000b34_2ffa_4914_97f3_73bafc243ec5.slice/crio-fb139508a7bfcfa79bc390a5099e941e0ea8733026950ffa331a38481b8bfd58 WatchSource:0}: Error finding container fb139508a7bfcfa79bc390a5099e941e0ea8733026950ffa331a38481b8bfd58: Status 404 returned error can't find the container with id fb139508a7bfcfa79bc390a5099e941e0ea8733026950ffa331a38481b8bfd58 Oct 31 04:39:15.350283 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:15.350257 4376 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 04:39:15.350459 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:15.350442 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-654741-6d9cc779bc-wxdnw"] Oct 31 04:39:15.679650 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:15.679614 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-654741-6d9cc779bc-wxdnw" event={"ID":"84000b34-2ffa-4914-97f3-73bafc243ec5","Type":"ContainerStarted","Data":"fb139508a7bfcfa79bc390a5099e941e0ea8733026950ffa331a38481b8bfd58"} Oct 31 04:39:16.686943 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:16.686902 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-654741-6d9cc779bc-wxdnw" event={"ID":"84000b34-2ffa-4914-97f3-73bafc243ec5","Type":"ContainerStarted","Data":"594800ec68e4702c2aece86edab37f7e0fd8a860c89daccceac4d6a698b9acd2"} Oct 31 04:39:16.687305 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:16.687124 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-654741-6d9cc779bc-wxdnw" Oct 31 04:39:16.733440 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:16.733392 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-654741-6d9cc779bc-wxdnw" podStartSLOduration=2.193244259 podStartE2EDuration="2.733379245s" totalImagesPullingTime="540.134986ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:39:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:39:16.730945629 +0000 UTC m=+2104.445280246" watchObservedRunningTime="2026-10-31 04:39:16.733379245 +0000 UTC m=+2104.447713897" Oct 31 04:39:21.488306 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:21.488268 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 04:39:21.932831 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:21.932790 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-9x5tt"] Oct 31 04:39:21.933043 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:21.933022 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-9x5tt" podUID="b61c092a-60dc-4bca-b24d-aa63f10dbf14" containerName="istio-proxy" containerID="cri-o://3563a5c957fb617b8bac32d8eaf616f37ceda13f239d12a496f054b98d6d9613" gracePeriod=30 Oct 31 04:39:22.679957 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:22.678529 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-401761-f8dbb88dc-9ks76" Oct 31 04:39:22.680487 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:22.679973 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-401761-6ddf8dc4bf-fbwrw" Oct 31 04:39:22.767361 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:22.766524 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-654741-6d9cc779bc-wxdnw" Oct 31 04:39:24.674461 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:24.674427 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-25f56e-bd47bbbd9-r4fhf" Oct 31 04:39:24.674863 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:24.674506 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-25f56e-69577647c5-wvb9t" Oct 31 04:39:27.182185 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:27.182145 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-afc519-68c697db57-tkv9n"] Oct 31 04:39:27.192888 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:27.192867 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-afc519-68c697db57-tkv9n" Oct 31 04:39:27.222908 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:27.214314 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-afc519-68c697db57-tkv9n"] Oct 31 04:39:27.222908 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:27.215045 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-afc519-serving-cert\"" Oct 31 04:39:27.222908 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:27.221606 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b65451e-2626-4b42-9340-d6db06b74760-maas-api-tls\") pod \"maas-api-e2e-extmod-a-afc519-68c697db57-tkv9n\" (UID: \"7b65451e-2626-4b42-9340-d6db06b74760\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-afc519-68c697db57-tkv9n" Oct 31 04:39:27.222908 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:27.221835 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b65451e-2626-4b42-9340-d6db06b74760-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-afc519-68c697db57-tkv9n\" (UID: \"7b65451e-2626-4b42-9340-d6db06b74760\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-afc519-68c697db57-tkv9n" Oct 31 04:39:27.222908 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:27.221932 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b65451e-2626-4b42-9340-d6db06b74760-metrics-certs\") pod \"maas-api-e2e-extmod-a-afc519-68c697db57-tkv9n\" (UID: \"7b65451e-2626-4b42-9340-d6db06b74760\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-afc519-68c697db57-tkv9n" Oct 31 04:39:27.222908 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:27.221972 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzh74\" (UniqueName: \"kubernetes.io/projected/7b65451e-2626-4b42-9340-d6db06b74760-kube-api-access-rzh74\") pod \"maas-api-e2e-extmod-a-afc519-68c697db57-tkv9n\" (UID: \"7b65451e-2626-4b42-9340-d6db06b74760\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-afc519-68c697db57-tkv9n" Oct 31 04:39:27.227418 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:27.227388 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-9x5tt" Oct 31 04:39:27.323387 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:27.323350 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b61c092a-60dc-4bca-b24d-aa63f10dbf14-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b61c092a-60dc-4bca-b24d-aa63f10dbf14-workload-socket\") pod \"b61c092a-60dc-4bca-b24d-aa63f10dbf14\" (UID: \"b61c092a-60dc-4bca-b24d-aa63f10dbf14\") " Oct 31 04:39:27.323572 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:27.323412 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b61c092a-60dc-4bca-b24d-aa63f10dbf14-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b61c092a-60dc-4bca-b24d-aa63f10dbf14-credential-socket\") pod \"b61c092a-60dc-4bca-b24d-aa63f10dbf14\" (UID: \"b61c092a-60dc-4bca-b24d-aa63f10dbf14\") " Oct 31 04:39:27.323572 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:27.323499 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b61c092a-60dc-4bca-b24d-aa63f10dbf14-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b61c092a-60dc-4bca-b24d-aa63f10dbf14-istio-data\") pod \"b61c092a-60dc-4bca-b24d-aa63f10dbf14\" (UID: \"b61c092a-60dc-4bca-b24d-aa63f10dbf14\") " Oct 31 04:39:27.323572 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:27.323545 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b61c092a-60dc-4bca-b24d-aa63f10dbf14-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b61c092a-60dc-4bca-b24d-aa63f10dbf14-istio-podinfo\") pod \"b61c092a-60dc-4bca-b24d-aa63f10dbf14\" (UID: \"b61c092a-60dc-4bca-b24d-aa63f10dbf14\") " Oct 31 04:39:27.323760 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:27.323651 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b61c092a-60dc-4bca-b24d-aa63f10dbf14-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b61c092a-60dc-4bca-b24d-aa63f10dbf14-istio-envoy\") pod \"b61c092a-60dc-4bca-b24d-aa63f10dbf14\" (UID: \"b61c092a-60dc-4bca-b24d-aa63f10dbf14\") " Oct 31 04:39:27.323760 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:27.323695 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b61c092a-60dc-4bca-b24d-aa63f10dbf14-kube-api-access-ns2l6\" (UniqueName: \"kubernetes.io/projected/b61c092a-60dc-4bca-b24d-aa63f10dbf14-kube-api-access-ns2l6\") pod \"b61c092a-60dc-4bca-b24d-aa63f10dbf14\" (UID: \"b61c092a-60dc-4bca-b24d-aa63f10dbf14\") " Oct 31 04:39:27.323760 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:27.323674 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b61c092a-60dc-4bca-b24d-aa63f10dbf14-workload-socket" pod "b61c092a-60dc-4bca-b24d-aa63f10dbf14" (UID: "b61c092a-60dc-4bca-b24d-aa63f10dbf14"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:39:27.323760 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:27.323731 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b61c092a-60dc-4bca-b24d-aa63f10dbf14-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b61c092a-60dc-4bca-b24d-aa63f10dbf14-istiod-ca-cert\") pod \"b61c092a-60dc-4bca-b24d-aa63f10dbf14\" (UID: \"b61c092a-60dc-4bca-b24d-aa63f10dbf14\") " Oct 31 04:39:27.323760 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:27.323734 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b61c092a-60dc-4bca-b24d-aa63f10dbf14-credential-socket" pod "b61c092a-60dc-4bca-b24d-aa63f10dbf14" (UID: "b61c092a-60dc-4bca-b24d-aa63f10dbf14"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:39:27.324048 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:27.323765 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b61c092a-60dc-4bca-b24d-aa63f10dbf14-istio-token\" (UniqueName: \"kubernetes.io/projected/b61c092a-60dc-4bca-b24d-aa63f10dbf14-istio-token\") pod \"b61c092a-60dc-4bca-b24d-aa63f10dbf14\" (UID: \"b61c092a-60dc-4bca-b24d-aa63f10dbf14\") " Oct 31 04:39:27.324048 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:27.323839 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b61c092a-60dc-4bca-b24d-aa63f10dbf14-istio-data" pod "b61c092a-60dc-4bca-b24d-aa63f10dbf14" (UID: "b61c092a-60dc-4bca-b24d-aa63f10dbf14"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:39:27.324186 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:27.324152 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b61c092a-60dc-4bca-b24d-aa63f10dbf14-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b61c092a-60dc-4bca-b24d-aa63f10dbf14-workload-certs\") pod \"b61c092a-60dc-4bca-b24d-aa63f10dbf14\" (UID: \"b61c092a-60dc-4bca-b24d-aa63f10dbf14\") " Oct 31 04:39:27.324243 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:27.324186 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b61c092a-60dc-4bca-b24d-aa63f10dbf14-istiod-ca-cert" pod "b61c092a-60dc-4bca-b24d-aa63f10dbf14" (UID: "b61c092a-60dc-4bca-b24d-aa63f10dbf14"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:39:27.324418 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:27.324394 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b61c092a-60dc-4bca-b24d-aa63f10dbf14-workload-certs" pod "b61c092a-60dc-4bca-b24d-aa63f10dbf14" (UID: "b61c092a-60dc-4bca-b24d-aa63f10dbf14"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:39:27.324496 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:27.324479 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b65451e-2626-4b42-9340-d6db06b74760-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-afc519-68c697db57-tkv9n\" (UID: \"7b65451e-2626-4b42-9340-d6db06b74760\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-afc519-68c697db57-tkv9n" Oct 31 04:39:27.324597 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:27.324581 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b65451e-2626-4b42-9340-d6db06b74760-metrics-certs\") pod \"maas-api-e2e-extmod-a-afc519-68c697db57-tkv9n\" (UID: \"7b65451e-2626-4b42-9340-d6db06b74760\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-afc519-68c697db57-tkv9n" Oct 31 04:39:27.324664 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:27.324625 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzh74\" (UniqueName: \"kubernetes.io/projected/7b65451e-2626-4b42-9340-d6db06b74760-kube-api-access-rzh74\") pod \"maas-api-e2e-extmod-a-afc519-68c697db57-tkv9n\" (UID: \"7b65451e-2626-4b42-9340-d6db06b74760\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-afc519-68c697db57-tkv9n" Oct 31 04:39:27.324745 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:27.324728 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b65451e-2626-4b42-9340-d6db06b74760-maas-api-tls\") pod \"maas-api-e2e-extmod-a-afc519-68c697db57-tkv9n\" (UID: \"7b65451e-2626-4b42-9340-d6db06b74760\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-afc519-68c697db57-tkv9n" Oct 31 04:39:27.325510 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:27.325047 4376 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b61c092a-60dc-4bca-b24d-aa63f10dbf14-istiod-ca-cert\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:39:27.325510 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:27.325067 4376 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b61c092a-60dc-4bca-b24d-aa63f10dbf14-workload-certs\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:39:27.325510 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:27.325080 4376 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b61c092a-60dc-4bca-b24d-aa63f10dbf14-workload-socket\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:39:27.325510 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:27.325093 4376 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b61c092a-60dc-4bca-b24d-aa63f10dbf14-credential-socket\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:39:27.325510 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:27.325106 4376 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b61c092a-60dc-4bca-b24d-aa63f10dbf14-istio-data\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:39:27.325510 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:39:27.325183 4376 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-afc519-serving-cert: secret "maas-api-e2e-extmod-a-afc519-serving-cert" not found Oct 31 04:39:27.325510 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:39:27.325249 4376 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b65451e-2626-4b42-9340-d6db06b74760-maas-api-tls podName:7b65451e-2626-4b42-9340-d6db06b74760 nodeName:}" failed. No retries permitted until 2026-10-31 04:39:27.825230412 +0000 UTC m=+2115.539565021 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7b65451e-2626-4b42-9340-d6db06b74760-maas-api-tls") pod "maas-api-e2e-extmod-a-afc519-68c697db57-tkv9n" (UID: "7b65451e-2626-4b42-9340-d6db06b74760") : secret "maas-api-e2e-extmod-a-afc519-serving-cert" not found Oct 31 04:39:27.325510 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:27.325306 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b65451e-2626-4b42-9340-d6db06b74760-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-afc519-68c697db57-tkv9n\" (UID: \"7b65451e-2626-4b42-9340-d6db06b74760\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-afc519-68c697db57-tkv9n" Oct 31 04:39:27.327289 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:27.327264 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b65451e-2626-4b42-9340-d6db06b74760-metrics-certs\") pod \"maas-api-e2e-extmod-a-afc519-68c697db57-tkv9n\" (UID: \"7b65451e-2626-4b42-9340-d6db06b74760\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-afc519-68c697db57-tkv9n" Oct 31 04:39:27.327439 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:27.327291 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b61c092a-60dc-4bca-b24d-aa63f10dbf14-istio-token" pod "b61c092a-60dc-4bca-b24d-aa63f10dbf14" (UID: "b61c092a-60dc-4bca-b24d-aa63f10dbf14"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:39:27.327439 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:27.327361 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b61c092a-60dc-4bca-b24d-aa63f10dbf14-kube-api-access-ns2l6" pod "b61c092a-60dc-4bca-b24d-aa63f10dbf14" (UID: "b61c092a-60dc-4bca-b24d-aa63f10dbf14"). InnerVolumeSpecName "kube-api-access-ns2l6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:39:27.327614 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:27.327593 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b61c092a-60dc-4bca-b24d-aa63f10dbf14-istio-envoy" pod "b61c092a-60dc-4bca-b24d-aa63f10dbf14" (UID: "b61c092a-60dc-4bca-b24d-aa63f10dbf14"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:39:27.328383 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:27.328360 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b61c092a-60dc-4bca-b24d-aa63f10dbf14-istio-podinfo" pod "b61c092a-60dc-4bca-b24d-aa63f10dbf14" (UID: "b61c092a-60dc-4bca-b24d-aa63f10dbf14"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 04:39:27.352649 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:27.352617 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzh74\" (UniqueName: \"kubernetes.io/projected/7b65451e-2626-4b42-9340-d6db06b74760-kube-api-access-rzh74\") pod \"maas-api-e2e-extmod-a-afc519-68c697db57-tkv9n\" (UID: \"7b65451e-2626-4b42-9340-d6db06b74760\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-afc519-68c697db57-tkv9n" Oct 31 04:39:27.426861 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:27.426816 4376 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b61c092a-60dc-4bca-b24d-aa63f10dbf14-istio-envoy\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:39:27.426861 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:27.426852 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ns2l6\" (UniqueName: \"kubernetes.io/projected/b61c092a-60dc-4bca-b24d-aa63f10dbf14-kube-api-access-ns2l6\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:39:27.426861 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:27.426867 4376 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b61c092a-60dc-4bca-b24d-aa63f10dbf14-istio-token\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:39:27.427113 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:27.426879 4376 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b61c092a-60dc-4bca-b24d-aa63f10dbf14-istio-podinfo\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:39:27.841455 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:27.841414 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b65451e-2626-4b42-9340-d6db06b74760-maas-api-tls\") pod \"maas-api-e2e-extmod-a-afc519-68c697db57-tkv9n\" (UID: \"7b65451e-2626-4b42-9340-d6db06b74760\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-afc519-68c697db57-tkv9n" Oct 31 04:39:27.844281 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:27.844249 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b65451e-2626-4b42-9340-d6db06b74760-maas-api-tls\") pod \"maas-api-e2e-extmod-a-afc519-68c697db57-tkv9n\" (UID: \"7b65451e-2626-4b42-9340-d6db06b74760\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-afc519-68c697db57-tkv9n" Oct 31 04:39:27.845097 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:27.845078 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-afc519-68c697db57-tkv9n" Oct 31 04:39:27.870847 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:27.870823 4376 generic.go:356] "Generic (PLEG): container finished" podID="b61c092a-60dc-4bca-b24d-aa63f10dbf14" containerID="3563a5c957fb617b8bac32d8eaf616f37ceda13f239d12a496f054b98d6d9613" exitCode=0 Oct 31 04:39:27.870960 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:27.870894 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-9x5tt" Oct 31 04:39:27.870960 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:27.870906 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-9x5tt" event={"ID":"b61c092a-60dc-4bca-b24d-aa63f10dbf14","Type":"ContainerDied","Data":"3563a5c957fb617b8bac32d8eaf616f37ceda13f239d12a496f054b98d6d9613"} Oct 31 04:39:27.871081 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:27.871002 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-9x5tt" event={"ID":"b61c092a-60dc-4bca-b24d-aa63f10dbf14","Type":"ContainerDied","Data":"6627681a17954fa1a19dac8f0c16173849ac80f07cfb33bae81320fd2e30e67c"} Oct 31 04:39:27.871081 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:27.871036 4376 scope.go:122] "RemoveContainer" containerID="3563a5c957fb617b8bac32d8eaf616f37ceda13f239d12a496f054b98d6d9613" Oct 31 04:39:27.883578 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:27.883561 4376 scope.go:122] "RemoveContainer" containerID="3563a5c957fb617b8bac32d8eaf616f37ceda13f239d12a496f054b98d6d9613" Oct 31 04:39:27.883869 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:39:27.883846 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3563a5c957fb617b8bac32d8eaf616f37ceda13f239d12a496f054b98d6d9613\": container with ID starting with 3563a5c957fb617b8bac32d8eaf616f37ceda13f239d12a496f054b98d6d9613 not found: ID does not exist" containerID="3563a5c957fb617b8bac32d8eaf616f37ceda13f239d12a496f054b98d6d9613" Oct 31 04:39:27.883962 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:27.883880 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3563a5c957fb617b8bac32d8eaf616f37ceda13f239d12a496f054b98d6d9613"} err="failed to get container status \"3563a5c957fb617b8bac32d8eaf616f37ceda13f239d12a496f054b98d6d9613\": rpc error: code = NotFound desc = could not find container \"3563a5c957fb617b8bac32d8eaf616f37ceda13f239d12a496f054b98d6d9613\": container with ID starting with 3563a5c957fb617b8bac32d8eaf616f37ceda13f239d12a496f054b98d6d9613 not found: ID does not exist" Oct 31 04:39:27.936431 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:27.936375 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-9x5tt"] Oct 31 04:39:27.939868 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:27.939841 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-9x5tt"] Oct 31 04:39:28.154969 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:39:28.154930 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b65451e_2626_4b42_9340_d6db06b74760.slice/crio-bb805b6c5d81cbd928ea8683dd5d8a30fc04758aa7b685697079611a97dfadb5 WatchSource:0}: Error finding container bb805b6c5d81cbd928ea8683dd5d8a30fc04758aa7b685697079611a97dfadb5: Status 404 returned error can't find the container with id bb805b6c5d81cbd928ea8683dd5d8a30fc04758aa7b685697079611a97dfadb5 Oct 31 04:39:28.155680 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:28.155653 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-afc519-68c697db57-tkv9n"] Oct 31 04:39:28.519750 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:28.519213 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b61c092a-60dc-4bca-b24d-aa63f10dbf14" path="/var/lib/kubelet/pods/b61c092a-60dc-4bca-b24d-aa63f10dbf14/volumes" Oct 31 04:39:28.882406 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:28.882365 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-afc519-68c697db57-tkv9n" event={"ID":"7b65451e-2626-4b42-9340-d6db06b74760","Type":"ContainerStarted","Data":"69b6eebacfbc54c2253bd6f8930c73f35aa4948bae2cf3f93702cb17ccba0979"} Oct 31 04:39:28.882601 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:28.882443 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-afc519-68c697db57-tkv9n" Oct 31 04:39:28.882601 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:28.882462 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-afc519-68c697db57-tkv9n" event={"ID":"7b65451e-2626-4b42-9340-d6db06b74760","Type":"ContainerStarted","Data":"bb805b6c5d81cbd928ea8683dd5d8a30fc04758aa7b685697079611a97dfadb5"} Oct 31 04:39:28.929136 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:28.929076 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-afc519-68c697db57-tkv9n" podStartSLOduration=1.53783767 podStartE2EDuration="1.929058531s" totalImagesPullingTime="391.220861ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:39:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:39:28.92651329 +0000 UTC m=+2116.640847907" watchObservedRunningTime="2026-10-31 04:39:28.929058531 +0000 UTC m=+2116.643393166" Oct 31 04:39:34.903675 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:34.903647 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-afc519-68c697db57-tkv9n" Oct 31 04:39:39.912580 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:39.912543 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-401761-c5bb964f-h6p9q"] Oct 31 04:39:39.913511 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:39.913491 4376 cpu_manager.go:414] "containerMap: removing container" podUID="b61c092a-60dc-4bca-b24d-aa63f10dbf14" containerName="istio-proxy" Oct 31 04:39:39.913580 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:39.913515 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b61c092a-60dc-4bca-b24d-aa63f10dbf14" containerName="istio-proxy" Oct 31 04:39:39.913664 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:39.913653 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="b61c092a-60dc-4bca-b24d-aa63f10dbf14" containerName="istio-proxy" Oct 31 04:39:39.917113 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:39.917097 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-401761-c5bb964f-h6p9q" Oct 31 04:39:39.935643 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:39.935614 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0b21674-516a-452f-86c5-b27115eb72e3-maas-api-tls\") pod \"maas-api-e2e-extmod-a-401761-c5bb964f-h6p9q\" (UID: \"c0b21674-516a-452f-86c5-b27115eb72e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-401761-c5bb964f-h6p9q" Oct 31 04:39:39.935788 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:39.935768 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bsg2n\" (UniqueName: \"kubernetes.io/projected/c0b21674-516a-452f-86c5-b27115eb72e3-kube-api-access-bsg2n\") pod \"maas-api-e2e-extmod-a-401761-c5bb964f-h6p9q\" (UID: \"c0b21674-516a-452f-86c5-b27115eb72e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-401761-c5bb964f-h6p9q" Oct 31 04:39:39.935898 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:39.935872 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0b21674-516a-452f-86c5-b27115eb72e3-metrics-certs\") pod \"maas-api-e2e-extmod-a-401761-c5bb964f-h6p9q\" (UID: \"c0b21674-516a-452f-86c5-b27115eb72e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-401761-c5bb964f-h6p9q" Oct 31 04:39:39.935996 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:39.935972 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0b21674-516a-452f-86c5-b27115eb72e3-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-401761-c5bb964f-h6p9q\" (UID: \"c0b21674-516a-452f-86c5-b27115eb72e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-401761-c5bb964f-h6p9q" Oct 31 04:39:39.937093 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:39.937075 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-401761-c5bb964f-h6p9q"] Oct 31 04:39:39.948051 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:39.948030 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-401761-serving-cert\"" Oct 31 04:39:40.037159 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:40.037113 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsg2n\" (UniqueName: \"kubernetes.io/projected/c0b21674-516a-452f-86c5-b27115eb72e3-kube-api-access-bsg2n\") pod \"maas-api-e2e-extmod-a-401761-c5bb964f-h6p9q\" (UID: \"c0b21674-516a-452f-86c5-b27115eb72e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-401761-c5bb964f-h6p9q" Oct 31 04:39:40.037391 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:40.037200 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0b21674-516a-452f-86c5-b27115eb72e3-metrics-certs\") pod \"maas-api-e2e-extmod-a-401761-c5bb964f-h6p9q\" (UID: \"c0b21674-516a-452f-86c5-b27115eb72e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-401761-c5bb964f-h6p9q" Oct 31 04:39:40.037391 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:40.037251 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0b21674-516a-452f-86c5-b27115eb72e3-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-401761-c5bb964f-h6p9q\" (UID: \"c0b21674-516a-452f-86c5-b27115eb72e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-401761-c5bb964f-h6p9q" Oct 31 04:39:40.037529 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:40.037398 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0b21674-516a-452f-86c5-b27115eb72e3-maas-api-tls\") pod \"maas-api-e2e-extmod-a-401761-c5bb964f-h6p9q\" (UID: \"c0b21674-516a-452f-86c5-b27115eb72e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-401761-c5bb964f-h6p9q" Oct 31 04:39:40.037529 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:39:40.037469 4376 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-401761-serving-cert: secret "maas-api-e2e-extmod-a-401761-serving-cert" not found Oct 31 04:39:40.037625 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:39:40.037556 4376 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0b21674-516a-452f-86c5-b27115eb72e3-maas-api-tls podName:c0b21674-516a-452f-86c5-b27115eb72e3 nodeName:}" failed. No retries permitted until 2026-10-31 04:39:40.53753565 +0000 UTC m=+2128.251870262 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c0b21674-516a-452f-86c5-b27115eb72e3-maas-api-tls") pod "maas-api-e2e-extmod-a-401761-c5bb964f-h6p9q" (UID: "c0b21674-516a-452f-86c5-b27115eb72e3") : secret "maas-api-e2e-extmod-a-401761-serving-cert" not found Oct 31 04:39:40.037979 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:40.037962 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0b21674-516a-452f-86c5-b27115eb72e3-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-401761-c5bb964f-h6p9q\" (UID: \"c0b21674-516a-452f-86c5-b27115eb72e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-401761-c5bb964f-h6p9q" Oct 31 04:39:40.039692 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:40.039669 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0b21674-516a-452f-86c5-b27115eb72e3-metrics-certs\") pod \"maas-api-e2e-extmod-a-401761-c5bb964f-h6p9q\" (UID: \"c0b21674-516a-452f-86c5-b27115eb72e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-401761-c5bb964f-h6p9q" Oct 31 04:39:40.067366 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:40.067321 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsg2n\" (UniqueName: \"kubernetes.io/projected/c0b21674-516a-452f-86c5-b27115eb72e3-kube-api-access-bsg2n\") pod \"maas-api-e2e-extmod-a-401761-c5bb964f-h6p9q\" (UID: \"c0b21674-516a-452f-86c5-b27115eb72e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-401761-c5bb964f-h6p9q" Oct 31 04:39:40.543863 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:40.543831 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0b21674-516a-452f-86c5-b27115eb72e3-maas-api-tls\") pod \"maas-api-e2e-extmod-a-401761-c5bb964f-h6p9q\" (UID: \"c0b21674-516a-452f-86c5-b27115eb72e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-401761-c5bb964f-h6p9q" Oct 31 04:39:40.546242 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:40.546213 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0b21674-516a-452f-86c5-b27115eb72e3-maas-api-tls\") pod \"maas-api-e2e-extmod-a-401761-c5bb964f-h6p9q\" (UID: \"c0b21674-516a-452f-86c5-b27115eb72e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-401761-c5bb964f-h6p9q" Oct 31 04:39:40.830789 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:40.830685 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-401761-c5bb964f-h6p9q" Oct 31 04:39:41.002217 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:39:41.002179 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 04:39:41.164348 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:41.157277 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-401761-c5bb964f-h6p9q"] Oct 31 04:39:41.982281 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:41.982243 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-401761-c5bb964f-h6p9q" event={"ID":"c0b21674-516a-452f-86c5-b27115eb72e3","Type":"ContainerStarted","Data":"67d6bd11e8fff289fab43efd61f5680cc67fb583a4b54de8a43f5269581e8079"} Oct 31 04:39:41.982462 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:41.982298 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-401761-c5bb964f-h6p9q" event={"ID":"c0b21674-516a-452f-86c5-b27115eb72e3","Type":"ContainerStarted","Data":"0386e8e25e5f08051eb536c461f392578b4ab96c2d20a0781708cfebf6da2e7e"} Oct 31 04:39:41.982462 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:41.982346 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-401761-c5bb964f-h6p9q" Oct 31 04:39:42.025906 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:42.025856 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-401761-c5bb964f-h6p9q" podStartSLOduration=2.64514727 podStartE2EDuration="3.02584209s" totalImagesPullingTime="380.69482ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:39:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:39:42.024619882 +0000 UTC m=+2129.738954521" watchObservedRunningTime="2026-10-31 04:39:42.02584209 +0000 UTC m=+2129.740176707" Oct 31 04:39:43.777139 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:43.777103 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-25f56e-85b84d5b9b-5n9vf"] Oct 31 04:39:43.782840 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:43.782817 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-25f56e-85b84d5b9b-5n9vf" Oct 31 04:39:43.791283 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:43.791252 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nj9t\" (UniqueName: \"kubernetes.io/projected/994761dc-71d9-4334-a0aa-891d7287a690-kube-api-access-4nj9t\") pod \"maas-api-e2e-shared-b-w0-25f56e-85b84d5b9b-5n9vf\" (UID: \"994761dc-71d9-4334-a0aa-891d7287a690\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-25f56e-85b84d5b9b-5n9vf" Oct 31 04:39:43.791452 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:43.791420 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/994761dc-71d9-4334-a0aa-891d7287a690-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-25f56e-85b84d5b9b-5n9vf\" (UID: \"994761dc-71d9-4334-a0aa-891d7287a690\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-25f56e-85b84d5b9b-5n9vf" Oct 31 04:39:43.791526 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:43.791492 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/994761dc-71d9-4334-a0aa-891d7287a690-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-25f56e-85b84d5b9b-5n9vf\" (UID: \"994761dc-71d9-4334-a0aa-891d7287a690\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-25f56e-85b84d5b9b-5n9vf" Oct 31 04:39:43.791701 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:43.791681 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/994761dc-71d9-4334-a0aa-891d7287a690-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-25f56e-85b84d5b9b-5n9vf\" (UID: \"994761dc-71d9-4334-a0aa-891d7287a690\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-25f56e-85b84d5b9b-5n9vf" Oct 31 04:39:43.812804 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:43.812764 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w0-25f56e-serving-cert\"" Oct 31 04:39:43.818768 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:43.818732 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-25f56e-85b84d5b9b-5n9vf"] Oct 31 04:39:43.893684 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:43.893651 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/994761dc-71d9-4334-a0aa-891d7287a690-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-25f56e-85b84d5b9b-5n9vf\" (UID: \"994761dc-71d9-4334-a0aa-891d7287a690\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-25f56e-85b84d5b9b-5n9vf" Oct 31 04:39:43.893684 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:43.893688 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/994761dc-71d9-4334-a0aa-891d7287a690-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-25f56e-85b84d5b9b-5n9vf\" (UID: \"994761dc-71d9-4334-a0aa-891d7287a690\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-25f56e-85b84d5b9b-5n9vf" Oct 31 04:39:43.893942 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:43.893830 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/994761dc-71d9-4334-a0aa-891d7287a690-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-25f56e-85b84d5b9b-5n9vf\" (UID: \"994761dc-71d9-4334-a0aa-891d7287a690\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-25f56e-85b84d5b9b-5n9vf" Oct 31 04:39:43.893942 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:43.893883 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nj9t\" (UniqueName: \"kubernetes.io/projected/994761dc-71d9-4334-a0aa-891d7287a690-kube-api-access-4nj9t\") pod \"maas-api-e2e-shared-b-w0-25f56e-85b84d5b9b-5n9vf\" (UID: \"994761dc-71d9-4334-a0aa-891d7287a690\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-25f56e-85b84d5b9b-5n9vf" Oct 31 04:39:43.894054 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:39:43.893960 4376 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-25f56e-serving-cert: secret "maas-api-e2e-shared-b-w0-25f56e-serving-cert" not found Oct 31 04:39:43.894054 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:39:43.894022 4376 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/994761dc-71d9-4334-a0aa-891d7287a690-maas-api-tls podName:994761dc-71d9-4334-a0aa-891d7287a690 nodeName:}" failed. No retries permitted until 2026-10-31 04:39:44.394006326 +0000 UTC m=+2132.108340921 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/994761dc-71d9-4334-a0aa-891d7287a690-maas-api-tls") pod "maas-api-e2e-shared-b-w0-25f56e-85b84d5b9b-5n9vf" (UID: "994761dc-71d9-4334-a0aa-891d7287a690") : secret "maas-api-e2e-shared-b-w0-25f56e-serving-cert" not found Oct 31 04:39:43.894499 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:43.894473 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/994761dc-71d9-4334-a0aa-891d7287a690-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-25f56e-85b84d5b9b-5n9vf\" (UID: \"994761dc-71d9-4334-a0aa-891d7287a690\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-25f56e-85b84d5b9b-5n9vf" Oct 31 04:39:43.896079 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:43.896059 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/994761dc-71d9-4334-a0aa-891d7287a690-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-25f56e-85b84d5b9b-5n9vf\" (UID: \"994761dc-71d9-4334-a0aa-891d7287a690\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-25f56e-85b84d5b9b-5n9vf" Oct 31 04:39:43.919733 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:43.919705 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nj9t\" (UniqueName: \"kubernetes.io/projected/994761dc-71d9-4334-a0aa-891d7287a690-kube-api-access-4nj9t\") pod \"maas-api-e2e-shared-b-w0-25f56e-85b84d5b9b-5n9vf\" (UID: \"994761dc-71d9-4334-a0aa-891d7287a690\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-25f56e-85b84d5b9b-5n9vf" Oct 31 04:39:44.402303 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:44.402269 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/994761dc-71d9-4334-a0aa-891d7287a690-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-25f56e-85b84d5b9b-5n9vf\" (UID: \"994761dc-71d9-4334-a0aa-891d7287a690\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-25f56e-85b84d5b9b-5n9vf" Oct 31 04:39:44.404793 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:44.404750 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/994761dc-71d9-4334-a0aa-891d7287a690-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-25f56e-85b84d5b9b-5n9vf\" (UID: \"994761dc-71d9-4334-a0aa-891d7287a690\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-25f56e-85b84d5b9b-5n9vf" Oct 31 04:39:44.697763 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:44.697357 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-25f56e-85b84d5b9b-5n9vf" Oct 31 04:39:45.039460 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:45.039429 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-25f56e-85b84d5b9b-5n9vf"] Oct 31 04:39:46.091617 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:46.091578 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-25f56e-85b84d5b9b-5n9vf" event={"ID":"994761dc-71d9-4334-a0aa-891d7287a690","Type":"ContainerStarted","Data":"e7ee00005b072dc8127f49f2ded0369bbd6f7b7a2cbda557c8c42a479f262c4b"} Oct 31 04:39:46.092058 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:46.091666 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-25f56e-85b84d5b9b-5n9vf" event={"ID":"994761dc-71d9-4334-a0aa-891d7287a690","Type":"ContainerStarted","Data":"88e14b49aeff2cd3990335ce3a3cddf12d20ee16bfd3d0316e63f6492f41923d"} Oct 31 04:39:46.092058 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:46.091694 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-25f56e-85b84d5b9b-5n9vf" Oct 31 04:39:46.137261 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:46.137211 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-25f56e-85b84d5b9b-5n9vf" podStartSLOduration=2.847588504 podStartE2EDuration="3.137197544s" totalImagesPullingTime="289.60904ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:39:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:39:46.134082551 +0000 UTC m=+2133.848417187" watchObservedRunningTime="2026-10-31 04:39:46.137197544 +0000 UTC m=+2133.851532157" Oct 31 04:39:47.994969 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:47.994941 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-401761-c5bb964f-h6p9q" Oct 31 04:39:52.103733 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:39:52.103693 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-25f56e-85b84d5b9b-5n9vf" Oct 31 04:40:03.261991 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:40:03.261912 4376 generic.go:356] "Generic (PLEG): container finished" podID="80cdb0ed-7002-415c-9916-7e1650f0481f" containerID="f7752764b4fabf3ad4cc5192c8e0217f044eed7de2108ea7b914eecd9885e170" exitCode=0 Oct 31 04:40:03.261991 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:40:03.261970 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-afc519-openshift-default-dfdbf9b97-qvdkl" event={"ID":"80cdb0ed-7002-415c-9916-7e1650f0481f","Type":"ContainerDied","Data":"f7752764b4fabf3ad4cc5192c8e0217f044eed7de2108ea7b914eecd9885e170"} Oct 31 04:40:03.262587 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:40:03.262489 4376 scope.go:122] "RemoveContainer" containerID="f7752764b4fabf3ad4cc5192c8e0217f044eed7de2108ea7b914eecd9885e170" Oct 31 04:40:03.267817 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:40:03.264608 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:40:03.267817 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:40:03.264705 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:40:03.267817 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:40:03.264752 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:40:04.278723 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:40:04.278681 4376 generic.go:356] "Generic (PLEG): container finished" podID="32954f51-d716-449a-ba69-ca46851a9ce2" containerID="1e419808a87c9def4ea0621cb542832602a03d85745a696cd6428c97d9fa8b62" exitCode=0 Oct 31 04:40:04.279488 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:40:04.279452 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-654741-openshift-default-8449896d9c-cf4pt" event={"ID":"32954f51-d716-449a-ba69-ca46851a9ce2","Type":"ContainerDied","Data":"1e419808a87c9def4ea0621cb542832602a03d85745a696cd6428c97d9fa8b62"} Oct 31 04:40:04.282651 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:40:04.282625 4376 scope.go:122] "RemoveContainer" containerID="1e419808a87c9def4ea0621cb542832602a03d85745a696cd6428c97d9fa8b62" Oct 31 04:40:04.284273 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:40:04.283639 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-afc519-openshift-default-dfdbf9b97-qvdkl" event={"ID":"80cdb0ed-7002-415c-9916-7e1650f0481f","Type":"ContainerStarted","Data":"18a8b0020f24aa5f77fd5ee2049d6452e048d21989c1bb5f75fa399f6b8afcd9"} Oct 31 04:40:04.285755 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:40:04.285721 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:40:04.285863 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:40:04.285842 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:40:04.285928 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:40:04.285908 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:40:05.253778 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:40:05.253741 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-afc519-openshift-default-dfdbf9b97-qvdkl" Oct 31 04:40:05.254143 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:40:05.254120 4376 patch_prober.go:29] interesting pod/e2e-extmod-a-afc519-openshift-default-dfdbf9b97-qvdkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body= Oct 31 04:40:05.254317 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:40:05.254291 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-afc519-openshift-default-dfdbf9b97-qvdkl" podUID="80cdb0ed-7002-415c-9916-7e1650f0481f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" Oct 31 04:40:05.295603 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:40:05.295574 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-654741-openshift-default-8449896d9c-cf4pt" event={"ID":"32954f51-d716-449a-ba69-ca46851a9ce2","Type":"ContainerStarted","Data":"1d8a428f018fe1e4348d1ece7c53b66f271eeee3ff675cdcfa0acf34491c573f"} Oct 31 04:40:06.178353 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:40:06.176873 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-654741-openshift-default-8449896d9c-cf4pt" Oct 31 04:40:06.178353 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:40:06.177009 4376 patch_prober.go:29] interesting pod/e2e-extmod-a-654741-openshift-default-8449896d9c-cf4pt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body= Oct 31 04:40:06.178353 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:40:06.177058 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-654741-openshift-default-8449896d9c-cf4pt" podUID="32954f51-d716-449a-ba69-ca46851a9ce2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" Oct 31 04:40:06.260815 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:40:06.260760 4376 patch_prober.go:29] interesting pod/e2e-extmod-a-afc519-openshift-default-dfdbf9b97-qvdkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body= Oct 31 04:40:06.261110 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:40:06.261078 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-afc519-openshift-default-dfdbf9b97-qvdkl" podUID="80cdb0ed-7002-415c-9916-7e1650f0481f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" Oct 31 04:40:07.175781 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:40:07.175741 4376 patch_prober.go:29] interesting pod/e2e-extmod-a-654741-openshift-default-8449896d9c-cf4pt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body= Oct 31 04:40:07.176434 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:40:07.176401 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-654741-openshift-default-8449896d9c-cf4pt" podUID="32954f51-d716-449a-ba69-ca46851a9ce2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" Oct 31 04:40:07.250350 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:40:07.249725 4376 patch_prober.go:29] interesting pod/e2e-extmod-a-afc519-openshift-default-dfdbf9b97-qvdkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body= Oct 31 04:40:07.250350 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:40:07.249785 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-afc519-openshift-default-dfdbf9b97-qvdkl" podUID="80cdb0ed-7002-415c-9916-7e1650f0481f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" Oct 31 04:40:08.175835 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:40:08.175791 4376 patch_prober.go:29] interesting pod/e2e-extmod-a-654741-openshift-default-8449896d9c-cf4pt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body= Oct 31 04:40:08.176489 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:40:08.176454 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-654741-openshift-default-8449896d9c-cf4pt" podUID="32954f51-d716-449a-ba69-ca46851a9ce2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" Oct 31 04:40:08.254353 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:40:08.250771 4376 patch_prober.go:29] interesting pod/e2e-extmod-a-afc519-openshift-default-dfdbf9b97-qvdkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body= Oct 31 04:40:08.254353 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:40:08.250835 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-afc519-openshift-default-dfdbf9b97-qvdkl" podUID="80cdb0ed-7002-415c-9916-7e1650f0481f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" Oct 31 04:40:09.179354 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:40:09.178876 4376 patch_prober.go:29] interesting pod/e2e-extmod-a-654741-openshift-default-8449896d9c-cf4pt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body= Oct 31 04:40:09.179354 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:40:09.178957 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-654741-openshift-default-8449896d9c-cf4pt" podUID="32954f51-d716-449a-ba69-ca46851a9ce2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" Oct 31 04:40:09.255351 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:40:09.251727 4376 patch_prober.go:29] interesting pod/e2e-extmod-a-afc519-openshift-default-dfdbf9b97-qvdkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body= Oct 31 04:40:09.255351 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:40:09.251794 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-afc519-openshift-default-dfdbf9b97-qvdkl" podUID="80cdb0ed-7002-415c-9916-7e1650f0481f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" Oct 31 04:40:10.179615 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:40:10.177673 4376 patch_prober.go:29] interesting pod/e2e-extmod-a-654741-openshift-default-8449896d9c-cf4pt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body= Oct 31 04:40:10.179615 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:40:10.177734 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-654741-openshift-default-8449896d9c-cf4pt" podUID="32954f51-d716-449a-ba69-ca46851a9ce2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" Oct 31 04:40:10.311880 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:40:10.311846 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-afc519-openshift-default-dfdbf9b97-qvdkl" Oct 31 04:40:10.312675 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:40:10.312647 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-afc519-openshift-default-dfdbf9b97-qvdkl" Oct 31 04:40:10.314080 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:40:10.314062 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-afc519-openshift-default-dfdbf9b97-qvdkl" Oct 31 04:40:11.004936 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:40:11.004897 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 04:40:11.187127 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:40:11.185400 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-654741-openshift-default-8449896d9c-cf4pt" Oct 31 04:40:11.187127 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:40:11.186974 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-654741-openshift-default-8449896d9c-cf4pt" Oct 31 04:40:11.187701 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:40:11.187552 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-654741-openshift-default-8449896d9c-cf4pt" Oct 31 04:40:22.506365 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:40:22.493843 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 04:40:32.574914 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:40:32.574880 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-567ca6-86cc9d6977-pk7jd"] Oct 31 04:40:32.579221 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:40:32.579192 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-567ca6-86cc9d6977-pk7jd" Oct 31 04:40:32.580801 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:40:32.579097 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-567ca6-86cc9d6977-pk7jd"] Oct 31 04:40:32.612991 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:40:32.607397 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-567ca6-dockercfg-7vsl7\"" Oct 31 04:40:32.612991 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:40:32.607480 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-567ca6\"" Oct 31 04:40:32.693285 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:40:32.693247 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/681f54ec-b648-49cb-ae16-dc814e56bb71-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-567ca6-86cc9d6977-pk7jd\" (UID: \"681f54ec-b648-49cb-ae16-dc814e56bb71\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-567ca6-86cc9d6977-pk7jd" Oct 31 04:40:32.693482 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:40:32.693437 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjnn9\" (UniqueName: \"kubernetes.io/projected/681f54ec-b648-49cb-ae16-dc814e56bb71-kube-api-access-mjnn9\") pod \"payload-processing-e2e-extmod-b-567ca6-86cc9d6977-pk7jd\" (UID: \"681f54ec-b648-49cb-ae16-dc814e56bb71\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-567ca6-86cc9d6977-pk7jd" Oct 31 04:40:32.795472 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:40:32.795432 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjnn9\" (UniqueName: \"kubernetes.io/projected/681f54ec-b648-49cb-ae16-dc814e56bb71-kube-api-access-mjnn9\") pod \"payload-processing-e2e-extmod-b-567ca6-86cc9d6977-pk7jd\" (UID: \"681f54ec-b648-49cb-ae16-dc814e56bb71\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-567ca6-86cc9d6977-pk7jd" Oct 31 04:40:32.795718 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:40:32.795697 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/681f54ec-b648-49cb-ae16-dc814e56bb71-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-567ca6-86cc9d6977-pk7jd\" (UID: \"681f54ec-b648-49cb-ae16-dc814e56bb71\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-567ca6-86cc9d6977-pk7jd" Oct 31 04:40:32.796626 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:40:32.796604 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/681f54ec-b648-49cb-ae16-dc814e56bb71-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-567ca6-86cc9d6977-pk7jd\" (UID: \"681f54ec-b648-49cb-ae16-dc814e56bb71\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-567ca6-86cc9d6977-pk7jd" Oct 31 04:40:32.819449 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:40:32.819407 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjnn9\" (UniqueName: \"kubernetes.io/projected/681f54ec-b648-49cb-ae16-dc814e56bb71-kube-api-access-mjnn9\") pod \"payload-processing-e2e-extmod-b-567ca6-86cc9d6977-pk7jd\" (UID: \"681f54ec-b648-49cb-ae16-dc814e56bb71\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-567ca6-86cc9d6977-pk7jd" Oct 31 04:40:32.899975 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:40:32.899939 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-567ca6-86cc9d6977-pk7jd" Oct 31 04:40:33.207633 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:40:33.207602 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod681f54ec_b648_49cb_ae16_dc814e56bb71.slice/crio-66d9fbfbfa6ada854135eda468e9c36bf9a8560968d272b2cfe27710df2ad7a9 WatchSource:0}: Error finding container 66d9fbfbfa6ada854135eda468e9c36bf9a8560968d272b2cfe27710df2ad7a9: Status 404 returned error can't find the container with id 66d9fbfbfa6ada854135eda468e9c36bf9a8560968d272b2cfe27710df2ad7a9 Oct 31 04:40:33.210707 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:40:33.210679 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-567ca6-86cc9d6977-pk7jd"] Oct 31 04:40:33.623242 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:40:33.623196 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-567ca6-86cc9d6977-pk7jd" event={"ID":"681f54ec-b648-49cb-ae16-dc814e56bb71","Type":"ContainerStarted","Data":"b77677df9903f6cb36d75a5ec150f5127979dd15bf231ea68a0f64de6ff9d994"} Oct 31 04:40:33.623787 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:40:33.623317 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-567ca6-86cc9d6977-pk7jd" event={"ID":"681f54ec-b648-49cb-ae16-dc814e56bb71","Type":"ContainerStarted","Data":"66d9fbfbfa6ada854135eda468e9c36bf9a8560968d272b2cfe27710df2ad7a9"} Oct 31 04:40:33.623787 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:40:33.623490 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-567ca6-86cc9d6977-pk7jd" Oct 31 04:40:33.669122 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:40:33.669062 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-567ca6-86cc9d6977-pk7jd" podStartSLOduration=1.6690464600000001 podStartE2EDuration="1.66904646s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:40:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:40:33.667863931 +0000 UTC m=+2181.382198552" watchObservedRunningTime="2026-10-31 04:40:33.66904646 +0000 UTC m=+2181.383381076" Oct 31 04:40:35.529909 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:40:35.529876 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-8973e2-849bd8687-5kxc2"] Oct 31 04:40:35.564352 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:40:35.560616 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-8973e2-849bd8687-5kxc2" Oct 31 04:40:35.623378 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:40:35.611840 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-8973e2-849bd8687-5kxc2"] Oct 31 04:40:35.633357 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:40:35.631429 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-8973e2-dockercfg-j2r8t\"" Oct 31 04:40:35.633357 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:40:35.631499 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-8973e2\"" Oct 31 04:40:35.656233 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:40:35.656192 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a9e4834-09d5-4add-9004-04c30da0527c-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-8973e2-849bd8687-5kxc2\" (UID: \"2a9e4834-09d5-4add-9004-04c30da0527c\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-8973e2-849bd8687-5kxc2" Oct 31 04:40:35.656548 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:40:35.656524 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9qc5\" (UniqueName: \"kubernetes.io/projected/2a9e4834-09d5-4add-9004-04c30da0527c-kube-api-access-m9qc5\") pod \"payload-processing-e2e-extmod-a-8973e2-849bd8687-5kxc2\" (UID: \"2a9e4834-09d5-4add-9004-04c30da0527c\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-8973e2-849bd8687-5kxc2" Oct 31 04:40:35.696170 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:40:35.695939 4376 generic.go:356] "Generic (PLEG): container finished" podID="caf8772a-4894-4924-9f18-f2b191aadad7" containerID="440604183b7edcc8cd815d9149aa46b04f0ad06c051c8897a66a7ef0bd0d4761" exitCode=0 Oct 31 04:40:35.696170 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:40:35.696047 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-25f56e-openshift-default-6465df847-6j725" event={"ID":"caf8772a-4894-4924-9f18-f2b191aadad7","Type":"ContainerDied","Data":"440604183b7edcc8cd815d9149aa46b04f0ad06c051c8897a66a7ef0bd0d4761"} Oct 31 04:40:35.697596 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:40:35.697521 4376 scope.go:122] "RemoveContainer" containerID="440604183b7edcc8cd815d9149aa46b04f0ad06c051c8897a66a7ef0bd0d4761" Oct 31 04:40:35.702951 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:40:35.702912 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:40:35.703072 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:40:35.703018 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:40:35.703136 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:40:35.703071 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:40:35.758136 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:40:35.758102 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a9e4834-09d5-4add-9004-04c30da0527c-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-8973e2-849bd8687-5kxc2\" (UID: \"2a9e4834-09d5-4add-9004-04c30da0527c\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-8973e2-849bd8687-5kxc2" Oct 31 04:40:35.758439 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:40:35.758172 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9qc5\" (UniqueName: \"kubernetes.io/projected/2a9e4834-09d5-4add-9004-04c30da0527c-kube-api-access-m9qc5\") pod \"payload-processing-e2e-extmod-a-8973e2-849bd8687-5kxc2\" (UID: \"2a9e4834-09d5-4add-9004-04c30da0527c\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-8973e2-849bd8687-5kxc2" Oct 31 04:40:35.759036 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:40:35.759009 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a9e4834-09d5-4add-9004-04c30da0527c-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-8973e2-849bd8687-5kxc2\" (UID: \"2a9e4834-09d5-4add-9004-04c30da0527c\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-8973e2-849bd8687-5kxc2" Oct 31 04:40:35.783157 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:40:35.783073 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9qc5\" (UniqueName: \"kubernetes.io/projected/2a9e4834-09d5-4add-9004-04c30da0527c-kube-api-access-m9qc5\") pod \"payload-processing-e2e-extmod-a-8973e2-849bd8687-5kxc2\" (UID: \"2a9e4834-09d5-4add-9004-04c30da0527c\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-8973e2-849bd8687-5kxc2" Oct 31 04:40:36.009940 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:40:36.009907 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-8973e2-849bd8687-5kxc2" Oct 31 04:40:36.316143 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:40:36.316109 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a9e4834_09d5_4add_9004_04c30da0527c.slice/crio-89a0bc099a5458770d546a0e675e74b99eefb31366ff3741050fe27581170daa WatchSource:0}: Error finding container 89a0bc099a5458770d546a0e675e74b99eefb31366ff3741050fe27581170daa: Status 404 returned error can't find the container with id 89a0bc099a5458770d546a0e675e74b99eefb31366ff3741050fe27581170daa Oct 31 04:40:36.318869 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:40:36.318840 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-8973e2-849bd8687-5kxc2"] Oct 31 04:40:36.705848 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:40:36.705803 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-25f56e-openshift-default-6465df847-6j725" event={"ID":"caf8772a-4894-4924-9f18-f2b191aadad7","Type":"ContainerStarted","Data":"f404f437b5bd699a47c12a58483993d7b7b47ad54b19ee53bb2b7863526c98f2"} Oct 31 04:40:36.707820 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:40:36.707769 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-8973e2-849bd8687-5kxc2" event={"ID":"2a9e4834-09d5-4add-9004-04c30da0527c","Type":"ContainerStarted","Data":"cf0df277d55a85c5e084bb3a35e3afd985254f6fe7bdb826bd7737a7e3a50f86"} Oct 31 04:40:36.707940 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:40:36.707830 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-8973e2-849bd8687-5kxc2" event={"ID":"2a9e4834-09d5-4add-9004-04c30da0527c","Type":"ContainerStarted","Data":"89a0bc099a5458770d546a0e675e74b99eefb31366ff3741050fe27581170daa"} Oct 31 04:40:36.707940 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:40:36.707854 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-8973e2-849bd8687-5kxc2" Oct 31 04:40:36.801011 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:40:36.800951 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-8973e2-849bd8687-5kxc2" podStartSLOduration=1.800935352 podStartE2EDuration="1.800935352s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:40:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:40:36.797828638 +0000 UTC m=+2184.512163292" watchObservedRunningTime="2026-10-31 04:40:36.800935352 +0000 UTC m=+2184.515269968" Oct 31 04:40:37.330878 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:40:37.330560 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-25f56e-openshift-default-6465df847-6j725" Oct 31 04:40:37.332495 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:40:37.332461 4376 patch_prober.go:29] interesting pod/e2e-shared-b-w0-25f56e-openshift-default-6465df847-6j725 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body= Oct 31 04:40:37.332660 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:40:37.332537 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-25f56e-openshift-default-6465df847-6j725" podUID="caf8772a-4894-4924-9f18-f2b191aadad7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" Oct 31 04:40:38.330706 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:40:38.330664 4376 patch_prober.go:29] interesting pod/e2e-shared-b-w0-25f56e-openshift-default-6465df847-6j725 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body= Oct 31 04:40:38.331115 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:40:38.330741 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-25f56e-openshift-default-6465df847-6j725" podUID="caf8772a-4894-4924-9f18-f2b191aadad7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" Oct 31 04:40:38.608629 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:40:38.608530 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-25f56e-openshift-default-6465df847-6j725" Oct 31 04:40:39.331186 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:40:39.331143 4376 patch_prober.go:29] interesting pod/e2e-shared-b-w0-25f56e-openshift-default-6465df847-6j725 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body= Oct 31 04:40:39.331714 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:40:39.331199 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-25f56e-openshift-default-6465df847-6j725" podUID="caf8772a-4894-4924-9f18-f2b191aadad7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" Oct 31 04:40:40.330674 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:40:40.330633 4376 patch_prober.go:29] interesting pod/e2e-shared-b-w0-25f56e-openshift-default-6465df847-6j725 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body= Oct 31 04:40:40.330886 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:40:40.330686 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-25f56e-openshift-default-6465df847-6j725" podUID="caf8772a-4894-4924-9f18-f2b191aadad7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" Oct 31 04:40:41.015206 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:40:41.015175 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 04:40:41.331003 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:40:41.330916 4376 patch_prober.go:29] interesting pod/e2e-shared-b-w0-25f56e-openshift-default-6465df847-6j725 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body= Oct 31 04:40:41.331003 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:40:41.330974 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-25f56e-openshift-default-6465df847-6j725" podUID="caf8772a-4894-4924-9f18-f2b191aadad7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" Oct 31 04:40:42.376075 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:40:42.376040 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-25f56e-openshift-default-6465df847-6j725" Oct 31 04:40:42.379595 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:40:42.379561 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-25f56e-openshift-default-6465df847-6j725" Oct 31 04:40:44.633448 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:40:44.633418 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-567ca6-86cc9d6977-pk7jd" Oct 31 04:40:47.718968 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:40:47.718939 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-8973e2-849bd8687-5kxc2" Oct 31 04:40:55.688368 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:40:55.688285 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-a99f2a-6df854df67-qh6wn"] Oct 31 04:40:55.691861 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:40:55.691844 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a99f2a-6df854df67-qh6wn" Oct 31 04:40:55.714521 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:40:55.714498 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-a99f2a-6df854df67-qh6wn"] Oct 31 04:40:55.721303 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:40:55.721282 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-a99f2a-serving-cert\"" Oct 31 04:40:55.819844 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:40:55.819810 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3e4def5-df8d-42d2-810c-8fa6764d338b-maas-api-tls\") pod \"maas-api-e2e-extmod-b-a99f2a-6df854df67-qh6wn\" (UID: \"b3e4def5-df8d-42d2-810c-8fa6764d338b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a99f2a-6df854df67-qh6wn" Oct 31 04:40:55.820006 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:40:55.819870 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3e4def5-df8d-42d2-810c-8fa6764d338b-metrics-certs\") pod \"maas-api-e2e-extmod-b-a99f2a-6df854df67-qh6wn\" (UID: \"b3e4def5-df8d-42d2-810c-8fa6764d338b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a99f2a-6df854df67-qh6wn" Oct 31 04:40:55.820053 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:40:55.820041 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gjrh\" (UniqueName: \"kubernetes.io/projected/b3e4def5-df8d-42d2-810c-8fa6764d338b-kube-api-access-9gjrh\") pod \"maas-api-e2e-extmod-b-a99f2a-6df854df67-qh6wn\" (UID: \"b3e4def5-df8d-42d2-810c-8fa6764d338b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a99f2a-6df854df67-qh6wn" Oct 31 04:40:55.820095 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:40:55.820075 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3e4def5-df8d-42d2-810c-8fa6764d338b-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-a99f2a-6df854df67-qh6wn\" (UID: \"b3e4def5-df8d-42d2-810c-8fa6764d338b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a99f2a-6df854df67-qh6wn" Oct 31 04:40:55.921236 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:40:55.921208 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gjrh\" (UniqueName: \"kubernetes.io/projected/b3e4def5-df8d-42d2-810c-8fa6764d338b-kube-api-access-9gjrh\") pod \"maas-api-e2e-extmod-b-a99f2a-6df854df67-qh6wn\" (UID: \"b3e4def5-df8d-42d2-810c-8fa6764d338b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a99f2a-6df854df67-qh6wn" Oct 31 04:40:55.921391 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:40:55.921249 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3e4def5-df8d-42d2-810c-8fa6764d338b-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-a99f2a-6df854df67-qh6wn\" (UID: \"b3e4def5-df8d-42d2-810c-8fa6764d338b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a99f2a-6df854df67-qh6wn" Oct 31 04:40:55.921442 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:40:55.921397 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3e4def5-df8d-42d2-810c-8fa6764d338b-maas-api-tls\") pod \"maas-api-e2e-extmod-b-a99f2a-6df854df67-qh6wn\" (UID: \"b3e4def5-df8d-42d2-810c-8fa6764d338b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a99f2a-6df854df67-qh6wn" Oct 31 04:40:55.921442 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:40:55.921428 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3e4def5-df8d-42d2-810c-8fa6764d338b-metrics-certs\") pod \"maas-api-e2e-extmod-b-a99f2a-6df854df67-qh6wn\" (UID: \"b3e4def5-df8d-42d2-810c-8fa6764d338b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a99f2a-6df854df67-qh6wn" Oct 31 04:40:55.921990 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:40:55.921963 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3e4def5-df8d-42d2-810c-8fa6764d338b-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-a99f2a-6df854df67-qh6wn\" (UID: \"b3e4def5-df8d-42d2-810c-8fa6764d338b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a99f2a-6df854df67-qh6wn" Oct 31 04:40:55.923835 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:40:55.923818 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3e4def5-df8d-42d2-810c-8fa6764d338b-maas-api-tls\") pod \"maas-api-e2e-extmod-b-a99f2a-6df854df67-qh6wn\" (UID: \"b3e4def5-df8d-42d2-810c-8fa6764d338b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a99f2a-6df854df67-qh6wn" Oct 31 04:40:55.923926 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:40:55.923872 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3e4def5-df8d-42d2-810c-8fa6764d338b-metrics-certs\") pod \"maas-api-e2e-extmod-b-a99f2a-6df854df67-qh6wn\" (UID: \"b3e4def5-df8d-42d2-810c-8fa6764d338b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a99f2a-6df854df67-qh6wn" Oct 31 04:40:55.945415 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:40:55.945355 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gjrh\" (UniqueName: \"kubernetes.io/projected/b3e4def5-df8d-42d2-810c-8fa6764d338b-kube-api-access-9gjrh\") pod \"maas-api-e2e-extmod-b-a99f2a-6df854df67-qh6wn\" (UID: \"b3e4def5-df8d-42d2-810c-8fa6764d338b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a99f2a-6df854df67-qh6wn" Oct 31 04:40:56.008693 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:40:56.008655 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a99f2a-6df854df67-qh6wn" Oct 31 04:40:56.322191 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:40:56.322158 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3e4def5_df8d_42d2_810c_8fa6764d338b.slice/crio-44c9cf30b79ebe8205d5a17fdf58ad2c4e8a3dd40633e46f7ffd52009b86114d WatchSource:0}: Error finding container 44c9cf30b79ebe8205d5a17fdf58ad2c4e8a3dd40633e46f7ffd52009b86114d: Status 404 returned error can't find the container with id 44c9cf30b79ebe8205d5a17fdf58ad2c4e8a3dd40633e46f7ffd52009b86114d Oct 31 04:40:56.323528 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:40:56.323503 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-a99f2a-6df854df67-qh6wn"] Oct 31 04:40:56.922235 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:40:56.922130 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a99f2a-6df854df67-qh6wn" event={"ID":"b3e4def5-df8d-42d2-810c-8fa6764d338b","Type":"ContainerStarted","Data":"ea82adb5b66aaaa5e43624dfb39d35e0b275a57c8eb9b3661aff0a890ee75b35"} Oct 31 04:40:56.922235 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:40:56.922206 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a99f2a-6df854df67-qh6wn" event={"ID":"b3e4def5-df8d-42d2-810c-8fa6764d338b","Type":"ContainerStarted","Data":"44c9cf30b79ebe8205d5a17fdf58ad2c4e8a3dd40633e46f7ffd52009b86114d"} Oct 31 04:40:56.939406 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:40:56.923697 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a99f2a-6df854df67-qh6wn" Oct 31 04:40:56.973664 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:40:56.973613 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a99f2a-6df854df67-qh6wn" podStartSLOduration=1.653212351 podStartE2EDuration="1.973595506s" totalImagesPullingTime="320.383155ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:40:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:40:56.97039969 +0000 UTC m=+2204.684734331" watchObservedRunningTime="2026-10-31 04:40:56.973595506 +0000 UTC m=+2204.687930155" Oct 31 04:41:03.941208 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:41:03.941179 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a99f2a-6df854df67-qh6wn" Oct 31 04:41:11.002486 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:41:11.002446 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 04:41:11.984731 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:41:11.984696 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-567ca6-59b4b45645-98ghw"] Oct 31 04:41:11.988448 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:41:11.988427 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-567ca6-59b4b45645-98ghw" Oct 31 04:41:12.024537 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:41:12.024501 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-567ca6-serving-cert\"" Oct 31 04:41:12.025080 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:41:12.024656 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-567ca6-59b4b45645-98ghw"] Oct 31 04:41:12.146956 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:41:12.146924 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/649b78d1-3eb4-4780-8462-f6c6a5cf7c6c-maas-api-tls\") pod \"maas-api-e2e-extmod-b-567ca6-59b4b45645-98ghw\" (UID: \"649b78d1-3eb4-4780-8462-f6c6a5cf7c6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-567ca6-59b4b45645-98ghw" Oct 31 04:41:12.147409 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:41:12.146976 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/649b78d1-3eb4-4780-8462-f6c6a5cf7c6c-metrics-certs\") pod \"maas-api-e2e-extmod-b-567ca6-59b4b45645-98ghw\" (UID: \"649b78d1-3eb4-4780-8462-f6c6a5cf7c6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-567ca6-59b4b45645-98ghw" Oct 31 04:41:12.147409 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:41:12.147105 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/649b78d1-3eb4-4780-8462-f6c6a5cf7c6c-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-567ca6-59b4b45645-98ghw\" (UID: \"649b78d1-3eb4-4780-8462-f6c6a5cf7c6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-567ca6-59b4b45645-98ghw" Oct 31 04:41:12.147409 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:41:12.147233 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxsml\" (UniqueName: \"kubernetes.io/projected/649b78d1-3eb4-4780-8462-f6c6a5cf7c6c-kube-api-access-hxsml\") pod \"maas-api-e2e-extmod-b-567ca6-59b4b45645-98ghw\" (UID: \"649b78d1-3eb4-4780-8462-f6c6a5cf7c6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-567ca6-59b4b45645-98ghw" Oct 31 04:41:12.249218 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:41:12.249120 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/649b78d1-3eb4-4780-8462-f6c6a5cf7c6c-maas-api-tls\") pod \"maas-api-e2e-extmod-b-567ca6-59b4b45645-98ghw\" (UID: \"649b78d1-3eb4-4780-8462-f6c6a5cf7c6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-567ca6-59b4b45645-98ghw" Oct 31 04:41:12.249218 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:41:12.249193 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/649b78d1-3eb4-4780-8462-f6c6a5cf7c6c-metrics-certs\") pod \"maas-api-e2e-extmod-b-567ca6-59b4b45645-98ghw\" (UID: \"649b78d1-3eb4-4780-8462-f6c6a5cf7c6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-567ca6-59b4b45645-98ghw" Oct 31 04:41:12.249536 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:41:12.249263 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/649b78d1-3eb4-4780-8462-f6c6a5cf7c6c-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-567ca6-59b4b45645-98ghw\" (UID: \"649b78d1-3eb4-4780-8462-f6c6a5cf7c6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-567ca6-59b4b45645-98ghw" Oct 31 04:41:12.249536 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:41:12.249447 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxsml\" (UniqueName: \"kubernetes.io/projected/649b78d1-3eb4-4780-8462-f6c6a5cf7c6c-kube-api-access-hxsml\") pod \"maas-api-e2e-extmod-b-567ca6-59b4b45645-98ghw\" (UID: \"649b78d1-3eb4-4780-8462-f6c6a5cf7c6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-567ca6-59b4b45645-98ghw" Oct 31 04:41:12.250004 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:41:12.249975 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/649b78d1-3eb4-4780-8462-f6c6a5cf7c6c-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-567ca6-59b4b45645-98ghw\" (UID: \"649b78d1-3eb4-4780-8462-f6c6a5cf7c6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-567ca6-59b4b45645-98ghw" Oct 31 04:41:12.252070 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:41:12.252042 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/649b78d1-3eb4-4780-8462-f6c6a5cf7c6c-maas-api-tls\") pod \"maas-api-e2e-extmod-b-567ca6-59b4b45645-98ghw\" (UID: \"649b78d1-3eb4-4780-8462-f6c6a5cf7c6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-567ca6-59b4b45645-98ghw" Oct 31 04:41:12.252216 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:41:12.252125 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/649b78d1-3eb4-4780-8462-f6c6a5cf7c6c-metrics-certs\") pod \"maas-api-e2e-extmod-b-567ca6-59b4b45645-98ghw\" (UID: \"649b78d1-3eb4-4780-8462-f6c6a5cf7c6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-567ca6-59b4b45645-98ghw" Oct 31 04:41:12.273770 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:41:12.273736 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxsml\" (UniqueName: \"kubernetes.io/projected/649b78d1-3eb4-4780-8462-f6c6a5cf7c6c-kube-api-access-hxsml\") pod \"maas-api-e2e-extmod-b-567ca6-59b4b45645-98ghw\" (UID: \"649b78d1-3eb4-4780-8462-f6c6a5cf7c6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-567ca6-59b4b45645-98ghw" Oct 31 04:41:12.303599 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:41:12.303568 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-567ca6-59b4b45645-98ghw" Oct 31 04:41:12.610148 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:41:12.610114 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod649b78d1_3eb4_4780_8462_f6c6a5cf7c6c.slice/crio-7342d9b2b8ebf43fec2d23c49f0aece37e86bcbfbc610b1ee148b805efda80a0 WatchSource:0}: Error finding container 7342d9b2b8ebf43fec2d23c49f0aece37e86bcbfbc610b1ee148b805efda80a0: Status 404 returned error can't find the container with id 7342d9b2b8ebf43fec2d23c49f0aece37e86bcbfbc610b1ee148b805efda80a0 Oct 31 04:41:12.610708 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:41:12.610686 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-567ca6-59b4b45645-98ghw"] Oct 31 04:41:13.043758 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:41:13.043721 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-567ca6-59b4b45645-98ghw" event={"ID":"649b78d1-3eb4-4780-8462-f6c6a5cf7c6c","Type":"ContainerStarted","Data":"cbd13093204d28e5a920b81f077ede0be2004eb54d991427492f1b562d91b89b"} Oct 31 04:41:13.044153 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:41:13.043779 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-567ca6-59b4b45645-98ghw" event={"ID":"649b78d1-3eb4-4780-8462-f6c6a5cf7c6c","Type":"ContainerStarted","Data":"7342d9b2b8ebf43fec2d23c49f0aece37e86bcbfbc610b1ee148b805efda80a0"} Oct 31 04:41:13.044153 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:41:13.043802 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-567ca6-59b4b45645-98ghw" Oct 31 04:41:13.089592 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:41:13.089541 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-567ca6-59b4b45645-98ghw" podStartSLOduration=1.792875803 podStartE2EDuration="2.089526812s" totalImagesPullingTime="296.651009ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:41:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:41:13.087062923 +0000 UTC m=+2220.801397544" watchObservedRunningTime="2026-10-31 04:41:13.089526812 +0000 UTC m=+2220.803861429" Oct 31 04:41:19.055428 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:41:19.055400 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-567ca6-59b4b45645-98ghw" Oct 31 04:41:30.416399 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:41:30.416362 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 04:41:41.000584 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:41:41.000534 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 04:42:09.364817 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:42:09.364783 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 04:42:11.003076 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:42:11.003044 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 04:42:13.769440 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:42:13.769404 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-227c93-74b5c495fc-v4rxp"] Oct 31 04:42:13.834604 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:42:13.833573 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-227c93-74b5c495fc-v4rxp"] Oct 31 04:42:13.834604 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:42:13.833714 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-227c93-74b5c495fc-v4rxp" Oct 31 04:42:13.849581 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:42:13.849555 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-227c93\"" Oct 31 04:42:13.864604 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:42:13.864577 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-227c93-dockercfg-9jfts\"" Oct 31 04:42:13.947031 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:42:13.946996 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e0104d4-9c91-4dc6-93ae-a7844c9d89c8-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-227c93-74b5c495fc-v4rxp\" (UID: \"8e0104d4-9c91-4dc6-93ae-a7844c9d89c8\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-227c93-74b5c495fc-v4rxp" Oct 31 04:42:13.947223 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:42:13.947166 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkts2\" (UniqueName: \"kubernetes.io/projected/8e0104d4-9c91-4dc6-93ae-a7844c9d89c8-kube-api-access-mkts2\") pod \"payload-processing-e2e-extmod-b-227c93-74b5c495fc-v4rxp\" (UID: \"8e0104d4-9c91-4dc6-93ae-a7844c9d89c8\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-227c93-74b5c495fc-v4rxp" Oct 31 04:42:14.048751 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:42:14.048677 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e0104d4-9c91-4dc6-93ae-a7844c9d89c8-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-227c93-74b5c495fc-v4rxp\" (UID: \"8e0104d4-9c91-4dc6-93ae-a7844c9d89c8\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-227c93-74b5c495fc-v4rxp" Oct 31 04:42:14.048937 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:42:14.048786 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkts2\" (UniqueName: \"kubernetes.io/projected/8e0104d4-9c91-4dc6-93ae-a7844c9d89c8-kube-api-access-mkts2\") pod \"payload-processing-e2e-extmod-b-227c93-74b5c495fc-v4rxp\" (UID: \"8e0104d4-9c91-4dc6-93ae-a7844c9d89c8\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-227c93-74b5c495fc-v4rxp" Oct 31 04:42:14.049420 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:42:14.049401 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e0104d4-9c91-4dc6-93ae-a7844c9d89c8-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-227c93-74b5c495fc-v4rxp\" (UID: \"8e0104d4-9c91-4dc6-93ae-a7844c9d89c8\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-227c93-74b5c495fc-v4rxp" Oct 31 04:42:14.072924 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:42:14.072893 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkts2\" (UniqueName: \"kubernetes.io/projected/8e0104d4-9c91-4dc6-93ae-a7844c9d89c8-kube-api-access-mkts2\") pod \"payload-processing-e2e-extmod-b-227c93-74b5c495fc-v4rxp\" (UID: \"8e0104d4-9c91-4dc6-93ae-a7844c9d89c8\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-227c93-74b5c495fc-v4rxp" Oct 31 04:42:14.162745 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:42:14.162714 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-227c93-74b5c495fc-v4rxp" Oct 31 04:42:14.462377 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:42:14.462322 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e0104d4_9c91_4dc6_93ae_a7844c9d89c8.slice/crio-56656f11975716a78a93973f340db8d8fbc5ac99b544b43c22ccd3c52b36a84a WatchSource:0}: Error finding container 56656f11975716a78a93973f340db8d8fbc5ac99b544b43c22ccd3c52b36a84a: Status 404 returned error can't find the container with id 56656f11975716a78a93973f340db8d8fbc5ac99b544b43c22ccd3c52b36a84a Oct 31 04:42:14.467541 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:42:14.464427 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-227c93-74b5c495fc-v4rxp"] Oct 31 04:42:14.646444 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:42:14.646408 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-227c93-74b5c495fc-v4rxp" event={"ID":"8e0104d4-9c91-4dc6-93ae-a7844c9d89c8","Type":"ContainerStarted","Data":"d5184cac0849763780d398b0d1e264c99217941e194b41ae3f9d4e6eb5fd9059"} Oct 31 04:42:14.646651 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:42:14.646464 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-227c93-74b5c495fc-v4rxp" event={"ID":"8e0104d4-9c91-4dc6-93ae-a7844c9d89c8","Type":"ContainerStarted","Data":"56656f11975716a78a93973f340db8d8fbc5ac99b544b43c22ccd3c52b36a84a"} Oct 31 04:42:14.646651 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:42:14.646543 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-227c93-74b5c495fc-v4rxp" Oct 31 04:42:14.692265 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:42:14.692206 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-227c93-74b5c495fc-v4rxp" podStartSLOduration=1.692187482 podStartE2EDuration="1.692187482s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:42:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:42:14.687371617 +0000 UTC m=+2282.401706233" watchObservedRunningTime="2026-10-31 04:42:14.692187482 +0000 UTC m=+2282.406522100" Oct 31 04:42:25.655804 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:42:25.655728 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-227c93-74b5c495fc-v4rxp" Oct 31 04:42:41.000864 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:42:41.000806 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 04:42:50.501325 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:42:50.501291 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 04:43:10.999822 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:43:10.999781 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 04:43:11.839946 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:43:11.839914 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-rs8fm"] Oct 31 04:43:11.840140 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:43:11.840120 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-rs8fm" podUID="360e59d6-8295-48fd-a8d2-2e7fe1e4f402" containerName="istio-proxy" containerID="cri-o://07af01032d3a5a87cd5d9e32bb49c9d0b5fc89561af67062aa6a19b20aa70153" gracePeriod=30 Oct 31 04:43:15.519872 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:43:15.519843 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-47c409-8649bf6df7-rbdch"] Oct 31 04:43:15.524519 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:43:15.524499 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-47c409-8649bf6df7-rbdch" Oct 31 04:43:15.550750 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:43:15.550725 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-47c409-8649bf6df7-rbdch"] Oct 31 04:43:15.577197 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:43:15.577168 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-47c409-dockercfg-vbmzr\"" Oct 31 04:43:15.577382 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:43:15.577239 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-47c409\"" Oct 31 04:43:15.578137 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:43:15.578110 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f48d27d4-dc90-4bf7-aa79-2b14c97859a2-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-47c409-8649bf6df7-rbdch\" (UID: \"f48d27d4-dc90-4bf7-aa79-2b14c97859a2\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-47c409-8649bf6df7-rbdch" Oct 31 04:43:15.578244 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:43:15.578199 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rb87b\" (UniqueName: \"kubernetes.io/projected/f48d27d4-dc90-4bf7-aa79-2b14c97859a2-kube-api-access-rb87b\") pod \"payload-processing-e2e-extmod-a-47c409-8649bf6df7-rbdch\" (UID: \"f48d27d4-dc90-4bf7-aa79-2b14c97859a2\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-47c409-8649bf6df7-rbdch" Oct 31 04:43:15.688912 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:43:15.681195 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f48d27d4-dc90-4bf7-aa79-2b14c97859a2-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-47c409-8649bf6df7-rbdch\" (UID: \"f48d27d4-dc90-4bf7-aa79-2b14c97859a2\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-47c409-8649bf6df7-rbdch" Oct 31 04:43:15.703537 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:43:15.703276 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f48d27d4-dc90-4bf7-aa79-2b14c97859a2-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-47c409-8649bf6df7-rbdch\" (UID: \"f48d27d4-dc90-4bf7-aa79-2b14c97859a2\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-47c409-8649bf6df7-rbdch" Oct 31 04:43:15.712351 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:43:15.703800 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rb87b\" (UniqueName: \"kubernetes.io/projected/f48d27d4-dc90-4bf7-aa79-2b14c97859a2-kube-api-access-rb87b\") pod \"payload-processing-e2e-extmod-a-47c409-8649bf6df7-rbdch\" (UID: \"f48d27d4-dc90-4bf7-aa79-2b14c97859a2\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-47c409-8649bf6df7-rbdch" Oct 31 04:43:15.789496 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:43:15.789208 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rb87b\" (UniqueName: \"kubernetes.io/projected/f48d27d4-dc90-4bf7-aa79-2b14c97859a2-kube-api-access-rb87b\") pod \"payload-processing-e2e-extmod-a-47c409-8649bf6df7-rbdch\" (UID: \"f48d27d4-dc90-4bf7-aa79-2b14c97859a2\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-47c409-8649bf6df7-rbdch" Oct 31 04:43:15.840007 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:43:15.839981 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-47c409-8649bf6df7-rbdch" Oct 31 04:43:16.142439 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:43:16.142407 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf48d27d4_dc90_4bf7_aa79_2b14c97859a2.slice/crio-e4c3c1a1e804a9e62972838a33072da41f6d8b840fd064abfb71caded92048fa WatchSource:0}: Error finding container e4c3c1a1e804a9e62972838a33072da41f6d8b840fd064abfb71caded92048fa: Status 404 returned error can't find the container with id e4c3c1a1e804a9e62972838a33072da41f6d8b840fd064abfb71caded92048fa Oct 31 04:43:16.145947 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:43:16.145915 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-47c409-8649bf6df7-rbdch"] Oct 31 04:43:17.108298 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:43:17.108233 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-rs8fm" Oct 31 04:43:17.156451 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:43:17.156423 4376 generic.go:356] "Generic (PLEG): container finished" podID="360e59d6-8295-48fd-a8d2-2e7fe1e4f402" containerID="07af01032d3a5a87cd5d9e32bb49c9d0b5fc89561af67062aa6a19b20aa70153" exitCode=0 Oct 31 04:43:17.156579 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:43:17.156467 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-rs8fm" Oct 31 04:43:17.156579 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:43:17.156506 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-rs8fm" event={"ID":"360e59d6-8295-48fd-a8d2-2e7fe1e4f402","Type":"ContainerDied","Data":"07af01032d3a5a87cd5d9e32bb49c9d0b5fc89561af67062aa6a19b20aa70153"} Oct 31 04:43:17.156579 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:43:17.156567 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-rs8fm" event={"ID":"360e59d6-8295-48fd-a8d2-2e7fe1e4f402","Type":"ContainerDied","Data":"b570658e1a489a3f0dd32cb35ab901342af22ac396b73e34a65483f4e8d32ff3"} Oct 31 04:43:17.156764 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:43:17.156597 4376 scope.go:122] "RemoveContainer" containerID="07af01032d3a5a87cd5d9e32bb49c9d0b5fc89561af67062aa6a19b20aa70153" Oct 31 04:43:17.158263 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:43:17.158240 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-47c409-8649bf6df7-rbdch" event={"ID":"f48d27d4-dc90-4bf7-aa79-2b14c97859a2","Type":"ContainerStarted","Data":"0c6c5b9c319ec0aeb984e2f930be79e000795979f7e74f1a42dafcc2a8c8a1f5"} Oct 31 04:43:17.158360 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:43:17.158278 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-47c409-8649bf6df7-rbdch" event={"ID":"f48d27d4-dc90-4bf7-aa79-2b14c97859a2","Type":"ContainerStarted","Data":"e4c3c1a1e804a9e62972838a33072da41f6d8b840fd064abfb71caded92048fa"} Oct 31 04:43:17.158414 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:43:17.158384 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-47c409-8649bf6df7-rbdch" Oct 31 04:43:17.166719 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:43:17.166702 4376 scope.go:122] "RemoveContainer" containerID="07af01032d3a5a87cd5d9e32bb49c9d0b5fc89561af67062aa6a19b20aa70153" Oct 31 04:43:17.167028 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:43:17.167009 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07af01032d3a5a87cd5d9e32bb49c9d0b5fc89561af67062aa6a19b20aa70153\": container with ID starting with 07af01032d3a5a87cd5d9e32bb49c9d0b5fc89561af67062aa6a19b20aa70153 not found: ID does not exist" containerID="07af01032d3a5a87cd5d9e32bb49c9d0b5fc89561af67062aa6a19b20aa70153" Oct 31 04:43:17.167093 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:43:17.167039 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07af01032d3a5a87cd5d9e32bb49c9d0b5fc89561af67062aa6a19b20aa70153"} err="failed to get container status \"07af01032d3a5a87cd5d9e32bb49c9d0b5fc89561af67062aa6a19b20aa70153\": rpc error: code = NotFound desc = could not find container \"07af01032d3a5a87cd5d9e32bb49c9d0b5fc89561af67062aa6a19b20aa70153\": container with ID starting with 07af01032d3a5a87cd5d9e32bb49c9d0b5fc89561af67062aa6a19b20aa70153 not found: ID does not exist" Oct 31 04:43:17.173139 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:43:17.173118 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/360e59d6-8295-48fd-a8d2-2e7fe1e4f402-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/360e59d6-8295-48fd-a8d2-2e7fe1e4f402-istio-envoy\") pod \"360e59d6-8295-48fd-a8d2-2e7fe1e4f402\" (UID: \"360e59d6-8295-48fd-a8d2-2e7fe1e4f402\") " Oct 31 04:43:17.173429 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:43:17.173403 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/360e59d6-8295-48fd-a8d2-2e7fe1e4f402-kube-api-access-sjxl5\" (UniqueName: \"kubernetes.io/projected/360e59d6-8295-48fd-a8d2-2e7fe1e4f402-kube-api-access-sjxl5\") pod \"360e59d6-8295-48fd-a8d2-2e7fe1e4f402\" (UID: \"360e59d6-8295-48fd-a8d2-2e7fe1e4f402\") " Oct 31 04:43:17.173536 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:43:17.173448 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/360e59d6-8295-48fd-a8d2-2e7fe1e4f402-istio-token\" (UniqueName: \"kubernetes.io/projected/360e59d6-8295-48fd-a8d2-2e7fe1e4f402-istio-token\") pod \"360e59d6-8295-48fd-a8d2-2e7fe1e4f402\" (UID: \"360e59d6-8295-48fd-a8d2-2e7fe1e4f402\") " Oct 31 04:43:17.173536 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:43:17.173503 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/360e59d6-8295-48fd-a8d2-2e7fe1e4f402-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/360e59d6-8295-48fd-a8d2-2e7fe1e4f402-credential-socket\") pod \"360e59d6-8295-48fd-a8d2-2e7fe1e4f402\" (UID: \"360e59d6-8295-48fd-a8d2-2e7fe1e4f402\") " Oct 31 04:43:17.173752 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:43:17.173734 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/360e59d6-8295-48fd-a8d2-2e7fe1e4f402-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/360e59d6-8295-48fd-a8d2-2e7fe1e4f402-workload-socket\") pod \"360e59d6-8295-48fd-a8d2-2e7fe1e4f402\" (UID: \"360e59d6-8295-48fd-a8d2-2e7fe1e4f402\") " Oct 31 04:43:17.173835 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:43:17.173749 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/360e59d6-8295-48fd-a8d2-2e7fe1e4f402-credential-socket" pod "360e59d6-8295-48fd-a8d2-2e7fe1e4f402" (UID: "360e59d6-8295-48fd-a8d2-2e7fe1e4f402"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:43:17.173835 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:43:17.173774 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/360e59d6-8295-48fd-a8d2-2e7fe1e4f402-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/360e59d6-8295-48fd-a8d2-2e7fe1e4f402-istio-podinfo\") pod \"360e59d6-8295-48fd-a8d2-2e7fe1e4f402\" (UID: \"360e59d6-8295-48fd-a8d2-2e7fe1e4f402\") " Oct 31 04:43:17.173835 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:43:17.173808 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/360e59d6-8295-48fd-a8d2-2e7fe1e4f402-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/360e59d6-8295-48fd-a8d2-2e7fe1e4f402-istiod-ca-cert\") pod \"360e59d6-8295-48fd-a8d2-2e7fe1e4f402\" (UID: \"360e59d6-8295-48fd-a8d2-2e7fe1e4f402\") " Oct 31 04:43:17.173988 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:43:17.173940 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/360e59d6-8295-48fd-a8d2-2e7fe1e4f402-workload-socket" pod "360e59d6-8295-48fd-a8d2-2e7fe1e4f402" (UID: "360e59d6-8295-48fd-a8d2-2e7fe1e4f402"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:43:17.174044 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:43:17.174001 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/360e59d6-8295-48fd-a8d2-2e7fe1e4f402-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/360e59d6-8295-48fd-a8d2-2e7fe1e4f402-workload-certs\") pod \"360e59d6-8295-48fd-a8d2-2e7fe1e4f402\" (UID: \"360e59d6-8295-48fd-a8d2-2e7fe1e4f402\") " Oct 31 04:43:17.174096 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:43:17.174076 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/360e59d6-8295-48fd-a8d2-2e7fe1e4f402-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/360e59d6-8295-48fd-a8d2-2e7fe1e4f402-istio-data\") pod \"360e59d6-8295-48fd-a8d2-2e7fe1e4f402\" (UID: \"360e59d6-8295-48fd-a8d2-2e7fe1e4f402\") " Oct 31 04:43:17.174145 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:43:17.174116 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/360e59d6-8295-48fd-a8d2-2e7fe1e4f402-istiod-ca-cert" pod "360e59d6-8295-48fd-a8d2-2e7fe1e4f402" (UID: "360e59d6-8295-48fd-a8d2-2e7fe1e4f402"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:43:17.174392 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:43:17.174365 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/360e59d6-8295-48fd-a8d2-2e7fe1e4f402-workload-certs" pod "360e59d6-8295-48fd-a8d2-2e7fe1e4f402" (UID: "360e59d6-8295-48fd-a8d2-2e7fe1e4f402"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:43:17.174488 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:43:17.174405 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/360e59d6-8295-48fd-a8d2-2e7fe1e4f402-istio-data" pod "360e59d6-8295-48fd-a8d2-2e7fe1e4f402" (UID: "360e59d6-8295-48fd-a8d2-2e7fe1e4f402"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:43:17.174870 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:43:17.174849 4376 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/360e59d6-8295-48fd-a8d2-2e7fe1e4f402-credential-socket\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:43:17.174959 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:43:17.174877 4376 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/360e59d6-8295-48fd-a8d2-2e7fe1e4f402-workload-socket\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:43:17.174959 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:43:17.174892 4376 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/360e59d6-8295-48fd-a8d2-2e7fe1e4f402-istiod-ca-cert\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:43:17.174959 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:43:17.174905 4376 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/360e59d6-8295-48fd-a8d2-2e7fe1e4f402-workload-certs\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:43:17.174959 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:43:17.174917 4376 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/360e59d6-8295-48fd-a8d2-2e7fe1e4f402-istio-data\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:43:17.176158 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:43:17.176130 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/360e59d6-8295-48fd-a8d2-2e7fe1e4f402-istio-token" pod "360e59d6-8295-48fd-a8d2-2e7fe1e4f402" (UID: "360e59d6-8295-48fd-a8d2-2e7fe1e4f402"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:43:17.176423 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:43:17.176401 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/360e59d6-8295-48fd-a8d2-2e7fe1e4f402-istio-podinfo" pod "360e59d6-8295-48fd-a8d2-2e7fe1e4f402" (UID: "360e59d6-8295-48fd-a8d2-2e7fe1e4f402"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 04:43:17.176550 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:43:17.176532 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/360e59d6-8295-48fd-a8d2-2e7fe1e4f402-kube-api-access-sjxl5" pod "360e59d6-8295-48fd-a8d2-2e7fe1e4f402" (UID: "360e59d6-8295-48fd-a8d2-2e7fe1e4f402"). InnerVolumeSpecName "kube-api-access-sjxl5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:43:17.176865 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:43:17.176847 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/360e59d6-8295-48fd-a8d2-2e7fe1e4f402-istio-envoy" pod "360e59d6-8295-48fd-a8d2-2e7fe1e4f402" (UID: "360e59d6-8295-48fd-a8d2-2e7fe1e4f402"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:43:17.224850 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:43:17.224794 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-47c409-8649bf6df7-rbdch" podStartSLOduration=2.224781499 podStartE2EDuration="2.224781499s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:43:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:43:17.221785257 +0000 UTC m=+2344.936119875" watchObservedRunningTime="2026-10-31 04:43:17.224781499 +0000 UTC m=+2344.939116133" Oct 31 04:43:17.276350 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:43:17.276301 4376 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/360e59d6-8295-48fd-a8d2-2e7fe1e4f402-istio-envoy\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:43:17.276451 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:43:17.276353 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjxl5\" (UniqueName: \"kubernetes.io/projected/360e59d6-8295-48fd-a8d2-2e7fe1e4f402-kube-api-access-sjxl5\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:43:17.276451 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:43:17.276371 4376 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/360e59d6-8295-48fd-a8d2-2e7fe1e4f402-istio-token\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:43:17.276451 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:43:17.276383 4376 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/360e59d6-8295-48fd-a8d2-2e7fe1e4f402-istio-podinfo\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:43:17.517540 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:43:17.517484 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-rs8fm"] Oct 31 04:43:17.520943 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:43:17.520917 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-rs8fm"] Oct 31 04:43:18.444632 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:43:18.444594 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="360e59d6-8295-48fd-a8d2-2e7fe1e4f402" path="/var/lib/kubelet/pods/360e59d6-8295-48fd-a8d2-2e7fe1e4f402/volumes" Oct 31 04:43:28.169710 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:43:28.169681 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-47c409-8649bf6df7-rbdch" Oct 31 04:43:32.447081 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:43:32.447047 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-7d6cmbx6k"] Oct 31 04:43:32.447769 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:43:32.447751 4376 cpu_manager.go:414] "containerMap: removing container" podUID="360e59d6-8295-48fd-a8d2-2e7fe1e4f402" containerName="istio-proxy" Oct 31 04:43:32.447813 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:43:32.447770 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="360e59d6-8295-48fd-a8d2-2e7fe1e4f402" containerName="istio-proxy" Oct 31 04:43:32.447922 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:43:32.447913 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="360e59d6-8295-48fd-a8d2-2e7fe1e4f402" containerName="istio-proxy" Oct 31 04:43:32.452846 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:43:32.452828 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-7d6cmbx6k" Oct 31 04:43:32.464680 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:43:32.464661 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-7d6cmbx6k"] Oct 31 04:43:32.468104 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:43:32.468080 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-a99f2a\"/\"payload-processing-external-model-plugins-e2e-extmod-b-a99f2a\"" Oct 31 04:43:32.498372 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:43:32.485042 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-a99f2a\"/\"routing-overlay\"" Oct 31 04:43:32.498776 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:43:32.498746 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-a99f2a\"/\"agc-cred-add5bfec2790\"" Oct 31 04:43:32.499070 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:43:32.499053 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-a99f2a\"/\"payload-processing-external-model-e2e--c07b99ee-dockercfg-gzb75\"" Oct 31 04:43:32.499250 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:43:32.499236 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-a99f2a\"/\"agc-cred-48afeaacb106\"" Oct 31 04:43:32.499418 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:43:32.499400 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-a99f2a\"/\"openshift-service-ca.crt\"" Oct 31 04:43:32.586506 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:43:32.523528 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-a99f2a\"/\"agc-cred-7194275e28ab\"" Oct 31 04:43:32.586506 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:43:32.523676 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-a99f2a\"/\"kube-root-ca.crt\"" Oct 31 04:43:32.633084 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:43:32.633048 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/92a38bb0-a1e4-45d6-82db-2f9cb1652064-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-a99f2a-7d6cmbx6k\" (UID: \"92a38bb0-a1e4-45d6-82db-2f9cb1652064\") " pod="ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-7d6cmbx6k" Oct 31 04:43:32.633404 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:43:32.633374 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92a38bb0-a1e4-45d6-82db-2f9cb1652064-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a99f2a-7d6cmbx6k\" (UID: \"92a38bb0-a1e4-45d6-82db-2f9cb1652064\") " pod="ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-7d6cmbx6k" Oct 31 04:43:32.633597 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:43:32.633563 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/92a38bb0-a1e4-45d6-82db-2f9cb1652064-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a99f2a-7d6cmbx6k\" (UID: \"92a38bb0-a1e4-45d6-82db-2f9cb1652064\") " pod="ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-7d6cmbx6k" Oct 31 04:43:32.633672 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:43:32.633653 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8b55\" (UniqueName: \"kubernetes.io/projected/92a38bb0-a1e4-45d6-82db-2f9cb1652064-kube-api-access-c8b55\") pod \"payload-processing-external-model-e2e-extmod-b-a99f2a-7d6cmbx6k\" (UID: \"92a38bb0-a1e4-45d6-82db-2f9cb1652064\") " pod="ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-7d6cmbx6k" Oct 31 04:43:32.734826 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:43:32.734734 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/92a38bb0-a1e4-45d6-82db-2f9cb1652064-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-a99f2a-7d6cmbx6k\" (UID: \"92a38bb0-a1e4-45d6-82db-2f9cb1652064\") " pod="ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-7d6cmbx6k" Oct 31 04:43:32.735135 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:43:32.735110 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92a38bb0-a1e4-45d6-82db-2f9cb1652064-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a99f2a-7d6cmbx6k\" (UID: \"92a38bb0-a1e4-45d6-82db-2f9cb1652064\") " pod="ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-7d6cmbx6k" Oct 31 04:43:32.735288 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:43:32.735244 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/92a38bb0-a1e4-45d6-82db-2f9cb1652064-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a99f2a-7d6cmbx6k\" (UID: \"92a38bb0-a1e4-45d6-82db-2f9cb1652064\") " pod="ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-7d6cmbx6k" Oct 31 04:43:32.735370 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:43:32.735292 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8b55\" (UniqueName: \"kubernetes.io/projected/92a38bb0-a1e4-45d6-82db-2f9cb1652064-kube-api-access-c8b55\") pod \"payload-processing-external-model-e2e-extmod-b-a99f2a-7d6cmbx6k\" (UID: \"92a38bb0-a1e4-45d6-82db-2f9cb1652064\") " pod="ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-7d6cmbx6k" Oct 31 04:43:32.735838 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:43:32.735817 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92a38bb0-a1e4-45d6-82db-2f9cb1652064-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a99f2a-7d6cmbx6k\" (UID: \"92a38bb0-a1e4-45d6-82db-2f9cb1652064\") " pod="ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-7d6cmbx6k" Oct 31 04:43:32.736085 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:43:32.736066 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/92a38bb0-a1e4-45d6-82db-2f9cb1652064-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a99f2a-7d6cmbx6k\" (UID: \"92a38bb0-a1e4-45d6-82db-2f9cb1652064\") " pod="ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-7d6cmbx6k" Oct 31 04:43:32.737297 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:43:32.737279 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/92a38bb0-a1e4-45d6-82db-2f9cb1652064-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-a99f2a-7d6cmbx6k\" (UID: \"92a38bb0-a1e4-45d6-82db-2f9cb1652064\") " pod="ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-7d6cmbx6k" Oct 31 04:43:32.758727 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:43:32.758699 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8b55\" (UniqueName: \"kubernetes.io/projected/92a38bb0-a1e4-45d6-82db-2f9cb1652064-kube-api-access-c8b55\") pod \"payload-processing-external-model-e2e-extmod-b-a99f2a-7d6cmbx6k\" (UID: \"92a38bb0-a1e4-45d6-82db-2f9cb1652064\") " pod="ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-7d6cmbx6k" Oct 31 04:43:32.770414 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:43:32.770387 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-7d6cmbx6k" Oct 31 04:43:33.141759 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:43:33.141728 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92a38bb0_a1e4_45d6_82db_2f9cb1652064.slice/crio-535fb8f8d2f71dac18b2a8e70e11884cd371ffad615b03aecd3dbbdd2d5c3233 WatchSource:0}: Error finding container 535fb8f8d2f71dac18b2a8e70e11884cd371ffad615b03aecd3dbbdd2d5c3233: Status 404 returned error can't find the container with id 535fb8f8d2f71dac18b2a8e70e11884cd371ffad615b03aecd3dbbdd2d5c3233 Oct 31 04:43:33.145892 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:43:33.145852 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-7d6cmbx6k"] Oct 31 04:43:33.317863 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:43:33.317830 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-7d6cmbx6k" event={"ID":"92a38bb0-a1e4-45d6-82db-2f9cb1652064","Type":"ContainerStarted","Data":"1b8a62b784294736887c557a9b23ca8bca3ec5142367c459c9070a9b4d0ebef7"} Oct 31 04:43:33.318006 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:43:33.317886 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-7d6cmbx6k" event={"ID":"92a38bb0-a1e4-45d6-82db-2f9cb1652064","Type":"ContainerStarted","Data":"535fb8f8d2f71dac18b2a8e70e11884cd371ffad615b03aecd3dbbdd2d5c3233"} Oct 31 04:43:33.318006 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:43:33.317914 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-7d6cmbx6k" Oct 31 04:43:33.363423 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:43:33.363369 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-7d6cmbx6k" podStartSLOduration=1.363341993 podStartE2EDuration="1.363341993s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:43:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:43:33.360295502 +0000 UTC m=+2361.074630118" watchObservedRunningTime="2026-10-31 04:43:33.363341993 +0000 UTC m=+2361.077676609" Oct 31 04:43:41.000534 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:43:41.000494 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 04:43:44.327839 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:43:44.327810 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-7d6cmbx6k" Oct 31 04:44:02.751882 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:44:02.751852 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 04:44:11.003393 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:44:11.003355 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 04:44:41.001145 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:44:41.001112 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 04:44:54.983731 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:44:54.983696 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-7d6cmbx6k"] Oct 31 04:44:59.385769 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:44:59.385736 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 04:44:59.985424 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:44:59.985317 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-7d6cmbx6k" podUID="92a38bb0-a1e4-45d6-82db-2f9cb1652064" containerName="payload-processing" containerID="cri-o://1b8a62b784294736887c557a9b23ca8bca3ec5142367c459c9070a9b4d0ebef7" gracePeriod=25 Oct 31 04:45:00.117560 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:00.117528 4376 generic.go:356] "Generic (PLEG): container finished" podID="92a38bb0-a1e4-45d6-82db-2f9cb1652064" containerID="1b8a62b784294736887c557a9b23ca8bca3ec5142367c459c9070a9b4d0ebef7" exitCode=0 Oct 31 04:45:00.117692 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:00.117597 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-7d6cmbx6k" event={"ID":"92a38bb0-a1e4-45d6-82db-2f9cb1652064","Type":"ContainerDied","Data":"1b8a62b784294736887c557a9b23ca8bca3ec5142367c459c9070a9b4d0ebef7"} Oct 31 04:45:00.223201 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:00.223120 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-47c409-29890365-vsw27"] Oct 31 04:45:00.230007 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:00.229983 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c9707f-29890365-fp5bb"] Oct 31 04:45:00.230144 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:00.230105 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-47c409-29890365-vsw27" Oct 31 04:45:00.235318 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:00.235296 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c9707f-29890365-fp5bb" Oct 31 04:45:00.256790 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:00.252937 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-47c409-29890365-vsw27"] Oct 31 04:45:00.260773 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:00.258151 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-401761-29890365-xnh62"] Oct 31 04:45:00.265528 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:00.265505 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c9707f-29890365-fp5bb"] Oct 31 04:45:00.265638 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:00.265611 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-401761-29890365-xnh62" Oct 31 04:45:00.269664 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:00.269627 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890365-5gwm4"] Oct 31 04:45:00.273882 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:00.273858 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890365-5gwm4" Oct 31 04:45:00.287378 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:00.287358 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-7d6cmbx6k" Oct 31 04:45:00.315640 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:00.315173 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-401761-29890365-xnh62"] Oct 31 04:45:00.323397 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:00.323375 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-227c93-29890365-94kfm"] Oct 31 04:45:00.324162 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:00.324142 4376 cpu_manager.go:414] "containerMap: removing container" podUID="92a38bb0-a1e4-45d6-82db-2f9cb1652064" containerName="payload-processing" Oct 31 04:45:00.324162 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:00.324162 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92a38bb0-a1e4-45d6-82db-2f9cb1652064" containerName="payload-processing" Oct 31 04:45:00.324318 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:00.324287 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="92a38bb0-a1e4-45d6-82db-2f9cb1652064" containerName="payload-processing" Oct 31 04:45:00.329565 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:00.329544 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890365-5gwm4"] Oct 31 04:45:00.329681 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:00.329648 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-227c93-29890365-94kfm" Oct 31 04:45:00.331769 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:00.331750 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzb8r\"" Oct 31 04:45:00.350185 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:00.337406 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92a38bb0-a1e4-45d6-82db-2f9cb1652064-provider-credentials\" (UniqueName: \"kubernetes.io/projected/92a38bb0-a1e4-45d6-82db-2f9cb1652064-provider-credentials\") pod \"92a38bb0-a1e4-45d6-82db-2f9cb1652064\" (UID: \"92a38bb0-a1e4-45d6-82db-2f9cb1652064\") " Oct 31 04:45:00.350185 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:00.337481 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92a38bb0-a1e4-45d6-82db-2f9cb1652064-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92a38bb0-a1e4-45d6-82db-2f9cb1652064-plugins-config-volume\") pod \"92a38bb0-a1e4-45d6-82db-2f9cb1652064\" (UID: \"92a38bb0-a1e4-45d6-82db-2f9cb1652064\") " Oct 31 04:45:00.350185 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:00.337606 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92a38bb0-a1e4-45d6-82db-2f9cb1652064-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/92a38bb0-a1e4-45d6-82db-2f9cb1652064-routing-overlay-volume\") pod \"92a38bb0-a1e4-45d6-82db-2f9cb1652064\" (UID: \"92a38bb0-a1e4-45d6-82db-2f9cb1652064\") " Oct 31 04:45:00.350185 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:00.337769 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92a38bb0-a1e4-45d6-82db-2f9cb1652064-kube-api-access-c8b55\" (UniqueName: \"kubernetes.io/projected/92a38bb0-a1e4-45d6-82db-2f9cb1652064-kube-api-access-c8b55\") pod \"92a38bb0-a1e4-45d6-82db-2f9cb1652064\" (UID: \"92a38bb0-a1e4-45d6-82db-2f9cb1652064\") " Oct 31 04:45:00.350185 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:00.338153 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5059d94-428a-4df5-99c8-4b2e4f3f87ab-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-c9707f-29890365-fp5bb\" (UID: \"d5059d94-428a-4df5-99c8-4b2e4f3f87ab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c9707f-29890365-fp5bb" Oct 31 04:45:00.350185 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:00.338210 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbee2356-f532-40d7-90b6-2572284f0b55-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-47c409-29890365-vsw27\" (UID: \"fbee2356-f532-40d7-90b6-2572284f0b55\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-47c409-29890365-vsw27" Oct 31 04:45:00.350185 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:00.338273 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjw7m\" (UniqueName: \"kubernetes.io/projected/8032699a-3ca2-4cca-9366-10006ea03b4d-kube-api-access-wjw7m\") pod \"maas-api-key-cleanup-29890365-5gwm4\" (UID: \"8032699a-3ca2-4cca-9366-10006ea03b4d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890365-5gwm4" Oct 31 04:45:00.350185 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:00.338395 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psp6l\" (UniqueName: \"kubernetes.io/projected/fbee2356-f532-40d7-90b6-2572284f0b55-kube-api-access-psp6l\") pod \"maas-api-key-cleanup-e2e-extmod-a-47c409-29890365-vsw27\" (UID: \"fbee2356-f532-40d7-90b6-2572284f0b55\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-47c409-29890365-vsw27" Oct 31 04:45:00.350185 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:00.338438 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdn94\" (UniqueName: \"kubernetes.io/projected/d5059d94-428a-4df5-99c8-4b2e4f3f87ab-kube-api-access-tdn94\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-c9707f-29890365-fp5bb\" (UID: \"d5059d94-428a-4df5-99c8-4b2e4f3f87ab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c9707f-29890365-fp5bb" Oct 31 04:45:00.350185 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:00.338483 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2608ed82-e75d-41bf-8710-0faf0ce0ae08-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-401761-29890365-xnh62\" (UID: \"2608ed82-e75d-41bf-8710-0faf0ce0ae08\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-401761-29890365-xnh62" Oct 31 04:45:00.350185 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:00.338512 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8032699a-3ca2-4cca-9366-10006ea03b4d-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890365-5gwm4\" (UID: \"8032699a-3ca2-4cca-9366-10006ea03b4d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890365-5gwm4" Oct 31 04:45:00.350185 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:00.338537 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98glr\" (UniqueName: \"kubernetes.io/projected/2608ed82-e75d-41bf-8710-0faf0ce0ae08-kube-api-access-98glr\") pod \"maas-api-key-cleanup-e2e-extmod-a-401761-29890365-xnh62\" (UID: \"2608ed82-e75d-41bf-8710-0faf0ce0ae08\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-401761-29890365-xnh62" Oct 31 04:45:00.375992 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:00.365763 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92a38bb0-a1e4-45d6-82db-2f9cb1652064-provider-credentials" pod "92a38bb0-a1e4-45d6-82db-2f9cb1652064" (UID: "92a38bb0-a1e4-45d6-82db-2f9cb1652064"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:45:00.375992 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:00.373525 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92a38bb0-a1e4-45d6-82db-2f9cb1652064-routing-overlay-volume" pod "92a38bb0-a1e4-45d6-82db-2f9cb1652064" (UID: "92a38bb0-a1e4-45d6-82db-2f9cb1652064"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:45:00.375992 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:00.373776 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92a38bb0-a1e4-45d6-82db-2f9cb1652064-kube-api-access-c8b55" pod "92a38bb0-a1e4-45d6-82db-2f9cb1652064" (UID: "92a38bb0-a1e4-45d6-82db-2f9cb1652064"). InnerVolumeSpecName "kube-api-access-c8b55". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:45:00.398091 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:00.396091 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92a38bb0-a1e4-45d6-82db-2f9cb1652064-plugins-config-volume" pod "92a38bb0-a1e4-45d6-82db-2f9cb1652064" (UID: "92a38bb0-a1e4-45d6-82db-2f9cb1652064"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:45:00.423734 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:00.421587 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-227c93-29890365-94kfm"] Oct 31 04:45:00.442023 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:00.440215 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjw7m\" (UniqueName: \"kubernetes.io/projected/8032699a-3ca2-4cca-9366-10006ea03b4d-kube-api-access-wjw7m\") pod \"maas-api-key-cleanup-29890365-5gwm4\" (UID: \"8032699a-3ca2-4cca-9366-10006ea03b4d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890365-5gwm4" Oct 31 04:45:00.442023 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:00.440307 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psp6l\" (UniqueName: \"kubernetes.io/projected/fbee2356-f532-40d7-90b6-2572284f0b55-kube-api-access-psp6l\") pod \"maas-api-key-cleanup-e2e-extmod-a-47c409-29890365-vsw27\" (UID: \"fbee2356-f532-40d7-90b6-2572284f0b55\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-47c409-29890365-vsw27" Oct 31 04:45:00.442023 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:00.440366 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdn94\" (UniqueName: \"kubernetes.io/projected/d5059d94-428a-4df5-99c8-4b2e4f3f87ab-kube-api-access-tdn94\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-c9707f-29890365-fp5bb\" (UID: \"d5059d94-428a-4df5-99c8-4b2e4f3f87ab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c9707f-29890365-fp5bb" Oct 31 04:45:00.442023 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:00.440411 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2608ed82-e75d-41bf-8710-0faf0ce0ae08-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-401761-29890365-xnh62\" (UID: \"2608ed82-e75d-41bf-8710-0faf0ce0ae08\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-401761-29890365-xnh62" Oct 31 04:45:00.442023 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:00.440443 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8032699a-3ca2-4cca-9366-10006ea03b4d-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890365-5gwm4\" (UID: \"8032699a-3ca2-4cca-9366-10006ea03b4d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890365-5gwm4" Oct 31 04:45:00.442023 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:00.440471 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98glr\" (UniqueName: \"kubernetes.io/projected/2608ed82-e75d-41bf-8710-0faf0ce0ae08-kube-api-access-98glr\") pod \"maas-api-key-cleanup-e2e-extmod-a-401761-29890365-xnh62\" (UID: \"2608ed82-e75d-41bf-8710-0faf0ce0ae08\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-401761-29890365-xnh62" Oct 31 04:45:00.442023 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:00.440520 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/18cae0c8-c13a-4ebe-9c1d-3f87232d46fb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-227c93-29890365-94kfm\" (UID: \"18cae0c8-c13a-4ebe-9c1d-3f87232d46fb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-227c93-29890365-94kfm" Oct 31 04:45:00.442023 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:00.440720 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbtxk\" (UniqueName: \"kubernetes.io/projected/18cae0c8-c13a-4ebe-9c1d-3f87232d46fb-kube-api-access-rbtxk\") pod \"maas-api-key-cleanup-e2e-extmod-b-227c93-29890365-94kfm\" (UID: \"18cae0c8-c13a-4ebe-9c1d-3f87232d46fb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-227c93-29890365-94kfm" Oct 31 04:45:00.442023 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:00.440759 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5059d94-428a-4df5-99c8-4b2e4f3f87ab-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-c9707f-29890365-fp5bb\" (UID: \"d5059d94-428a-4df5-99c8-4b2e4f3f87ab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c9707f-29890365-fp5bb" Oct 31 04:45:00.442023 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:00.440821 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbee2356-f532-40d7-90b6-2572284f0b55-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-47c409-29890365-vsw27\" (UID: \"fbee2356-f532-40d7-90b6-2572284f0b55\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-47c409-29890365-vsw27" Oct 31 04:45:00.442023 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:00.440901 4376 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/92a38bb0-a1e4-45d6-82db-2f9cb1652064-provider-credentials\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:45:00.442023 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:00.440918 4376 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92a38bb0-a1e4-45d6-82db-2f9cb1652064-plugins-config-volume\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:45:00.442023 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:00.440932 4376 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/92a38bb0-a1e4-45d6-82db-2f9cb1652064-routing-overlay-volume\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:45:00.442023 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:00.440948 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8b55\" (UniqueName: \"kubernetes.io/projected/92a38bb0-a1e4-45d6-82db-2f9cb1652064-kube-api-access-c8b55\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:45:00.442023 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:00.441690 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbee2356-f532-40d7-90b6-2572284f0b55-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-47c409-29890365-vsw27\" (UID: \"fbee2356-f532-40d7-90b6-2572284f0b55\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-47c409-29890365-vsw27" Oct 31 04:45:00.443001 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:00.442971 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2608ed82-e75d-41bf-8710-0faf0ce0ae08-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-401761-29890365-xnh62\" (UID: \"2608ed82-e75d-41bf-8710-0faf0ce0ae08\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-401761-29890365-xnh62" Oct 31 04:45:00.443682 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:00.443659 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8032699a-3ca2-4cca-9366-10006ea03b4d-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890365-5gwm4\" (UID: \"8032699a-3ca2-4cca-9366-10006ea03b4d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890365-5gwm4" Oct 31 04:45:00.446031 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:00.444473 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5059d94-428a-4df5-99c8-4b2e4f3f87ab-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-c9707f-29890365-fp5bb\" (UID: \"d5059d94-428a-4df5-99c8-4b2e4f3f87ab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c9707f-29890365-fp5bb" Oct 31 04:45:00.469032 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:00.469002 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdn94\" (UniqueName: \"kubernetes.io/projected/d5059d94-428a-4df5-99c8-4b2e4f3f87ab-kube-api-access-tdn94\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-c9707f-29890365-fp5bb\" (UID: \"d5059d94-428a-4df5-99c8-4b2e4f3f87ab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c9707f-29890365-fp5bb" Oct 31 04:45:00.469521 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:00.469497 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-psp6l\" (UniqueName: \"kubernetes.io/projected/fbee2356-f532-40d7-90b6-2572284f0b55-kube-api-access-psp6l\") pod \"maas-api-key-cleanup-e2e-extmod-a-47c409-29890365-vsw27\" (UID: \"fbee2356-f532-40d7-90b6-2572284f0b55\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-47c409-29890365-vsw27" Oct 31 04:45:00.469583 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:00.469535 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjw7m\" (UniqueName: \"kubernetes.io/projected/8032699a-3ca2-4cca-9366-10006ea03b4d-kube-api-access-wjw7m\") pod \"maas-api-key-cleanup-29890365-5gwm4\" (UID: \"8032699a-3ca2-4cca-9366-10006ea03b4d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890365-5gwm4" Oct 31 04:45:00.469622 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:00.469603 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98glr\" (UniqueName: \"kubernetes.io/projected/2608ed82-e75d-41bf-8710-0faf0ce0ae08-kube-api-access-98glr\") pod \"maas-api-key-cleanup-e2e-extmod-a-401761-29890365-xnh62\" (UID: \"2608ed82-e75d-41bf-8710-0faf0ce0ae08\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-401761-29890365-xnh62" Oct 31 04:45:00.542385 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:00.542285 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbtxk\" (UniqueName: \"kubernetes.io/projected/18cae0c8-c13a-4ebe-9c1d-3f87232d46fb-kube-api-access-rbtxk\") pod \"maas-api-key-cleanup-e2e-extmod-b-227c93-29890365-94kfm\" (UID: \"18cae0c8-c13a-4ebe-9c1d-3f87232d46fb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-227c93-29890365-94kfm" Oct 31 04:45:00.542553 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:00.542493 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/18cae0c8-c13a-4ebe-9c1d-3f87232d46fb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-227c93-29890365-94kfm\" (UID: \"18cae0c8-c13a-4ebe-9c1d-3f87232d46fb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-227c93-29890365-94kfm" Oct 31 04:45:00.543078 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:00.543052 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/18cae0c8-c13a-4ebe-9c1d-3f87232d46fb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-227c93-29890365-94kfm\" (UID: \"18cae0c8-c13a-4ebe-9c1d-3f87232d46fb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-227c93-29890365-94kfm" Oct 31 04:45:00.565986 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:00.565957 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbtxk\" (UniqueName: \"kubernetes.io/projected/18cae0c8-c13a-4ebe-9c1d-3f87232d46fb-kube-api-access-rbtxk\") pod \"maas-api-key-cleanup-e2e-extmod-b-227c93-29890365-94kfm\" (UID: \"18cae0c8-c13a-4ebe-9c1d-3f87232d46fb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-227c93-29890365-94kfm" Oct 31 04:45:00.580231 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:00.580207 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-47c409-29890365-vsw27" Oct 31 04:45:00.589071 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:00.589049 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c9707f-29890365-fp5bb" Oct 31 04:45:00.601132 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:00.601107 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-401761-29890365-xnh62" Oct 31 04:45:00.605893 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:00.605869 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890365-5gwm4" Oct 31 04:45:00.714929 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:00.714753 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-227c93-29890365-94kfm" Oct 31 04:45:00.906735 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:00.906701 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-47c409-29890365-vsw27"] Oct 31 04:45:00.906888 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:45:00.906841 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbee2356_f532_40d7_90b6_2572284f0b55.slice/crio-f5773e5338e022c426d02f5bee6a315805ddaa5fd39920163625357905682cd8 WatchSource:0}: Error finding container f5773e5338e022c426d02f5bee6a315805ddaa5fd39920163625357905682cd8: Status 404 returned error can't find the container with id f5773e5338e022c426d02f5bee6a315805ddaa5fd39920163625357905682cd8 Oct 31 04:45:00.926900 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:45:00.926867 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5059d94_428a_4df5_99c8_4b2e4f3f87ab.slice/crio-bad5169f219c40fb0261a489d9e47d242f295aab6b9c92f82cd0c19a8f5722c2 WatchSource:0}: Error finding container bad5169f219c40fb0261a489d9e47d242f295aab6b9c92f82cd0c19a8f5722c2: Status 404 returned error can't find the container with id bad5169f219c40fb0261a489d9e47d242f295aab6b9c92f82cd0c19a8f5722c2 Oct 31 04:45:00.930864 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:00.930838 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c9707f-29890365-fp5bb"] Oct 31 04:45:01.059254 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:01.059156 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-401761-29890365-xnh62"] Oct 31 04:45:01.066903 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:01.066876 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890365-5gwm4"] Oct 31 04:45:01.090884 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:01.090860 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-227c93-29890365-94kfm"] Oct 31 04:45:01.128522 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:01.128489 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890365-5gwm4" event={"ID":"8032699a-3ca2-4cca-9366-10006ea03b4d","Type":"ContainerStarted","Data":"db5d9824f5541631b4669c1c8a44cf8bd53cfbf452a1e728f849a003f63dd858"} Oct 31 04:45:01.129874 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:01.129850 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c9707f-29890365-fp5bb" event={"ID":"d5059d94-428a-4df5-99c8-4b2e4f3f87ab","Type":"ContainerStarted","Data":"bad5169f219c40fb0261a489d9e47d242f295aab6b9c92f82cd0c19a8f5722c2"} Oct 31 04:45:01.131118 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:01.131098 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-47c409-29890365-vsw27" event={"ID":"fbee2356-f532-40d7-90b6-2572284f0b55","Type":"ContainerStarted","Data":"f5773e5338e022c426d02f5bee6a315805ddaa5fd39920163625357905682cd8"} Oct 31 04:45:01.132370 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:01.132322 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-7d6cmbx6k" Oct 31 04:45:01.132450 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:01.132363 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-7d6cmbx6k" event={"ID":"92a38bb0-a1e4-45d6-82db-2f9cb1652064","Type":"ContainerDied","Data":"535fb8f8d2f71dac18b2a8e70e11884cd371ffad615b03aecd3dbbdd2d5c3233"} Oct 31 04:45:01.132450 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:01.132411 4376 scope.go:122] "RemoveContainer" containerID="1b8a62b784294736887c557a9b23ca8bca3ec5142367c459c9070a9b4d0ebef7" Oct 31 04:45:01.140694 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:45:01.140664 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2608ed82_e75d_41bf_8710_0faf0ce0ae08.slice/crio-5be17e93c5ace15cb3155f32866517509fd4ecb7a93c29e44848ba4aab3f7ce9 WatchSource:0}: Error finding container 5be17e93c5ace15cb3155f32866517509fd4ecb7a93c29e44848ba4aab3f7ce9: Status 404 returned error can't find the container with id 5be17e93c5ace15cb3155f32866517509fd4ecb7a93c29e44848ba4aab3f7ce9 Oct 31 04:45:01.144815 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:45:01.144769 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18cae0c8_c13a_4ebe_9c1d_3f87232d46fb.slice/crio-c53de2aeba74e0630dfb014bb76f0d5bb717420d9b3398efd726b1dc99e4fc8c WatchSource:0}: Error finding container c53de2aeba74e0630dfb014bb76f0d5bb717420d9b3398efd726b1dc99e4fc8c: Status 404 returned error can't find the container with id c53de2aeba74e0630dfb014bb76f0d5bb717420d9b3398efd726b1dc99e4fc8c Oct 31 04:45:01.187252 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:01.187224 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-7d6cmbx6k"] Oct 31 04:45:01.190728 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:01.190706 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-7d6cmbx6k"] Oct 31 04:45:01.378770 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:01.378734 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-55cfq6jfg"] Oct 31 04:45:01.383434 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:01.383410 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-55cfq6jfg" Oct 31 04:45:01.399315 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:01.399268 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-ded435\"/\"routing-overlay\"" Oct 31 04:45:01.399315 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:01.399307 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-ded435\"/\"payload-processing-external-model-plugins-e2e-extmod-b-ded435\"" Oct 31 04:45:01.399686 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:01.399411 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-ded435\"/\"openshift-service-ca.crt\"" Oct 31 04:45:01.402274 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:01.402202 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-ded435\"/\"payload-processing-external-model-e2e--b0123f89-dockercfg-hvjs2\"" Oct 31 04:45:01.402274 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:01.402251 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-ded435\"/\"agc-cred-3637b2c5d50f\"" Oct 31 04:45:01.402538 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:01.402353 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-55cfq6jfg"] Oct 31 04:45:01.489982 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:01.489948 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7c143dfc-5cce-460b-b348-c5ed65ab662a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ded435-55cfq6jfg\" (UID: \"7c143dfc-5cce-460b-b348-c5ed65ab662a\") " pod="ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-55cfq6jfg" Oct 31 04:45:01.490136 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:01.490117 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svq5k\" (UniqueName: \"kubernetes.io/projected/7c143dfc-5cce-460b-b348-c5ed65ab662a-kube-api-access-svq5k\") pod \"payload-processing-external-model-e2e-extmod-b-ded435-55cfq6jfg\" (UID: \"7c143dfc-5cce-460b-b348-c5ed65ab662a\") " pod="ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-55cfq6jfg" Oct 31 04:45:01.490211 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:01.490155 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c143dfc-5cce-460b-b348-c5ed65ab662a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ded435-55cfq6jfg\" (UID: \"7c143dfc-5cce-460b-b348-c5ed65ab662a\") " pod="ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-55cfq6jfg" Oct 31 04:45:01.490382 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:01.490359 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7c143dfc-5cce-460b-b348-c5ed65ab662a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-ded435-55cfq6jfg\" (UID: \"7c143dfc-5cce-460b-b348-c5ed65ab662a\") " pod="ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-55cfq6jfg" Oct 31 04:45:01.499263 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:01.499246 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-ded435\"/\"kube-root-ca.crt\"" Oct 31 04:45:01.591268 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:01.591243 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svq5k\" (UniqueName: \"kubernetes.io/projected/7c143dfc-5cce-460b-b348-c5ed65ab662a-kube-api-access-svq5k\") pod \"payload-processing-external-model-e2e-extmod-b-ded435-55cfq6jfg\" (UID: \"7c143dfc-5cce-460b-b348-c5ed65ab662a\") " pod="ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-55cfq6jfg" Oct 31 04:45:01.591389 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:01.591279 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c143dfc-5cce-460b-b348-c5ed65ab662a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ded435-55cfq6jfg\" (UID: \"7c143dfc-5cce-460b-b348-c5ed65ab662a\") " pod="ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-55cfq6jfg" Oct 31 04:45:01.591444 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:01.591392 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7c143dfc-5cce-460b-b348-c5ed65ab662a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-ded435-55cfq6jfg\" (UID: \"7c143dfc-5cce-460b-b348-c5ed65ab662a\") " pod="ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-55cfq6jfg" Oct 31 04:45:01.591640 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:01.591616 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7c143dfc-5cce-460b-b348-c5ed65ab662a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ded435-55cfq6jfg\" (UID: \"7c143dfc-5cce-460b-b348-c5ed65ab662a\") " pod="ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-55cfq6jfg" Oct 31 04:45:01.592089 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:01.592072 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c143dfc-5cce-460b-b348-c5ed65ab662a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ded435-55cfq6jfg\" (UID: \"7c143dfc-5cce-460b-b348-c5ed65ab662a\") " pod="ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-55cfq6jfg" Oct 31 04:45:01.592146 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:01.592126 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7c143dfc-5cce-460b-b348-c5ed65ab662a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ded435-55cfq6jfg\" (UID: \"7c143dfc-5cce-460b-b348-c5ed65ab662a\") " pod="ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-55cfq6jfg" Oct 31 04:45:01.593688 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:01.593669 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7c143dfc-5cce-460b-b348-c5ed65ab662a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-ded435-55cfq6jfg\" (UID: \"7c143dfc-5cce-460b-b348-c5ed65ab662a\") " pod="ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-55cfq6jfg" Oct 31 04:45:01.620071 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:01.620045 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-svq5k\" (UniqueName: \"kubernetes.io/projected/7c143dfc-5cce-460b-b348-c5ed65ab662a-kube-api-access-svq5k\") pod \"payload-processing-external-model-e2e-extmod-b-ded435-55cfq6jfg\" (UID: \"7c143dfc-5cce-460b-b348-c5ed65ab662a\") " pod="ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-55cfq6jfg" Oct 31 04:45:01.697604 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:01.697584 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-55cfq6jfg" Oct 31 04:45:02.002169 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:02.002138 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-55cfq6jfg"] Oct 31 04:45:02.009479 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:45:02.009451 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c143dfc_5cce_460b_b348_c5ed65ab662a.slice/crio-a3bd5e9117cac169884a39965ec3cc8ebc3b3b0eb0e6b0e0f2644ded07e37c4f WatchSource:0}: Error finding container a3bd5e9117cac169884a39965ec3cc8ebc3b3b0eb0e6b0e0f2644ded07e37c4f: Status 404 returned error can't find the container with id a3bd5e9117cac169884a39965ec3cc8ebc3b3b0eb0e6b0e0f2644ded07e37c4f Oct 31 04:45:02.145667 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:02.145636 4376 generic.go:356] "Generic (PLEG): container finished" podID="d5059d94-428a-4df5-99c8-4b2e4f3f87ab" containerID="b65df97c4fc593e30fccc21bbf9d3334bb1b23509144d689bd8ae368425ed38d" exitCode=60 Oct 31 04:45:02.145830 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:02.145804 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c9707f-29890365-fp5bb" event={"ID":"d5059d94-428a-4df5-99c8-4b2e4f3f87ab","Type":"ContainerDied","Data":"b65df97c4fc593e30fccc21bbf9d3334bb1b23509144d689bd8ae368425ed38d"} Oct 31 04:45:02.146226 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:02.146207 4376 scope.go:122] "RemoveContainer" containerID="b65df97c4fc593e30fccc21bbf9d3334bb1b23509144d689bd8ae368425ed38d" Oct 31 04:45:02.149997 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:02.149966 4376 generic.go:356] "Generic (PLEG): container finished" podID="fbee2356-f532-40d7-90b6-2572284f0b55" containerID="384ba6f6b7edf48a6f897f2b2caf058f12aad22f7c35ca5e317ae2497c5e264a" exitCode=60 Oct 31 04:45:02.150103 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:02.150064 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-47c409-29890365-vsw27" event={"ID":"fbee2356-f532-40d7-90b6-2572284f0b55","Type":"ContainerDied","Data":"384ba6f6b7edf48a6f897f2b2caf058f12aad22f7c35ca5e317ae2497c5e264a"} Oct 31 04:45:02.150354 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:02.150311 4376 scope.go:122] "RemoveContainer" containerID="384ba6f6b7edf48a6f897f2b2caf058f12aad22f7c35ca5e317ae2497c5e264a" Oct 31 04:45:02.152942 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:02.152919 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-227c93-29890365-94kfm" event={"ID":"18cae0c8-c13a-4ebe-9c1d-3f87232d46fb","Type":"ContainerStarted","Data":"601381926c323751036860c0d479827e81631249c8df3e3e4f3b3df3a4fb962b"} Oct 31 04:45:02.153044 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:02.152964 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-227c93-29890365-94kfm" event={"ID":"18cae0c8-c13a-4ebe-9c1d-3f87232d46fb","Type":"ContainerStarted","Data":"c53de2aeba74e0630dfb014bb76f0d5bb717420d9b3398efd726b1dc99e4fc8c"} Oct 31 04:45:02.154691 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:02.154667 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-401761-29890365-xnh62" event={"ID":"2608ed82-e75d-41bf-8710-0faf0ce0ae08","Type":"ContainerStarted","Data":"45372ebb91a16fe4020d7adac9f0b7760f1f87346462e787df9a71dcc22e0c45"} Oct 31 04:45:02.154796 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:02.154711 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-401761-29890365-xnh62" event={"ID":"2608ed82-e75d-41bf-8710-0faf0ce0ae08","Type":"ContainerStarted","Data":"5be17e93c5ace15cb3155f32866517509fd4ecb7a93c29e44848ba4aab3f7ce9"} Oct 31 04:45:02.156295 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:02.156270 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890365-5gwm4" event={"ID":"8032699a-3ca2-4cca-9366-10006ea03b4d","Type":"ContainerStarted","Data":"5a8a113ad9d1858f63ebbe71289b41ebf42b8e0e84b6379d48171799d628bc83"} Oct 31 04:45:02.158546 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:02.158523 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-55cfq6jfg" event={"ID":"7c143dfc-5cce-460b-b348-c5ed65ab662a","Type":"ContainerStarted","Data":"a3bd5e9117cac169884a39965ec3cc8ebc3b3b0eb0e6b0e0f2644ded07e37c4f"} Oct 31 04:45:02.158740 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:02.158720 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-55cfq6jfg" Oct 31 04:45:02.229951 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:02.229907 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890365-5gwm4" podStartSLOduration=2.229893709 podStartE2EDuration="2.229893709s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:45:02.226073006 +0000 UTC m=+2449.940407622" watchObservedRunningTime="2026-10-31 04:45:02.229893709 +0000 UTC m=+2449.944228319" Oct 31 04:45:02.273358 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:02.273294 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-227c93-29890365-94kfm" podStartSLOduration=2.273279565 podStartE2EDuration="2.273279565s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:45:02.27015508 +0000 UTC m=+2449.984489717" watchObservedRunningTime="2026-10-31 04:45:02.273279565 +0000 UTC m=+2449.987614180" Oct 31 04:45:02.312985 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:02.312938 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-55cfq6jfg" podStartSLOduration=1.312924904 podStartE2EDuration="1.312924904s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:45:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:45:02.312231544 +0000 UTC m=+2450.026566159" watchObservedRunningTime="2026-10-31 04:45:02.312924904 +0000 UTC m=+2450.027259519" Oct 31 04:45:02.394040 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:02.393996 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-401761-29890365-xnh62" podStartSLOduration=2.393983858 podStartE2EDuration="2.393983858s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:45:02.391876856 +0000 UTC m=+2450.106211495" watchObservedRunningTime="2026-10-31 04:45:02.393983858 +0000 UTC m=+2450.108318473" Oct 31 04:45:02.446643 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:02.446615 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92a38bb0-a1e4-45d6-82db-2f9cb1652064" path="/var/lib/kubelet/pods/92a38bb0-a1e4-45d6-82db-2f9cb1652064/volumes" Oct 31 04:45:03.169868 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:03.169841 4376 generic.go:356] "Generic (PLEG): container finished" podID="18cae0c8-c13a-4ebe-9c1d-3f87232d46fb" containerID="601381926c323751036860c0d479827e81631249c8df3e3e4f3b3df3a4fb962b" exitCode=60 Oct 31 04:45:03.170005 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:03.169926 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-227c93-29890365-94kfm" event={"ID":"18cae0c8-c13a-4ebe-9c1d-3f87232d46fb","Type":"ContainerDied","Data":"601381926c323751036860c0d479827e81631249c8df3e3e4f3b3df3a4fb962b"} Oct 31 04:45:03.170251 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:03.170230 4376 scope.go:122] "RemoveContainer" containerID="601381926c323751036860c0d479827e81631249c8df3e3e4f3b3df3a4fb962b" Oct 31 04:45:03.171785 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:03.171763 4376 generic.go:356] "Generic (PLEG): container finished" podID="2608ed82-e75d-41bf-8710-0faf0ce0ae08" containerID="45372ebb91a16fe4020d7adac9f0b7760f1f87346462e787df9a71dcc22e0c45" exitCode=60 Oct 31 04:45:03.171887 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:03.171855 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-401761-29890365-xnh62" event={"ID":"2608ed82-e75d-41bf-8710-0faf0ce0ae08","Type":"ContainerDied","Data":"45372ebb91a16fe4020d7adac9f0b7760f1f87346462e787df9a71dcc22e0c45"} Oct 31 04:45:03.172113 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:03.172084 4376 scope.go:122] "RemoveContainer" containerID="45372ebb91a16fe4020d7adac9f0b7760f1f87346462e787df9a71dcc22e0c45" Oct 31 04:45:03.174926 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:03.174772 4376 generic.go:356] "Generic (PLEG): container finished" podID="8032699a-3ca2-4cca-9366-10006ea03b4d" containerID="5a8a113ad9d1858f63ebbe71289b41ebf42b8e0e84b6379d48171799d628bc83" exitCode=60 Oct 31 04:45:03.174926 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:03.174857 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890365-5gwm4" event={"ID":"8032699a-3ca2-4cca-9366-10006ea03b4d","Type":"ContainerDied","Data":"5a8a113ad9d1858f63ebbe71289b41ebf42b8e0e84b6379d48171799d628bc83"} Oct 31 04:45:03.175125 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:03.175097 4376 scope.go:122] "RemoveContainer" containerID="5a8a113ad9d1858f63ebbe71289b41ebf42b8e0e84b6379d48171799d628bc83" Oct 31 04:45:03.178309 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:03.178286 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-55cfq6jfg" event={"ID":"7c143dfc-5cce-460b-b348-c5ed65ab662a","Type":"ContainerStarted","Data":"c6da9cdf59c1208f7a633c9a1d4884ab5a9aacd256b7b92ad426b7ba2567427f"} Oct 31 04:45:03.180387 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:03.180365 4376 generic.go:356] "Generic (PLEG): container finished" podID="d5059d94-428a-4df5-99c8-4b2e4f3f87ab" containerID="f24c3d85d2cea9de41f32a0f137918969aba98789c853fa50a1d52eb859ca7e2" exitCode=60 Oct 31 04:45:03.180477 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:03.180440 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c9707f-29890365-fp5bb" event={"ID":"d5059d94-428a-4df5-99c8-4b2e4f3f87ab","Type":"ContainerDied","Data":"f24c3d85d2cea9de41f32a0f137918969aba98789c853fa50a1d52eb859ca7e2"} Oct 31 04:45:03.180531 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:03.180490 4376 scope.go:122] "RemoveContainer" containerID="b65df97c4fc593e30fccc21bbf9d3334bb1b23509144d689bd8ae368425ed38d" Oct 31 04:45:03.180637 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:03.180620 4376 scope.go:122] "RemoveContainer" containerID="f24c3d85d2cea9de41f32a0f137918969aba98789c853fa50a1d52eb859ca7e2" Oct 31 04:45:03.180878 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:45:03.180853 4376 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-c9707f-29890365-fp5bb_odh-ai-gateway-infra(d5059d94-428a-4df5-99c8-4b2e4f3f87ab)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c9707f-29890365-fp5bb" podUID="d5059d94-428a-4df5-99c8-4b2e4f3f87ab" Oct 31 04:45:03.182205 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:03.182187 4376 generic.go:356] "Generic (PLEG): container finished" podID="fbee2356-f532-40d7-90b6-2572284f0b55" containerID="c08f3a719a9de100d5cc2ab560c363e95748626d54c51e265e8ab5c39db3ac1c" exitCode=60 Oct 31 04:45:03.182314 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:03.182214 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-47c409-29890365-vsw27" event={"ID":"fbee2356-f532-40d7-90b6-2572284f0b55","Type":"ContainerDied","Data":"c08f3a719a9de100d5cc2ab560c363e95748626d54c51e265e8ab5c39db3ac1c"} Oct 31 04:45:03.182443 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:03.182429 4376 scope.go:122] "RemoveContainer" containerID="c08f3a719a9de100d5cc2ab560c363e95748626d54c51e265e8ab5c39db3ac1c" Oct 31 04:45:03.182633 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:45:03.182614 4376 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-47c409-29890365-vsw27_odh-ai-gateway-infra(fbee2356-f532-40d7-90b6-2572284f0b55)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-47c409-29890365-vsw27" podUID="fbee2356-f532-40d7-90b6-2572284f0b55" Oct 31 04:45:03.261714 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:03.261289 4376 scope.go:122] "RemoveContainer" containerID="384ba6f6b7edf48a6f897f2b2caf058f12aad22f7c35ca5e317ae2497c5e264a" Oct 31 04:45:04.192449 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:04.192424 4376 scope.go:122] "RemoveContainer" containerID="c08f3a719a9de100d5cc2ab560c363e95748626d54c51e265e8ab5c39db3ac1c" Oct 31 04:45:04.192826 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:45:04.192788 4376 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-47c409-29890365-vsw27_odh-ai-gateway-infra(fbee2356-f532-40d7-90b6-2572284f0b55)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-47c409-29890365-vsw27" podUID="fbee2356-f532-40d7-90b6-2572284f0b55" Oct 31 04:45:04.194156 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:04.194132 4376 generic.go:356] "Generic (PLEG): container finished" podID="18cae0c8-c13a-4ebe-9c1d-3f87232d46fb" containerID="7321bff73d704d8f9605d175345e161ba936a51d7c4a50414f78d3ee07700134" exitCode=60 Oct 31 04:45:04.194239 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:04.194210 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-227c93-29890365-94kfm" event={"ID":"18cae0c8-c13a-4ebe-9c1d-3f87232d46fb","Type":"ContainerDied","Data":"7321bff73d704d8f9605d175345e161ba936a51d7c4a50414f78d3ee07700134"} Oct 31 04:45:04.194283 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:04.194264 4376 scope.go:122] "RemoveContainer" containerID="601381926c323751036860c0d479827e81631249c8df3e3e4f3b3df3a4fb962b" Oct 31 04:45:04.194477 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:04.194456 4376 scope.go:122] "RemoveContainer" containerID="7321bff73d704d8f9605d175345e161ba936a51d7c4a50414f78d3ee07700134" Oct 31 04:45:04.194804 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:45:04.194779 4376 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-227c93-29890365-94kfm_odh-ai-gateway-infra(18cae0c8-c13a-4ebe-9c1d-3f87232d46fb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-227c93-29890365-94kfm" podUID="18cae0c8-c13a-4ebe-9c1d-3f87232d46fb" Oct 31 04:45:04.196118 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:04.196097 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-401761-29890365-xnh62" event={"ID":"2608ed82-e75d-41bf-8710-0faf0ce0ae08","Type":"ContainerStarted","Data":"f2ead6982435cd3381a7eef463556f46dd19ae9cf083f2e6544a906aeeddd287"} Oct 31 04:45:04.198003 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:04.197981 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890365-5gwm4" event={"ID":"8032699a-3ca2-4cca-9366-10006ea03b4d","Type":"ContainerStarted","Data":"e6919354c0289ca277957ca101cf9c37041631a83fb89440ca1af189528ef947"} Oct 31 04:45:04.199790 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:04.199776 4376 scope.go:122] "RemoveContainer" containerID="f24c3d85d2cea9de41f32a0f137918969aba98789c853fa50a1d52eb859ca7e2" Oct 31 04:45:04.199964 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:45:04.199948 4376 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-c9707f-29890365-fp5bb_odh-ai-gateway-infra(d5059d94-428a-4df5-99c8-4b2e4f3f87ab)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c9707f-29890365-fp5bb" podUID="d5059d94-428a-4df5-99c8-4b2e4f3f87ab" Oct 31 04:45:05.208737 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:05.208699 4376 scope.go:122] "RemoveContainer" containerID="7321bff73d704d8f9605d175345e161ba936a51d7c4a50414f78d3ee07700134" Oct 31 04:45:05.209172 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:45:05.209057 4376 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-227c93-29890365-94kfm_odh-ai-gateway-infra(18cae0c8-c13a-4ebe-9c1d-3f87232d46fb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-227c93-29890365-94kfm" podUID="18cae0c8-c13a-4ebe-9c1d-3f87232d46fb" Oct 31 04:45:05.210283 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:05.210258 4376 generic.go:356] "Generic (PLEG): container finished" podID="2608ed82-e75d-41bf-8710-0faf0ce0ae08" containerID="f2ead6982435cd3381a7eef463556f46dd19ae9cf083f2e6544a906aeeddd287" exitCode=60 Oct 31 04:45:05.210381 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:05.210352 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-401761-29890365-xnh62" event={"ID":"2608ed82-e75d-41bf-8710-0faf0ce0ae08","Type":"ContainerDied","Data":"f2ead6982435cd3381a7eef463556f46dd19ae9cf083f2e6544a906aeeddd287"} Oct 31 04:45:05.210451 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:05.210410 4376 scope.go:122] "RemoveContainer" containerID="45372ebb91a16fe4020d7adac9f0b7760f1f87346462e787df9a71dcc22e0c45" Oct 31 04:45:05.210613 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:05.210596 4376 scope.go:122] "RemoveContainer" containerID="f2ead6982435cd3381a7eef463556f46dd19ae9cf083f2e6544a906aeeddd287" Oct 31 04:45:05.210941 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:45:05.210920 4376 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-401761-29890365-xnh62_odh-ai-gateway-infra(2608ed82-e75d-41bf-8710-0faf0ce0ae08)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-401761-29890365-xnh62" podUID="2608ed82-e75d-41bf-8710-0faf0ce0ae08" Oct 31 04:45:05.212258 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:05.212238 4376 generic.go:356] "Generic (PLEG): container finished" podID="8032699a-3ca2-4cca-9366-10006ea03b4d" containerID="e6919354c0289ca277957ca101cf9c37041631a83fb89440ca1af189528ef947" exitCode=60 Oct 31 04:45:05.212353 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:05.212316 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890365-5gwm4" event={"ID":"8032699a-3ca2-4cca-9366-10006ea03b4d","Type":"ContainerDied","Data":"e6919354c0289ca277957ca101cf9c37041631a83fb89440ca1af189528ef947"} Oct 31 04:45:05.212475 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:05.212461 4376 scope.go:122] "RemoveContainer" containerID="e6919354c0289ca277957ca101cf9c37041631a83fb89440ca1af189528ef947" Oct 31 04:45:05.212659 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:45:05.212643 4376 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890365-5gwm4_odh-ai-gateway-infra(8032699a-3ca2-4cca-9366-10006ea03b4d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890365-5gwm4" podUID="8032699a-3ca2-4cca-9366-10006ea03b4d" Oct 31 04:45:05.224133 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:05.224116 4376 scope.go:122] "RemoveContainer" containerID="5a8a113ad9d1858f63ebbe71289b41ebf42b8e0e84b6379d48171799d628bc83" Oct 31 04:45:06.222381 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:06.222354 4376 scope.go:122] "RemoveContainer" containerID="f2ead6982435cd3381a7eef463556f46dd19ae9cf083f2e6544a906aeeddd287" Oct 31 04:45:06.222831 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:45:06.222770 4376 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-401761-29890365-xnh62_odh-ai-gateway-infra(2608ed82-e75d-41bf-8710-0faf0ce0ae08)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-401761-29890365-xnh62" podUID="2608ed82-e75d-41bf-8710-0faf0ce0ae08" Oct 31 04:45:06.223991 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:06.223968 4376 scope.go:122] "RemoveContainer" containerID="e6919354c0289ca277957ca101cf9c37041631a83fb89440ca1af189528ef947" Oct 31 04:45:06.224286 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:45:06.224266 4376 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890365-5gwm4_odh-ai-gateway-infra(8032699a-3ca2-4cca-9366-10006ea03b4d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890365-5gwm4" podUID="8032699a-3ca2-4cca-9366-10006ea03b4d" Oct 31 04:45:07.727265 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:07.727231 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-7bc522j9n"] Oct 31 04:45:07.733937 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:07.733919 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-7bc522j9n" Oct 31 04:45:07.755698 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:07.755657 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-7bc522j9n"] Oct 31 04:45:07.758617 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:07.758594 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-a99f2a\"/\"agc-cred-7194275e28ab\"" Oct 31 04:45:07.758857 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:07.758842 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-a99f2a\"/\"openshift-service-ca.crt\"" Oct 31 04:45:07.759073 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:07.759059 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-a99f2a\"/\"routing-overlay\"" Oct 31 04:45:07.759268 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:07.759254 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-a99f2a\"/\"kube-root-ca.crt\"" Oct 31 04:45:07.762058 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:07.762032 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-a99f2a\"/\"payload-processing-external-model-plugins-e2e-extmod-b-a99f2a\"" Oct 31 04:45:07.762181 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:07.762063 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-a99f2a\"/\"payload-processing-external-model-e2e--c07b99ee-dockercfg-gzb75\"" Oct 31 04:45:07.883540 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:07.883510 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5e17d6fa-35eb-44f6-8a4a-bd1bd937a7b2-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a99f2a-7bc522j9n\" (UID: \"5e17d6fa-35eb-44f6-8a4a-bd1bd937a7b2\") " pod="ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-7bc522j9n" Oct 31 04:45:07.883707 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:07.883671 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d862r\" (UniqueName: \"kubernetes.io/projected/5e17d6fa-35eb-44f6-8a4a-bd1bd937a7b2-kube-api-access-d862r\") pod \"payload-processing-external-model-e2e-extmod-b-a99f2a-7bc522j9n\" (UID: \"5e17d6fa-35eb-44f6-8a4a-bd1bd937a7b2\") " pod="ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-7bc522j9n" Oct 31 04:45:07.883758 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:07.883712 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5e17d6fa-35eb-44f6-8a4a-bd1bd937a7b2-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-a99f2a-7bc522j9n\" (UID: \"5e17d6fa-35eb-44f6-8a4a-bd1bd937a7b2\") " pod="ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-7bc522j9n" Oct 31 04:45:07.883888 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:07.883865 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e17d6fa-35eb-44f6-8a4a-bd1bd937a7b2-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a99f2a-7bc522j9n\" (UID: \"5e17d6fa-35eb-44f6-8a4a-bd1bd937a7b2\") " pod="ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-7bc522j9n" Oct 31 04:45:07.985267 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:07.985193 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d862r\" (UniqueName: \"kubernetes.io/projected/5e17d6fa-35eb-44f6-8a4a-bd1bd937a7b2-kube-api-access-d862r\") pod \"payload-processing-external-model-e2e-extmod-b-a99f2a-7bc522j9n\" (UID: \"5e17d6fa-35eb-44f6-8a4a-bd1bd937a7b2\") " pod="ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-7bc522j9n" Oct 31 04:45:07.985267 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:07.985252 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5e17d6fa-35eb-44f6-8a4a-bd1bd937a7b2-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-a99f2a-7bc522j9n\" (UID: \"5e17d6fa-35eb-44f6-8a4a-bd1bd937a7b2\") " pod="ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-7bc522j9n" Oct 31 04:45:07.985508 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:07.985386 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e17d6fa-35eb-44f6-8a4a-bd1bd937a7b2-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a99f2a-7bc522j9n\" (UID: \"5e17d6fa-35eb-44f6-8a4a-bd1bd937a7b2\") " pod="ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-7bc522j9n" Oct 31 04:45:07.985508 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:07.985477 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5e17d6fa-35eb-44f6-8a4a-bd1bd937a7b2-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a99f2a-7bc522j9n\" (UID: \"5e17d6fa-35eb-44f6-8a4a-bd1bd937a7b2\") " pod="ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-7bc522j9n" Oct 31 04:45:07.986062 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:07.986029 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5e17d6fa-35eb-44f6-8a4a-bd1bd937a7b2-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a99f2a-7bc522j9n\" (UID: \"5e17d6fa-35eb-44f6-8a4a-bd1bd937a7b2\") " pod="ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-7bc522j9n" Oct 31 04:45:07.986062 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:07.986033 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e17d6fa-35eb-44f6-8a4a-bd1bd937a7b2-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a99f2a-7bc522j9n\" (UID: \"5e17d6fa-35eb-44f6-8a4a-bd1bd937a7b2\") " pod="ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-7bc522j9n" Oct 31 04:45:07.987614 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:07.987592 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5e17d6fa-35eb-44f6-8a4a-bd1bd937a7b2-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-a99f2a-7bc522j9n\" (UID: \"5e17d6fa-35eb-44f6-8a4a-bd1bd937a7b2\") " pod="ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-7bc522j9n" Oct 31 04:45:08.010977 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:08.010954 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d862r\" (UniqueName: \"kubernetes.io/projected/5e17d6fa-35eb-44f6-8a4a-bd1bd937a7b2-kube-api-access-d862r\") pod \"payload-processing-external-model-e2e-extmod-b-a99f2a-7bc522j9n\" (UID: \"5e17d6fa-35eb-44f6-8a4a-bd1bd937a7b2\") " pod="ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-7bc522j9n" Oct 31 04:45:08.049365 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:08.049340 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-7bc522j9n" Oct 31 04:45:08.350053 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:45:08.350018 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e17d6fa_35eb_44f6_8a4a_bd1bd937a7b2.slice/crio-d909450a82c07a7a314bce9a1e47b9f792df5d277a772ea7754a780c4eeaef35 WatchSource:0}: Error finding container d909450a82c07a7a314bce9a1e47b9f792df5d277a772ea7754a780c4eeaef35: Status 404 returned error can't find the container with id d909450a82c07a7a314bce9a1e47b9f792df5d277a772ea7754a780c4eeaef35 Oct 31 04:45:08.352277 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:08.352256 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-7bc522j9n"] Oct 31 04:45:09.262860 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:09.262820 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-7bc522j9n" event={"ID":"5e17d6fa-35eb-44f6-8a4a-bd1bd937a7b2","Type":"ContainerStarted","Data":"2e96e77a0cf6d88cca591a349a522fedf6c9f0777b91505642079b6ae79d6684"} Oct 31 04:45:09.263290 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:09.262879 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-7bc522j9n" event={"ID":"5e17d6fa-35eb-44f6-8a4a-bd1bd937a7b2","Type":"ContainerStarted","Data":"d909450a82c07a7a314bce9a1e47b9f792df5d277a772ea7754a780c4eeaef35"} Oct 31 04:45:09.263290 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:09.262916 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-7bc522j9n" Oct 31 04:45:09.307841 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:09.307801 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-7bc522j9n" podStartSLOduration=2.30778953 podStartE2EDuration="2.30778953s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:45:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:45:09.305370041 +0000 UTC m=+2457.019704653" watchObservedRunningTime="2026-10-31 04:45:09.30778953 +0000 UTC m=+2457.022124146" Oct 31 04:45:10.286397 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:10.286364 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b29b642e-cn6jt"] Oct 31 04:45:10.291064 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:10.291043 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b29b642e-cn6jt" Oct 31 04:45:10.310432 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:10.310401 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b29b642e-cn6jt"] Oct 31 04:45:10.318299 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:10.318273 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 04:45:10.418830 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:10.418797 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4dcd3fe-2bd0-43c9-8e61-12003a99f3c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b29b642e-cn6jt\" (UID: \"f4dcd3fe-2bd0-43c9-8e61-12003a99f3c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b29b642e-cn6jt" Oct 31 04:45:10.419066 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:10.419046 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4dcd3fe-2bd0-43c9-8e61-12003a99f3c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b29b642e-cn6jt\" (UID: \"f4dcd3fe-2bd0-43c9-8e61-12003a99f3c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b29b642e-cn6jt" Oct 31 04:45:10.419344 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:10.419312 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lngw\" (UniqueName: \"kubernetes.io/projected/f4dcd3fe-2bd0-43c9-8e61-12003a99f3c8-kube-api-access-6lngw\") pod \"maas-api-revoke-sub-b29b642e-cn6jt\" (UID: \"f4dcd3fe-2bd0-43c9-8e61-12003a99f3c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b29b642e-cn6jt" Oct 31 04:45:10.520386 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:10.520351 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4dcd3fe-2bd0-43c9-8e61-12003a99f3c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b29b642e-cn6jt\" (UID: \"f4dcd3fe-2bd0-43c9-8e61-12003a99f3c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b29b642e-cn6jt" Oct 31 04:45:10.520710 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:10.520465 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lngw\" (UniqueName: \"kubernetes.io/projected/f4dcd3fe-2bd0-43c9-8e61-12003a99f3c8-kube-api-access-6lngw\") pod \"maas-api-revoke-sub-b29b642e-cn6jt\" (UID: \"f4dcd3fe-2bd0-43c9-8e61-12003a99f3c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b29b642e-cn6jt" Oct 31 04:45:10.520710 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:10.520585 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4dcd3fe-2bd0-43c9-8e61-12003a99f3c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b29b642e-cn6jt\" (UID: \"f4dcd3fe-2bd0-43c9-8e61-12003a99f3c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b29b642e-cn6jt" Oct 31 04:45:10.521200 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:10.521175 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4dcd3fe-2bd0-43c9-8e61-12003a99f3c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b29b642e-cn6jt\" (UID: \"f4dcd3fe-2bd0-43c9-8e61-12003a99f3c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b29b642e-cn6jt" Oct 31 04:45:10.522653 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:10.522635 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4dcd3fe-2bd0-43c9-8e61-12003a99f3c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b29b642e-cn6jt\" (UID: \"f4dcd3fe-2bd0-43c9-8e61-12003a99f3c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b29b642e-cn6jt" Oct 31 04:45:10.544427 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:10.544375 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lngw\" (UniqueName: \"kubernetes.io/projected/f4dcd3fe-2bd0-43c9-8e61-12003a99f3c8-kube-api-access-6lngw\") pod \"maas-api-revoke-sub-b29b642e-cn6jt\" (UID: \"f4dcd3fe-2bd0-43c9-8e61-12003a99f3c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b29b642e-cn6jt" Oct 31 04:45:10.606241 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:10.606216 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b29b642e-cn6jt" Oct 31 04:45:10.906372 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:45:10.906341 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4dcd3fe_2bd0_43c9_8e61_12003a99f3c8.slice/crio-73b6cc78e63ac8d8a85e5d52f4c6c25b278111b26bb679bdd3ee092d270ff5a8 WatchSource:0}: Error finding container 73b6cc78e63ac8d8a85e5d52f4c6c25b278111b26bb679bdd3ee092d270ff5a8: Status 404 returned error can't find the container with id 73b6cc78e63ac8d8a85e5d52f4c6c25b278111b26bb679bdd3ee092d270ff5a8 Oct 31 04:45:10.910968 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:10.910945 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b29b642e-cn6jt"] Oct 31 04:45:11.001346 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:45:11.001296 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 04:45:11.279179 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:11.279146 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b29b642e-cn6jt" event={"ID":"f4dcd3fe-2bd0-43c9-8e61-12003a99f3c8","Type":"ContainerStarted","Data":"73b6cc78e63ac8d8a85e5d52f4c6c25b278111b26bb679bdd3ee092d270ff5a8"} Oct 31 04:45:12.287323 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:12.287236 4376 generic.go:356] "Generic (PLEG): container finished" podID="f4dcd3fe-2bd0-43c9-8e61-12003a99f3c8" containerID="4ff405c0a49e86d320cc0c0013ec6ebaff403a669e12eccefd3a06772019652e" exitCode=0 Oct 31 04:45:12.287689 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:12.287318 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b29b642e-cn6jt" event={"ID":"f4dcd3fe-2bd0-43c9-8e61-12003a99f3c8","Type":"ContainerDied","Data":"4ff405c0a49e86d320cc0c0013ec6ebaff403a669e12eccefd3a06772019652e"} Oct 31 04:45:13.443177 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:13.443153 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b29b642e-cn6jt" Oct 31 04:45:13.463609 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:13.463581 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4dcd3fe-2bd0-43c9-8e61-12003a99f3c8-kube-api-access-6lngw\" (UniqueName: \"kubernetes.io/projected/f4dcd3fe-2bd0-43c9-8e61-12003a99f3c8-kube-api-access-6lngw\") pod \"f4dcd3fe-2bd0-43c9-8e61-12003a99f3c8\" (UID: \"f4dcd3fe-2bd0-43c9-8e61-12003a99f3c8\") " Oct 31 04:45:13.463752 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:13.463740 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f4dcd3fe-2bd0-43c9-8e61-12003a99f3c8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4dcd3fe-2bd0-43c9-8e61-12003a99f3c8-maas-api-xks-ca\") pod \"f4dcd3fe-2bd0-43c9-8e61-12003a99f3c8\" (UID: \"f4dcd3fe-2bd0-43c9-8e61-12003a99f3c8\") " Oct 31 04:45:13.463819 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:13.463806 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4dcd3fe-2bd0-43c9-8e61-12003a99f3c8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4dcd3fe-2bd0-43c9-8e61-12003a99f3c8-maas-api-openshift-ca\") pod \"f4dcd3fe-2bd0-43c9-8e61-12003a99f3c8\" (UID: \"f4dcd3fe-2bd0-43c9-8e61-12003a99f3c8\") " Oct 31 04:45:13.464160 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:13.464134 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4dcd3fe-2bd0-43c9-8e61-12003a99f3c8-maas-api-openshift-ca" pod "f4dcd3fe-2bd0-43c9-8e61-12003a99f3c8" (UID: "f4dcd3fe-2bd0-43c9-8e61-12003a99f3c8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:45:13.464300 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:13.464225 4376 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4dcd3fe-2bd0-43c9-8e61-12003a99f3c8-maas-api-openshift-ca\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:45:13.465693 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:13.465664 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4dcd3fe-2bd0-43c9-8e61-12003a99f3c8-kube-api-access-6lngw" pod "f4dcd3fe-2bd0-43c9-8e61-12003a99f3c8" (UID: "f4dcd3fe-2bd0-43c9-8e61-12003a99f3c8"). InnerVolumeSpecName "kube-api-access-6lngw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:45:13.465693 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:13.465680 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4dcd3fe-2bd0-43c9-8e61-12003a99f3c8-maas-api-xks-ca" pod "f4dcd3fe-2bd0-43c9-8e61-12003a99f3c8" (UID: "f4dcd3fe-2bd0-43c9-8e61-12003a99f3c8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:45:13.565510 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:13.565436 4376 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4dcd3fe-2bd0-43c9-8e61-12003a99f3c8-maas-api-xks-ca\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:45:13.565510 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:13.565460 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6lngw\" (UniqueName: \"kubernetes.io/projected/f4dcd3fe-2bd0-43c9-8e61-12003a99f3c8-kube-api-access-6lngw\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:45:14.203160 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:14.203127 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-55cfq6jfg" Oct 31 04:45:14.305177 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:14.305130 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b29b642e-cn6jt" event={"ID":"f4dcd3fe-2bd0-43c9-8e61-12003a99f3c8","Type":"ContainerDied","Data":"73b6cc78e63ac8d8a85e5d52f4c6c25b278111b26bb679bdd3ee092d270ff5a8"} Oct 31 04:45:14.305177 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:14.305174 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b29b642e-cn6jt" Oct 31 04:45:14.305496 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:14.305181 4376 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="73b6cc78e63ac8d8a85e5d52f4c6c25b278111b26bb679bdd3ee092d270ff5a8" Oct 31 04:45:14.437036 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:14.436995 4376 scope.go:122] "RemoveContainer" containerID="7321bff73d704d8f9605d175345e161ba936a51d7c4a50414f78d3ee07700134" Oct 31 04:45:14.437233 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:14.437170 4376 scope.go:122] "RemoveContainer" containerID="f24c3d85d2cea9de41f32a0f137918969aba98789c853fa50a1d52eb859ca7e2" Oct 31 04:45:15.313965 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:15.313938 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-227c93-29890365-94kfm" event={"ID":"18cae0c8-c13a-4ebe-9c1d-3f87232d46fb","Type":"ContainerStarted","Data":"5a198476a23989a0dd41c8a75fb7049fdf919eb72c426b260c6cdcdd5012b486"} Oct 31 04:45:15.315996 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:15.315973 4376 generic.go:356] "Generic (PLEG): container finished" podID="d5059d94-428a-4df5-99c8-4b2e4f3f87ab" containerID="3d7b0ed50b65ed6f3f5742c448792dfaf3b614963dc4f1b86ab2b2e528fa1b90" exitCode=60 Oct 31 04:45:15.316118 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:15.316025 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c9707f-29890365-fp5bb" event={"ID":"d5059d94-428a-4df5-99c8-4b2e4f3f87ab","Type":"ContainerDied","Data":"3d7b0ed50b65ed6f3f5742c448792dfaf3b614963dc4f1b86ab2b2e528fa1b90"} Oct 31 04:45:15.316118 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:15.316071 4376 scope.go:122] "RemoveContainer" containerID="f24c3d85d2cea9de41f32a0f137918969aba98789c853fa50a1d52eb859ca7e2" Oct 31 04:45:15.316416 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:15.316399 4376 scope.go:122] "RemoveContainer" containerID="3d7b0ed50b65ed6f3f5742c448792dfaf3b614963dc4f1b86ab2b2e528fa1b90" Oct 31 04:45:15.316738 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:45:15.316710 4376 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-c9707f-29890365-fp5bb_odh-ai-gateway-infra(d5059d94-428a-4df5-99c8-4b2e4f3f87ab)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c9707f-29890365-fp5bb" podUID="d5059d94-428a-4df5-99c8-4b2e4f3f87ab" Oct 31 04:45:16.346355 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:16.343137 4376 generic.go:356] "Generic (PLEG): container finished" podID="18cae0c8-c13a-4ebe-9c1d-3f87232d46fb" containerID="5a198476a23989a0dd41c8a75fb7049fdf919eb72c426b260c6cdcdd5012b486" exitCode=60 Oct 31 04:45:16.346355 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:16.343377 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-227c93-29890365-94kfm" event={"ID":"18cae0c8-c13a-4ebe-9c1d-3f87232d46fb","Type":"ContainerDied","Data":"5a198476a23989a0dd41c8a75fb7049fdf919eb72c426b260c6cdcdd5012b486"} Oct 31 04:45:16.346355 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:16.343471 4376 scope.go:122] "RemoveContainer" containerID="7321bff73d704d8f9605d175345e161ba936a51d7c4a50414f78d3ee07700134" Oct 31 04:45:16.346355 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:16.343994 4376 scope.go:122] "RemoveContainer" containerID="5a198476a23989a0dd41c8a75fb7049fdf919eb72c426b260c6cdcdd5012b486" Oct 31 04:45:16.346355 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:45:16.344424 4376 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-227c93-29890365-94kfm_odh-ai-gateway-infra(18cae0c8-c13a-4ebe-9c1d-3f87232d46fb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-227c93-29890365-94kfm" podUID="18cae0c8-c13a-4ebe-9c1d-3f87232d46fb" Oct 31 04:45:16.413187 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:16.408661 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-227c93-29890365-94kfm"] Oct 31 04:45:16.431382 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:16.431351 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c9707f-29890365-fp5bb"] Oct 31 04:45:16.436205 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:16.436179 4376 scope.go:122] "RemoveContainer" containerID="e6919354c0289ca277957ca101cf9c37041631a83fb89440ca1af189528ef947" Oct 31 04:45:16.688975 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:16.688747 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c9707f-29890365-fp5bb" Oct 31 04:45:16.716475 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:16.716450 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5059d94-428a-4df5-99c8-4b2e4f3f87ab-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5059d94-428a-4df5-99c8-4b2e4f3f87ab-maas-api-service-ca\") pod \"d5059d94-428a-4df5-99c8-4b2e4f3f87ab\" (UID: \"d5059d94-428a-4df5-99c8-4b2e4f3f87ab\") " Oct 31 04:45:16.716625 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:16.716535 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5059d94-428a-4df5-99c8-4b2e4f3f87ab-kube-api-access-tdn94\" (UniqueName: \"kubernetes.io/projected/d5059d94-428a-4df5-99c8-4b2e4f3f87ab-kube-api-access-tdn94\") pod \"d5059d94-428a-4df5-99c8-4b2e4f3f87ab\" (UID: \"d5059d94-428a-4df5-99c8-4b2e4f3f87ab\") " Oct 31 04:45:16.716878 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:16.716855 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5059d94-428a-4df5-99c8-4b2e4f3f87ab-maas-api-service-ca" pod "d5059d94-428a-4df5-99c8-4b2e4f3f87ab" (UID: "d5059d94-428a-4df5-99c8-4b2e4f3f87ab"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:45:16.717134 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:16.717118 4376 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5059d94-428a-4df5-99c8-4b2e4f3f87ab-maas-api-service-ca\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:45:16.718597 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:16.718575 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5059d94-428a-4df5-99c8-4b2e4f3f87ab-kube-api-access-tdn94" pod "d5059d94-428a-4df5-99c8-4b2e4f3f87ab" (UID: "d5059d94-428a-4df5-99c8-4b2e4f3f87ab"). InnerVolumeSpecName "kube-api-access-tdn94". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:45:16.818708 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:16.818683 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdn94\" (UniqueName: \"kubernetes.io/projected/d5059d94-428a-4df5-99c8-4b2e4f3f87ab-kube-api-access-tdn94\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:45:17.379154 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:17.379129 4376 generic.go:356] "Generic (PLEG): container finished" podID="8032699a-3ca2-4cca-9366-10006ea03b4d" containerID="1e86211c01d01a5dcc5daf94582670750b145655f6ec100feee13b6d2c609432" exitCode=60 Oct 31 04:45:17.379549 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:17.379203 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890365-5gwm4" event={"ID":"8032699a-3ca2-4cca-9366-10006ea03b4d","Type":"ContainerDied","Data":"1e86211c01d01a5dcc5daf94582670750b145655f6ec100feee13b6d2c609432"} Oct 31 04:45:17.379549 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:17.379265 4376 scope.go:122] "RemoveContainer" containerID="e6919354c0289ca277957ca101cf9c37041631a83fb89440ca1af189528ef947" Oct 31 04:45:17.379677 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:17.379662 4376 scope.go:122] "RemoveContainer" containerID="1e86211c01d01a5dcc5daf94582670750b145655f6ec100feee13b6d2c609432" Oct 31 04:45:17.380028 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:45:17.380006 4376 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29890365-5gwm4_odh-ai-gateway-infra(8032699a-3ca2-4cca-9366-10006ea03b4d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890365-5gwm4" podUID="8032699a-3ca2-4cca-9366-10006ea03b4d" Oct 31 04:45:17.380903 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:17.380887 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c9707f-29890365-fp5bb" Oct 31 04:45:17.380980 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:17.380885 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c9707f-29890365-fp5bb" event={"ID":"d5059d94-428a-4df5-99c8-4b2e4f3f87ab","Type":"ContainerDied","Data":"bad5169f219c40fb0261a489d9e47d242f295aab6b9c92f82cd0c19a8f5722c2"} Oct 31 04:45:17.414015 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:17.413982 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-7bc522j9n"] Oct 31 04:45:17.414650 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:17.414551 4376 scope.go:122] "RemoveContainer" containerID="3d7b0ed50b65ed6f3f5742c448792dfaf3b614963dc4f1b86ab2b2e528fa1b90" Oct 31 04:45:17.417202 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:17.417182 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-7bc522j9n" Oct 31 04:45:17.436155 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:17.436125 4376 scope.go:122] "RemoveContainer" containerID="c08f3a719a9de100d5cc2ab560c363e95748626d54c51e265e8ab5c39db3ac1c" Oct 31 04:45:17.436273 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:17.436187 4376 scope.go:122] "RemoveContainer" containerID="f2ead6982435cd3381a7eef463556f46dd19ae9cf083f2e6544a906aeeddd287" Oct 31 04:45:17.471382 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:17.449345 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-b679dtfm4"] Oct 31 04:45:17.471382 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:17.450492 4376 cpu_manager.go:414] "containerMap: removing container" podUID="f4dcd3fe-2bd0-43c9-8e61-12003a99f3c8" containerName="revoke-keys" Oct 31 04:45:17.471382 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:17.450510 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4dcd3fe-2bd0-43c9-8e61-12003a99f3c8" containerName="revoke-keys" Oct 31 04:45:17.471382 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:17.450544 4376 cpu_manager.go:414] "containerMap: removing container" podUID="d5059d94-428a-4df5-99c8-4b2e4f3f87ab" containerName="cleanup" Oct 31 04:45:17.471382 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:17.450552 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5059d94-428a-4df5-99c8-4b2e4f3f87ab" containerName="cleanup" Oct 31 04:45:17.471382 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:17.450601 4376 cpu_manager.go:414] "containerMap: removing container" podUID="d5059d94-428a-4df5-99c8-4b2e4f3f87ab" containerName="cleanup" Oct 31 04:45:17.471382 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:17.450609 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5059d94-428a-4df5-99c8-4b2e4f3f87ab" containerName="cleanup" Oct 31 04:45:17.471382 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:17.450770 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5059d94-428a-4df5-99c8-4b2e4f3f87ab" containerName="cleanup" Oct 31 04:45:17.471382 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:17.450782 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5059d94-428a-4df5-99c8-4b2e4f3f87ab" containerName="cleanup" Oct 31 04:45:17.471382 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:17.450794 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4dcd3fe-2bd0-43c9-8e61-12003a99f3c8" containerName="revoke-keys" Oct 31 04:45:17.471382 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:17.450809 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5059d94-428a-4df5-99c8-4b2e4f3f87ab" containerName="cleanup" Oct 31 04:45:17.530054 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:17.530019 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-b679dtfm4"] Oct 31 04:45:17.530186 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:17.530141 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-b679dtfm4" Oct 31 04:45:17.547357 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:17.546195 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c9707f-29890365-fp5bb"] Oct 31 04:45:17.551708 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:17.551683 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c9707f-29890365-fp5bb"] Oct 31 04:45:17.551847 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:17.551833 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-a99f2a\"/\"agc-cred-4c8aa1529e84\"" Oct 31 04:45:17.631041 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:17.631009 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1110d483-5331-4844-8cff-c42f3b132678-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a99f2a-b679dtfm4\" (UID: \"1110d483-5331-4844-8cff-c42f3b132678\") " pod="ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-b679dtfm4" Oct 31 04:45:17.631170 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:17.631064 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqkjb\" (UniqueName: \"kubernetes.io/projected/1110d483-5331-4844-8cff-c42f3b132678-kube-api-access-xqkjb\") pod \"payload-processing-external-model-e2e-extmod-b-a99f2a-b679dtfm4\" (UID: \"1110d483-5331-4844-8cff-c42f3b132678\") " pod="ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-b679dtfm4" Oct 31 04:45:17.631170 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:17.631153 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1110d483-5331-4844-8cff-c42f3b132678-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-a99f2a-b679dtfm4\" (UID: \"1110d483-5331-4844-8cff-c42f3b132678\") " pod="ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-b679dtfm4" Oct 31 04:45:17.631324 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:17.631311 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1110d483-5331-4844-8cff-c42f3b132678-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a99f2a-b679dtfm4\" (UID: \"1110d483-5331-4844-8cff-c42f3b132678\") " pod="ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-b679dtfm4" Oct 31 04:45:17.683247 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:17.683227 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-227c93-29890365-94kfm" Oct 31 04:45:17.734841 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:17.734809 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18cae0c8-c13a-4ebe-9c1d-3f87232d46fb-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/18cae0c8-c13a-4ebe-9c1d-3f87232d46fb-maas-api-service-ca\") pod \"18cae0c8-c13a-4ebe-9c1d-3f87232d46fb\" (UID: \"18cae0c8-c13a-4ebe-9c1d-3f87232d46fb\") " Oct 31 04:45:17.735016 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:17.734916 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18cae0c8-c13a-4ebe-9c1d-3f87232d46fb-kube-api-access-rbtxk\" (UniqueName: \"kubernetes.io/projected/18cae0c8-c13a-4ebe-9c1d-3f87232d46fb-kube-api-access-rbtxk\") pod \"18cae0c8-c13a-4ebe-9c1d-3f87232d46fb\" (UID: \"18cae0c8-c13a-4ebe-9c1d-3f87232d46fb\") " Oct 31 04:45:17.735771 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:17.735387 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1110d483-5331-4844-8cff-c42f3b132678-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a99f2a-b679dtfm4\" (UID: \"1110d483-5331-4844-8cff-c42f3b132678\") " pod="ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-b679dtfm4" Oct 31 04:45:17.735771 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:17.735451 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqkjb\" (UniqueName: \"kubernetes.io/projected/1110d483-5331-4844-8cff-c42f3b132678-kube-api-access-xqkjb\") pod \"payload-processing-external-model-e2e-extmod-b-a99f2a-b679dtfm4\" (UID: \"1110d483-5331-4844-8cff-c42f3b132678\") " pod="ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-b679dtfm4" Oct 31 04:45:17.735771 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:17.735466 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18cae0c8-c13a-4ebe-9c1d-3f87232d46fb-maas-api-service-ca" pod "18cae0c8-c13a-4ebe-9c1d-3f87232d46fb" (UID: "18cae0c8-c13a-4ebe-9c1d-3f87232d46fb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:45:17.735771 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:17.735496 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1110d483-5331-4844-8cff-c42f3b132678-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-a99f2a-b679dtfm4\" (UID: \"1110d483-5331-4844-8cff-c42f3b132678\") " pod="ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-b679dtfm4" Oct 31 04:45:17.735771 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:17.735714 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1110d483-5331-4844-8cff-c42f3b132678-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a99f2a-b679dtfm4\" (UID: \"1110d483-5331-4844-8cff-c42f3b132678\") " pod="ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-b679dtfm4" Oct 31 04:45:17.736099 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:17.735845 4376 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/18cae0c8-c13a-4ebe-9c1d-3f87232d46fb-maas-api-service-ca\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:45:17.736906 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:17.736853 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1110d483-5331-4844-8cff-c42f3b132678-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a99f2a-b679dtfm4\" (UID: \"1110d483-5331-4844-8cff-c42f3b132678\") " pod="ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-b679dtfm4" Oct 31 04:45:17.739116 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:17.737847 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1110d483-5331-4844-8cff-c42f3b132678-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a99f2a-b679dtfm4\" (UID: \"1110d483-5331-4844-8cff-c42f3b132678\") " pod="ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-b679dtfm4" Oct 31 04:45:17.741626 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:17.741601 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18cae0c8-c13a-4ebe-9c1d-3f87232d46fb-kube-api-access-rbtxk" pod "18cae0c8-c13a-4ebe-9c1d-3f87232d46fb" (UID: "18cae0c8-c13a-4ebe-9c1d-3f87232d46fb"). InnerVolumeSpecName "kube-api-access-rbtxk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:45:17.742036 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:17.742015 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1110d483-5331-4844-8cff-c42f3b132678-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-a99f2a-b679dtfm4\" (UID: \"1110d483-5331-4844-8cff-c42f3b132678\") " pod="ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-b679dtfm4" Oct 31 04:45:17.759919 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:17.759900 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqkjb\" (UniqueName: \"kubernetes.io/projected/1110d483-5331-4844-8cff-c42f3b132678-kube-api-access-xqkjb\") pod \"payload-processing-external-model-e2e-extmod-b-a99f2a-b679dtfm4\" (UID: \"1110d483-5331-4844-8cff-c42f3b132678\") " pod="ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-b679dtfm4" Oct 31 04:45:17.836945 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:17.836916 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbtxk\" (UniqueName: \"kubernetes.io/projected/18cae0c8-c13a-4ebe-9c1d-3f87232d46fb-kube-api-access-rbtxk\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:45:17.870144 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:17.870122 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-b679dtfm4" Oct 31 04:45:18.173076 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:45:18.173044 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1110d483_5331_4844_8cff_c42f3b132678.slice/crio-f2f6bc399d5795f0b6334dd726513541f72a3a3b443b7704b12dba28a09ace1d WatchSource:0}: Error finding container f2f6bc399d5795f0b6334dd726513541f72a3a3b443b7704b12dba28a09ace1d: Status 404 returned error can't find the container with id f2f6bc399d5795f0b6334dd726513541f72a3a3b443b7704b12dba28a09ace1d Oct 31 04:45:18.175153 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:18.175131 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-b679dtfm4"] Oct 31 04:45:18.391384 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:18.391346 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-401761-29890365-xnh62" event={"ID":"2608ed82-e75d-41bf-8710-0faf0ce0ae08","Type":"ContainerStarted","Data":"485d8fb311649ba4b39e0e093be9f5a0dffca4bd33a1080ab5c58c3e05f2797c"} Oct 31 04:45:18.393026 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:18.392994 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-b679dtfm4" event={"ID":"1110d483-5331-4844-8cff-c42f3b132678","Type":"ContainerStarted","Data":"5161f219e78d8ccf736ecb8ef9538e840310aee402b5f3652fcdfe1b8f84769b"} Oct 31 04:45:18.393149 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:18.393035 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-b679dtfm4" event={"ID":"1110d483-5331-4844-8cff-c42f3b132678","Type":"ContainerStarted","Data":"f2f6bc399d5795f0b6334dd726513541f72a3a3b443b7704b12dba28a09ace1d"} Oct 31 04:45:18.393149 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:18.393063 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-b679dtfm4" Oct 31 04:45:18.397378 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:18.397354 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-47c409-29890365-vsw27" event={"ID":"fbee2356-f532-40d7-90b6-2572284f0b55","Type":"ContainerStarted","Data":"b017bec8a789e386d07af2e23e2452d08a11c3e5acd0d101eadbc80a872dfbf7"} Oct 31 04:45:18.398556 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:18.398537 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-227c93-29890365-94kfm" event={"ID":"18cae0c8-c13a-4ebe-9c1d-3f87232d46fb","Type":"ContainerDied","Data":"c53de2aeba74e0630dfb014bb76f0d5bb717420d9b3398efd726b1dc99e4fc8c"} Oct 31 04:45:18.398653 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:18.398577 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-227c93-29890365-94kfm" Oct 31 04:45:18.398653 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:18.398588 4376 scope.go:122] "RemoveContainer" containerID="5a198476a23989a0dd41c8a75fb7049fdf919eb72c426b260c6cdcdd5012b486" Oct 31 04:45:18.447065 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:18.446991 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5059d94-428a-4df5-99c8-4b2e4f3f87ab" path="/var/lib/kubelet/pods/d5059d94-428a-4df5-99c8-4b2e4f3f87ab/volumes" Oct 31 04:45:18.477406 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:18.477354 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-b679dtfm4" podStartSLOduration=1.477323428 podStartE2EDuration="1.477323428s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:45:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:45:18.474419316 +0000 UTC m=+2466.188753953" watchObservedRunningTime="2026-10-31 04:45:18.477323428 +0000 UTC m=+2466.191658042" Oct 31 04:45:18.512609 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:18.512582 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-227c93-29890365-94kfm"] Oct 31 04:45:18.519056 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:18.519032 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-227c93-29890365-94kfm"] Oct 31 04:45:18.556109 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:18.556063 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-47c409-29890365-vsw27" podStartSLOduration=18.556050694 podStartE2EDuration="18.556050694s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:45:18.5543452 +0000 UTC m=+2466.268679814" watchObservedRunningTime="2026-10-31 04:45:18.556050694 +0000 UTC m=+2466.270385340" Oct 31 04:45:18.597281 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:18.597254 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890365-5gwm4"] Oct 31 04:45:18.767409 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:18.767377 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890365-5gwm4" Oct 31 04:45:18.858418 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:18.858388 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8032699a-3ca2-4cca-9366-10006ea03b4d-kube-api-access-wjw7m\" (UniqueName: \"kubernetes.io/projected/8032699a-3ca2-4cca-9366-10006ea03b4d-kube-api-access-wjw7m\") pod \"8032699a-3ca2-4cca-9366-10006ea03b4d\" (UID: \"8032699a-3ca2-4cca-9366-10006ea03b4d\") " Oct 31 04:45:18.858633 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:18.858617 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8032699a-3ca2-4cca-9366-10006ea03b4d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8032699a-3ca2-4cca-9366-10006ea03b4d-maas-api-service-ca\") pod \"8032699a-3ca2-4cca-9366-10006ea03b4d\" (UID: \"8032699a-3ca2-4cca-9366-10006ea03b4d\") " Oct 31 04:45:18.859044 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:18.859004 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8032699a-3ca2-4cca-9366-10006ea03b4d-maas-api-service-ca" pod "8032699a-3ca2-4cca-9366-10006ea03b4d" (UID: "8032699a-3ca2-4cca-9366-10006ea03b4d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:45:18.859383 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:18.859364 4376 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8032699a-3ca2-4cca-9366-10006ea03b4d-maas-api-service-ca\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:45:18.860488 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:18.860467 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8032699a-3ca2-4cca-9366-10006ea03b4d-kube-api-access-wjw7m" pod "8032699a-3ca2-4cca-9366-10006ea03b4d" (UID: "8032699a-3ca2-4cca-9366-10006ea03b4d"). InnerVolumeSpecName "kube-api-access-wjw7m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:45:18.961161 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:18.961131 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjw7m\" (UniqueName: \"kubernetes.io/projected/8032699a-3ca2-4cca-9366-10006ea03b4d-kube-api-access-wjw7m\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:45:19.407962 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:19.407930 4376 generic.go:356] "Generic (PLEG): container finished" podID="2608ed82-e75d-41bf-8710-0faf0ce0ae08" containerID="485d8fb311649ba4b39e0e093be9f5a0dffca4bd33a1080ab5c58c3e05f2797c" exitCode=60 Oct 31 04:45:19.408436 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:19.408002 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-401761-29890365-xnh62" event={"ID":"2608ed82-e75d-41bf-8710-0faf0ce0ae08","Type":"ContainerDied","Data":"485d8fb311649ba4b39e0e093be9f5a0dffca4bd33a1080ab5c58c3e05f2797c"} Oct 31 04:45:19.408436 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:19.408064 4376 scope.go:122] "RemoveContainer" containerID="f2ead6982435cd3381a7eef463556f46dd19ae9cf083f2e6544a906aeeddd287" Oct 31 04:45:19.408550 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:19.408471 4376 scope.go:122] "RemoveContainer" containerID="485d8fb311649ba4b39e0e093be9f5a0dffca4bd33a1080ab5c58c3e05f2797c" Oct 31 04:45:19.408875 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:45:19.408851 4376 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-401761-29890365-xnh62_odh-ai-gateway-infra(2608ed82-e75d-41bf-8710-0faf0ce0ae08)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-401761-29890365-xnh62" podUID="2608ed82-e75d-41bf-8710-0faf0ce0ae08" Oct 31 04:45:19.409550 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:19.409535 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890365-5gwm4" Oct 31 04:45:19.409636 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:19.409593 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890365-5gwm4" event={"ID":"8032699a-3ca2-4cca-9366-10006ea03b4d","Type":"ContainerDied","Data":"db5d9824f5541631b4669c1c8a44cf8bd53cfbf452a1e728f849a003f63dd858"} Oct 31 04:45:19.411819 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:19.411798 4376 generic.go:356] "Generic (PLEG): container finished" podID="fbee2356-f532-40d7-90b6-2572284f0b55" containerID="b017bec8a789e386d07af2e23e2452d08a11c3e5acd0d101eadbc80a872dfbf7" exitCode=60 Oct 31 04:45:19.411913 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:19.411867 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-47c409-29890365-vsw27" event={"ID":"fbee2356-f532-40d7-90b6-2572284f0b55","Type":"ContainerDied","Data":"b017bec8a789e386d07af2e23e2452d08a11c3e5acd0d101eadbc80a872dfbf7"} Oct 31 04:45:19.412341 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:19.412307 4376 scope.go:122] "RemoveContainer" containerID="b017bec8a789e386d07af2e23e2452d08a11c3e5acd0d101eadbc80a872dfbf7" Oct 31 04:45:19.412632 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:45:19.412611 4376 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-47c409-29890365-vsw27_odh-ai-gateway-infra(fbee2356-f532-40d7-90b6-2572284f0b55)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-47c409-29890365-vsw27" podUID="fbee2356-f532-40d7-90b6-2572284f0b55" Oct 31 04:45:19.421469 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:19.421452 4376 scope.go:122] "RemoveContainer" containerID="1e86211c01d01a5dcc5daf94582670750b145655f6ec100feee13b6d2c609432" Oct 31 04:45:19.431106 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:19.431089 4376 scope.go:122] "RemoveContainer" containerID="c08f3a719a9de100d5cc2ab560c363e95748626d54c51e265e8ab5c39db3ac1c" Oct 31 04:45:19.444740 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:19.444720 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-401761-29890365-xnh62"] Oct 31 04:45:19.539505 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:19.539475 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890365-5gwm4"] Oct 31 04:45:19.547083 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:19.547055 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890365-5gwm4"] Oct 31 04:45:19.570398 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:19.570367 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-47c409-29890365-vsw27"] Oct 31 04:45:20.446313 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:20.446284 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18cae0c8-c13a-4ebe-9c1d-3f87232d46fb" path="/var/lib/kubelet/pods/18cae0c8-c13a-4ebe-9c1d-3f87232d46fb/volumes" Oct 31 04:45:20.446804 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:20.446789 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8032699a-3ca2-4cca-9366-10006ea03b4d" path="/var/lib/kubelet/pods/8032699a-3ca2-4cca-9366-10006ea03b4d/volumes" Oct 31 04:45:20.602971 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:20.602946 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-47c409-29890365-vsw27" Oct 31 04:45:20.641596 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:20.641573 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-401761-29890365-xnh62" Oct 31 04:45:20.685831 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:20.685748 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbee2356-f532-40d7-90b6-2572284f0b55-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbee2356-f532-40d7-90b6-2572284f0b55-maas-api-service-ca\") pod \"fbee2356-f532-40d7-90b6-2572284f0b55\" (UID: \"fbee2356-f532-40d7-90b6-2572284f0b55\") " Oct 31 04:45:20.685997 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:20.685879 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbee2356-f532-40d7-90b6-2572284f0b55-kube-api-access-psp6l\" (UniqueName: \"kubernetes.io/projected/fbee2356-f532-40d7-90b6-2572284f0b55-kube-api-access-psp6l\") pod \"fbee2356-f532-40d7-90b6-2572284f0b55\" (UID: \"fbee2356-f532-40d7-90b6-2572284f0b55\") " Oct 31 04:45:20.685997 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:20.685911 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2608ed82-e75d-41bf-8710-0faf0ce0ae08-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2608ed82-e75d-41bf-8710-0faf0ce0ae08-maas-api-service-ca\") pod \"2608ed82-e75d-41bf-8710-0faf0ce0ae08\" (UID: \"2608ed82-e75d-41bf-8710-0faf0ce0ae08\") " Oct 31 04:45:20.685997 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:20.685981 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2608ed82-e75d-41bf-8710-0faf0ce0ae08-kube-api-access-98glr\" (UniqueName: \"kubernetes.io/projected/2608ed82-e75d-41bf-8710-0faf0ce0ae08-kube-api-access-98glr\") pod \"2608ed82-e75d-41bf-8710-0faf0ce0ae08\" (UID: \"2608ed82-e75d-41bf-8710-0faf0ce0ae08\") " Oct 31 04:45:20.686173 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:20.686122 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbee2356-f532-40d7-90b6-2572284f0b55-maas-api-service-ca" pod "fbee2356-f532-40d7-90b6-2572284f0b55" (UID: "fbee2356-f532-40d7-90b6-2572284f0b55"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:45:20.686723 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:20.686694 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2608ed82-e75d-41bf-8710-0faf0ce0ae08-maas-api-service-ca" pod "2608ed82-e75d-41bf-8710-0faf0ce0ae08" (UID: "2608ed82-e75d-41bf-8710-0faf0ce0ae08"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:45:20.686850 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:20.686833 4376 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbee2356-f532-40d7-90b6-2572284f0b55-maas-api-service-ca\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:45:20.686926 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:20.686856 4376 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2608ed82-e75d-41bf-8710-0faf0ce0ae08-maas-api-service-ca\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:45:20.688621 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:20.688592 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2608ed82-e75d-41bf-8710-0faf0ce0ae08-kube-api-access-98glr" pod "2608ed82-e75d-41bf-8710-0faf0ce0ae08" (UID: "2608ed82-e75d-41bf-8710-0faf0ce0ae08"). InnerVolumeSpecName "kube-api-access-98glr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:45:20.689608 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:20.689569 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbee2356-f532-40d7-90b6-2572284f0b55-kube-api-access-psp6l" pod "fbee2356-f532-40d7-90b6-2572284f0b55" (UID: "fbee2356-f532-40d7-90b6-2572284f0b55"). InnerVolumeSpecName "kube-api-access-psp6l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:45:20.787937 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:20.787897 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-psp6l\" (UniqueName: \"kubernetes.io/projected/fbee2356-f532-40d7-90b6-2572284f0b55-kube-api-access-psp6l\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:45:20.787937 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:20.787931 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98glr\" (UniqueName: \"kubernetes.io/projected/2608ed82-e75d-41bf-8710-0faf0ce0ae08-kube-api-access-98glr\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:45:21.432710 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:21.432670 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-47c409-29890365-vsw27" Oct 31 04:45:21.432710 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:21.432692 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-47c409-29890365-vsw27" event={"ID":"fbee2356-f532-40d7-90b6-2572284f0b55","Type":"ContainerDied","Data":"f5773e5338e022c426d02f5bee6a315805ddaa5fd39920163625357905682cd8"} Oct 31 04:45:21.432947 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:21.432802 4376 scope.go:122] "RemoveContainer" containerID="b017bec8a789e386d07af2e23e2452d08a11c3e5acd0d101eadbc80a872dfbf7" Oct 31 04:45:21.434154 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:21.434137 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-401761-29890365-xnh62" Oct 31 04:45:21.434235 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:21.434174 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-401761-29890365-xnh62" event={"ID":"2608ed82-e75d-41bf-8710-0faf0ce0ae08","Type":"ContainerDied","Data":"5be17e93c5ace15cb3155f32866517509fd4ecb7a93c29e44848ba4aab3f7ce9"} Oct 31 04:45:21.444662 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:21.444613 4376 scope.go:122] "RemoveContainer" containerID="485d8fb311649ba4b39e0e093be9f5a0dffca4bd33a1080ab5c58c3e05f2797c" Oct 31 04:45:21.492410 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:21.492383 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-47c409-29890365-vsw27"] Oct 31 04:45:21.498011 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:21.497987 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-47c409-29890365-vsw27"] Oct 31 04:45:21.542392 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:21.542361 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-401761-29890365-xnh62"] Oct 31 04:45:21.550820 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:21.550794 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-401761-29890365-xnh62"] Oct 31 04:45:22.415243 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:22.415200 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-7bc522j9n" podUID="5e17d6fa-35eb-44f6-8a4a-bd1bd937a7b2" containerName="payload-processing" containerID="cri-o://2e96e77a0cf6d88cca591a349a522fedf6c9f0777b91505642079b6ae79d6684" gracePeriod=25 Oct 31 04:45:22.447128 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:22.447105 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2608ed82-e75d-41bf-8710-0faf0ce0ae08" path="/var/lib/kubelet/pods/2608ed82-e75d-41bf-8710-0faf0ce0ae08/volumes" Oct 31 04:45:22.447603 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:22.447587 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbee2356-f532-40d7-90b6-2572284f0b55" path="/var/lib/kubelet/pods/fbee2356-f532-40d7-90b6-2572284f0b55/volumes" Oct 31 04:45:22.950727 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:22.950701 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-7bc522j9n" Oct 31 04:45:23.045896 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:23.045853 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e17d6fa-35eb-44f6-8a4a-bd1bd937a7b2-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5e17d6fa-35eb-44f6-8a4a-bd1bd937a7b2-routing-overlay-volume\") pod \"5e17d6fa-35eb-44f6-8a4a-bd1bd937a7b2\" (UID: \"5e17d6fa-35eb-44f6-8a4a-bd1bd937a7b2\") " Oct 31 04:45:23.046098 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:23.045940 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e17d6fa-35eb-44f6-8a4a-bd1bd937a7b2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e17d6fa-35eb-44f6-8a4a-bd1bd937a7b2-plugins-config-volume\") pod \"5e17d6fa-35eb-44f6-8a4a-bd1bd937a7b2\" (UID: \"5e17d6fa-35eb-44f6-8a4a-bd1bd937a7b2\") " Oct 31 04:45:23.046098 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:23.046044 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e17d6fa-35eb-44f6-8a4a-bd1bd937a7b2-provider-credentials\" (UniqueName: \"kubernetes.io/projected/5e17d6fa-35eb-44f6-8a4a-bd1bd937a7b2-provider-credentials\") pod \"5e17d6fa-35eb-44f6-8a4a-bd1bd937a7b2\" (UID: \"5e17d6fa-35eb-44f6-8a4a-bd1bd937a7b2\") " Oct 31 04:45:23.046236 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:23.046104 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e17d6fa-35eb-44f6-8a4a-bd1bd937a7b2-kube-api-access-d862r\" (UniqueName: \"kubernetes.io/projected/5e17d6fa-35eb-44f6-8a4a-bd1bd937a7b2-kube-api-access-d862r\") pod \"5e17d6fa-35eb-44f6-8a4a-bd1bd937a7b2\" (UID: \"5e17d6fa-35eb-44f6-8a4a-bd1bd937a7b2\") " Oct 31 04:45:23.046349 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:23.046297 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e17d6fa-35eb-44f6-8a4a-bd1bd937a7b2-routing-overlay-volume" pod "5e17d6fa-35eb-44f6-8a4a-bd1bd937a7b2" (UID: "5e17d6fa-35eb-44f6-8a4a-bd1bd937a7b2"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:45:23.046349 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:23.046317 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e17d6fa-35eb-44f6-8a4a-bd1bd937a7b2-plugins-config-volume" pod "5e17d6fa-35eb-44f6-8a4a-bd1bd937a7b2" (UID: "5e17d6fa-35eb-44f6-8a4a-bd1bd937a7b2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:45:23.046716 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:23.046698 4376 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5e17d6fa-35eb-44f6-8a4a-bd1bd937a7b2-routing-overlay-volume\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:45:23.046767 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:23.046719 4376 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e17d6fa-35eb-44f6-8a4a-bd1bd937a7b2-plugins-config-volume\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:45:23.048138 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:23.048119 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e17d6fa-35eb-44f6-8a4a-bd1bd937a7b2-kube-api-access-d862r" pod "5e17d6fa-35eb-44f6-8a4a-bd1bd937a7b2" (UID: "5e17d6fa-35eb-44f6-8a4a-bd1bd937a7b2"). InnerVolumeSpecName "kube-api-access-d862r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:45:23.048201 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:23.048137 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e17d6fa-35eb-44f6-8a4a-bd1bd937a7b2-provider-credentials" pod "5e17d6fa-35eb-44f6-8a4a-bd1bd937a7b2" (UID: "5e17d6fa-35eb-44f6-8a4a-bd1bd937a7b2"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:45:23.147778 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:23.147734 4376 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5e17d6fa-35eb-44f6-8a4a-bd1bd937a7b2-provider-credentials\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:45:23.147778 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:23.147776 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d862r\" (UniqueName: \"kubernetes.io/projected/5e17d6fa-35eb-44f6-8a4a-bd1bd937a7b2-kube-api-access-d862r\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:45:23.454138 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:23.454048 4376 generic.go:356] "Generic (PLEG): container finished" podID="5e17d6fa-35eb-44f6-8a4a-bd1bd937a7b2" containerID="2e96e77a0cf6d88cca591a349a522fedf6c9f0777b91505642079b6ae79d6684" exitCode=0 Oct 31 04:45:23.454138 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:23.454092 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-7bc522j9n" Oct 31 04:45:23.454138 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:23.454128 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-7bc522j9n" event={"ID":"5e17d6fa-35eb-44f6-8a4a-bd1bd937a7b2","Type":"ContainerDied","Data":"2e96e77a0cf6d88cca591a349a522fedf6c9f0777b91505642079b6ae79d6684"} Oct 31 04:45:23.454433 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:23.454183 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-7bc522j9n" event={"ID":"5e17d6fa-35eb-44f6-8a4a-bd1bd937a7b2","Type":"ContainerDied","Data":"d909450a82c07a7a314bce9a1e47b9f792df5d277a772ea7754a780c4eeaef35"} Oct 31 04:45:23.454433 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:23.454204 4376 scope.go:122] "RemoveContainer" containerID="2e96e77a0cf6d88cca591a349a522fedf6c9f0777b91505642079b6ae79d6684" Oct 31 04:45:23.465728 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:23.465708 4376 scope.go:122] "RemoveContainer" containerID="2e96e77a0cf6d88cca591a349a522fedf6c9f0777b91505642079b6ae79d6684" Oct 31 04:45:23.466126 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:45:23.466106 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e96e77a0cf6d88cca591a349a522fedf6c9f0777b91505642079b6ae79d6684\": container with ID starting with 2e96e77a0cf6d88cca591a349a522fedf6c9f0777b91505642079b6ae79d6684 not found: ID does not exist" containerID="2e96e77a0cf6d88cca591a349a522fedf6c9f0777b91505642079b6ae79d6684" Oct 31 04:45:23.466201 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:23.466135 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e96e77a0cf6d88cca591a349a522fedf6c9f0777b91505642079b6ae79d6684"} err="failed to get container status \"2e96e77a0cf6d88cca591a349a522fedf6c9f0777b91505642079b6ae79d6684\": rpc error: code = NotFound desc = could not find container \"2e96e77a0cf6d88cca591a349a522fedf6c9f0777b91505642079b6ae79d6684\": container with ID starting with 2e96e77a0cf6d88cca591a349a522fedf6c9f0777b91505642079b6ae79d6684 not found: ID does not exist" Oct 31 04:45:23.618359 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:23.614383 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-7bc522j9n"] Oct 31 04:45:23.635699 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:23.635662 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-7bc522j9n"] Oct 31 04:45:24.445554 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:24.445514 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e17d6fa-35eb-44f6-8a4a-bd1bd937a7b2" path="/var/lib/kubelet/pods/5e17d6fa-35eb-44f6-8a4a-bd1bd937a7b2/volumes" Oct 31 04:45:29.415982 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:29.415953 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-b679dtfm4" Oct 31 04:45:41.022772 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:45:41.022727 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 04:45:42.446183 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:42.446153 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-kc5qw"] Oct 31 04:45:42.446851 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:42.446808 4376 cpu_manager.go:414] "containerMap: removing container" podUID="d5059d94-428a-4df5-99c8-4b2e4f3f87ab" containerName="cleanup" Oct 31 04:45:42.446851 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:42.446823 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5059d94-428a-4df5-99c8-4b2e4f3f87ab" containerName="cleanup" Oct 31 04:45:42.446851 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:42.446834 4376 cpu_manager.go:414] "containerMap: removing container" podUID="8032699a-3ca2-4cca-9366-10006ea03b4d" containerName="cleanup" Oct 31 04:45:42.446851 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:42.446839 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8032699a-3ca2-4cca-9366-10006ea03b4d" containerName="cleanup" Oct 31 04:45:42.446851 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:42.446856 4376 cpu_manager.go:414] "containerMap: removing container" podUID="2608ed82-e75d-41bf-8710-0faf0ce0ae08" containerName="cleanup" Oct 31 04:45:42.446851 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:42.446860 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2608ed82-e75d-41bf-8710-0faf0ce0ae08" containerName="cleanup" Oct 31 04:45:42.446851 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:42.446871 4376 cpu_manager.go:414] "containerMap: removing container" podUID="18cae0c8-c13a-4ebe-9c1d-3f87232d46fb" containerName="cleanup" Oct 31 04:45:42.447195 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:42.446879 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18cae0c8-c13a-4ebe-9c1d-3f87232d46fb" containerName="cleanup" Oct 31 04:45:42.447195 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:42.446888 4376 cpu_manager.go:414] "containerMap: removing container" podUID="8032699a-3ca2-4cca-9366-10006ea03b4d" containerName="cleanup" Oct 31 04:45:42.447195 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:42.446895 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8032699a-3ca2-4cca-9366-10006ea03b4d" containerName="cleanup" Oct 31 04:45:42.447195 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:42.446907 4376 cpu_manager.go:414] "containerMap: removing container" podUID="2608ed82-e75d-41bf-8710-0faf0ce0ae08" containerName="cleanup" Oct 31 04:45:42.447195 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:42.446914 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2608ed82-e75d-41bf-8710-0faf0ce0ae08" containerName="cleanup" Oct 31 04:45:42.447195 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:42.446928 4376 cpu_manager.go:414] "containerMap: removing container" podUID="8032699a-3ca2-4cca-9366-10006ea03b4d" containerName="cleanup" Oct 31 04:45:42.447195 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:42.446935 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8032699a-3ca2-4cca-9366-10006ea03b4d" containerName="cleanup" Oct 31 04:45:42.447195 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:42.446943 4376 cpu_manager.go:414] "containerMap: removing container" podUID="fbee2356-f532-40d7-90b6-2572284f0b55" containerName="cleanup" Oct 31 04:45:42.447195 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:42.446950 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbee2356-f532-40d7-90b6-2572284f0b55" containerName="cleanup" Oct 31 04:45:42.447195 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:42.446973 4376 cpu_manager.go:414] "containerMap: removing container" podUID="18cae0c8-c13a-4ebe-9c1d-3f87232d46fb" containerName="cleanup" Oct 31 04:45:42.447195 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:42.446980 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18cae0c8-c13a-4ebe-9c1d-3f87232d46fb" containerName="cleanup" Oct 31 04:45:42.447195 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:42.446995 4376 cpu_manager.go:414] "containerMap: removing container" podUID="fbee2356-f532-40d7-90b6-2572284f0b55" containerName="cleanup" Oct 31 04:45:42.447195 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:42.447002 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbee2356-f532-40d7-90b6-2572284f0b55" containerName="cleanup" Oct 31 04:45:42.447195 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:42.447025 4376 cpu_manager.go:414] "containerMap: removing container" podUID="5e17d6fa-35eb-44f6-8a4a-bd1bd937a7b2" containerName="payload-processing" Oct 31 04:45:42.447195 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:42.447031 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e17d6fa-35eb-44f6-8a4a-bd1bd937a7b2" containerName="payload-processing" Oct 31 04:45:42.447195 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:42.447181 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbee2356-f532-40d7-90b6-2572284f0b55" containerName="cleanup" Oct 31 04:45:42.447195 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:42.447193 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="18cae0c8-c13a-4ebe-9c1d-3f87232d46fb" containerName="cleanup" Oct 31 04:45:42.447195 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:42.447199 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="2608ed82-e75d-41bf-8710-0faf0ce0ae08" containerName="cleanup" Oct 31 04:45:42.447738 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:42.447205 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="8032699a-3ca2-4cca-9366-10006ea03b4d" containerName="cleanup" Oct 31 04:45:42.447738 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:42.447214 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="2608ed82-e75d-41bf-8710-0faf0ce0ae08" containerName="cleanup" Oct 31 04:45:42.447738 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:42.447221 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="18cae0c8-c13a-4ebe-9c1d-3f87232d46fb" containerName="cleanup" Oct 31 04:45:42.447738 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:42.447225 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="8032699a-3ca2-4cca-9366-10006ea03b4d" containerName="cleanup" Oct 31 04:45:42.447738 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:42.447232 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="2608ed82-e75d-41bf-8710-0faf0ce0ae08" containerName="cleanup" Oct 31 04:45:42.447738 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:42.447237 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="8032699a-3ca2-4cca-9366-10006ea03b4d" containerName="cleanup" Oct 31 04:45:42.447738 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:42.447242 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="18cae0c8-c13a-4ebe-9c1d-3f87232d46fb" containerName="cleanup" Oct 31 04:45:42.447738 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:42.447249 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbee2356-f532-40d7-90b6-2572284f0b55" containerName="cleanup" Oct 31 04:45:42.447738 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:42.447255 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e17d6fa-35eb-44f6-8a4a-bd1bd937a7b2" containerName="payload-processing" Oct 31 04:45:42.451025 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:42.451008 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-kc5qw" Oct 31 04:45:42.468109 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:42.468079 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-kc5qw"] Oct 31 04:45:42.476104 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:42.476073 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-567ca6-openshift-default-dockercfg-nclf8\"" Oct 31 04:45:42.539474 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:42.539441 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19a7ea83-1d36-494a-ab52-f24535365ff9-istio-data\") pod \"e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-kc5qw\" (UID: \"19a7ea83-1d36-494a-ab52-f24535365ff9\") " pod="openshift-ingress/e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-kc5qw" Oct 31 04:45:42.539676 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:42.539484 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19a7ea83-1d36-494a-ab52-f24535365ff9-workload-socket\") pod \"e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-kc5qw\" (UID: \"19a7ea83-1d36-494a-ab52-f24535365ff9\") " pod="openshift-ingress/e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-kc5qw" Oct 31 04:45:42.539676 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:42.539567 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hp8rf\" (UniqueName: \"kubernetes.io/projected/19a7ea83-1d36-494a-ab52-f24535365ff9-kube-api-access-hp8rf\") pod \"e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-kc5qw\" (UID: \"19a7ea83-1d36-494a-ab52-f24535365ff9\") " pod="openshift-ingress/e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-kc5qw" Oct 31 04:45:42.539814 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:42.539789 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19a7ea83-1d36-494a-ab52-f24535365ff9-istio-envoy\") pod \"e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-kc5qw\" (UID: \"19a7ea83-1d36-494a-ab52-f24535365ff9\") " pod="openshift-ingress/e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-kc5qw" Oct 31 04:45:42.539876 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:42.539860 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19a7ea83-1d36-494a-ab52-f24535365ff9-istio-podinfo\") pod \"e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-kc5qw\" (UID: \"19a7ea83-1d36-494a-ab52-f24535365ff9\") " pod="openshift-ingress/e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-kc5qw" Oct 31 04:45:42.539990 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:42.539975 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19a7ea83-1d36-494a-ab52-f24535365ff9-istio-token\") pod \"e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-kc5qw\" (UID: \"19a7ea83-1d36-494a-ab52-f24535365ff9\") " pod="openshift-ingress/e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-kc5qw" Oct 31 04:45:42.540046 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:42.540006 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19a7ea83-1d36-494a-ab52-f24535365ff9-istiod-ca-cert\") pod \"e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-kc5qw\" (UID: \"19a7ea83-1d36-494a-ab52-f24535365ff9\") " pod="openshift-ingress/e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-kc5qw" Oct 31 04:45:42.540100 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:42.540064 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19a7ea83-1d36-494a-ab52-f24535365ff9-credential-socket\") pod \"e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-kc5qw\" (UID: \"19a7ea83-1d36-494a-ab52-f24535365ff9\") " pod="openshift-ingress/e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-kc5qw" Oct 31 04:45:42.540203 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:42.540179 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19a7ea83-1d36-494a-ab52-f24535365ff9-workload-certs\") pod \"e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-kc5qw\" (UID: \"19a7ea83-1d36-494a-ab52-f24535365ff9\") " pod="openshift-ingress/e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-kc5qw" Oct 31 04:45:42.641600 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:42.641564 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19a7ea83-1d36-494a-ab52-f24535365ff9-workload-certs\") pod \"e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-kc5qw\" (UID: \"19a7ea83-1d36-494a-ab52-f24535365ff9\") " pod="openshift-ingress/e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-kc5qw" Oct 31 04:45:42.641791 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:42.641623 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19a7ea83-1d36-494a-ab52-f24535365ff9-istio-data\") pod \"e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-kc5qw\" (UID: \"19a7ea83-1d36-494a-ab52-f24535365ff9\") " pod="openshift-ingress/e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-kc5qw" Oct 31 04:45:42.641791 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:42.641662 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19a7ea83-1d36-494a-ab52-f24535365ff9-workload-socket\") pod \"e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-kc5qw\" (UID: \"19a7ea83-1d36-494a-ab52-f24535365ff9\") " pod="openshift-ingress/e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-kc5qw" Oct 31 04:45:42.641791 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:42.641701 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hp8rf\" (UniqueName: \"kubernetes.io/projected/19a7ea83-1d36-494a-ab52-f24535365ff9-kube-api-access-hp8rf\") pod \"e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-kc5qw\" (UID: \"19a7ea83-1d36-494a-ab52-f24535365ff9\") " pod="openshift-ingress/e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-kc5qw" Oct 31 04:45:42.641964 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:42.641803 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19a7ea83-1d36-494a-ab52-f24535365ff9-istio-envoy\") pod \"e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-kc5qw\" (UID: \"19a7ea83-1d36-494a-ab52-f24535365ff9\") " pod="openshift-ingress/e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-kc5qw" Oct 31 04:45:42.641964 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:42.641842 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19a7ea83-1d36-494a-ab52-f24535365ff9-istio-podinfo\") pod \"e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-kc5qw\" (UID: \"19a7ea83-1d36-494a-ab52-f24535365ff9\") " pod="openshift-ingress/e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-kc5qw" Oct 31 04:45:42.641964 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:42.641933 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19a7ea83-1d36-494a-ab52-f24535365ff9-istio-token\") pod \"e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-kc5qw\" (UID: \"19a7ea83-1d36-494a-ab52-f24535365ff9\") " pod="openshift-ingress/e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-kc5qw" Oct 31 04:45:42.642126 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:42.641963 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19a7ea83-1d36-494a-ab52-f24535365ff9-istiod-ca-cert\") pod \"e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-kc5qw\" (UID: \"19a7ea83-1d36-494a-ab52-f24535365ff9\") " pod="openshift-ingress/e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-kc5qw" Oct 31 04:45:42.642126 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:42.642002 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19a7ea83-1d36-494a-ab52-f24535365ff9-istio-data\") pod \"e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-kc5qw\" (UID: \"19a7ea83-1d36-494a-ab52-f24535365ff9\") " pod="openshift-ingress/e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-kc5qw" Oct 31 04:45:42.642126 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:42.642007 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19a7ea83-1d36-494a-ab52-f24535365ff9-credential-socket\") pod \"e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-kc5qw\" (UID: \"19a7ea83-1d36-494a-ab52-f24535365ff9\") " pod="openshift-ingress/e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-kc5qw" Oct 31 04:45:42.642126 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:42.642055 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19a7ea83-1d36-494a-ab52-f24535365ff9-workload-socket\") pod \"e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-kc5qw\" (UID: \"19a7ea83-1d36-494a-ab52-f24535365ff9\") " pod="openshift-ingress/e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-kc5qw" Oct 31 04:45:42.642379 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:42.641974 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19a7ea83-1d36-494a-ab52-f24535365ff9-workload-certs\") pod \"e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-kc5qw\" (UID: \"19a7ea83-1d36-494a-ab52-f24535365ff9\") " pod="openshift-ingress/e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-kc5qw" Oct 31 04:45:42.642379 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:42.642284 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19a7ea83-1d36-494a-ab52-f24535365ff9-credential-socket\") pod \"e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-kc5qw\" (UID: \"19a7ea83-1d36-494a-ab52-f24535365ff9\") " pod="openshift-ingress/e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-kc5qw" Oct 31 04:45:42.642746 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:42.642728 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19a7ea83-1d36-494a-ab52-f24535365ff9-istiod-ca-cert\") pod \"e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-kc5qw\" (UID: \"19a7ea83-1d36-494a-ab52-f24535365ff9\") " pod="openshift-ingress/e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-kc5qw" Oct 31 04:45:42.643995 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:42.643967 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19a7ea83-1d36-494a-ab52-f24535365ff9-istio-envoy\") pod \"e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-kc5qw\" (UID: \"19a7ea83-1d36-494a-ab52-f24535365ff9\") " pod="openshift-ingress/e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-kc5qw" Oct 31 04:45:42.644259 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:42.644240 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19a7ea83-1d36-494a-ab52-f24535365ff9-istio-podinfo\") pod \"e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-kc5qw\" (UID: \"19a7ea83-1d36-494a-ab52-f24535365ff9\") " pod="openshift-ingress/e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-kc5qw" Oct 31 04:45:42.665718 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:42.665697 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hp8rf\" (UniqueName: \"kubernetes.io/projected/19a7ea83-1d36-494a-ab52-f24535365ff9-kube-api-access-hp8rf\") pod \"e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-kc5qw\" (UID: \"19a7ea83-1d36-494a-ab52-f24535365ff9\") " pod="openshift-ingress/e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-kc5qw" Oct 31 04:45:42.665836 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:42.665786 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19a7ea83-1d36-494a-ab52-f24535365ff9-istio-token\") pod \"e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-kc5qw\" (UID: \"19a7ea83-1d36-494a-ab52-f24535365ff9\") " pod="openshift-ingress/e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-kc5qw" Oct 31 04:45:42.766105 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:42.766042 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-kc5qw" Oct 31 04:45:43.085507 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:45:43.085473 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19a7ea83_1d36_494a_ab52_f24535365ff9.slice/crio-59e64d053c8c3efcf67e8c460a60a67502db9e5b9f32ac6d7b5f1bdcacb50ff1 WatchSource:0}: Error finding container 59e64d053c8c3efcf67e8c460a60a67502db9e5b9f32ac6d7b5f1bdcacb50ff1: Status 404 returned error can't find the container with id 59e64d053c8c3efcf67e8c460a60a67502db9e5b9f32ac6d7b5f1bdcacb50ff1 Oct 31 04:45:43.086830 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:43.086802 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-kc5qw"] Oct 31 04:45:43.089385 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:43.089349 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:45:43.089463 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:43.089435 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:45:43.089508 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:43.089477 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:45:43.607915 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:43.607881 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-kc5qw" event={"ID":"19a7ea83-1d36-494a-ab52-f24535365ff9","Type":"ContainerStarted","Data":"ff3dbd23adb8f169e27436691ba272353a719f518e28fe4cb18bcbb7ab5e317d"} Oct 31 04:45:43.608535 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:43.608491 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-kc5qw" event={"ID":"19a7ea83-1d36-494a-ab52-f24535365ff9","Type":"ContainerStarted","Data":"59e64d053c8c3efcf67e8c460a60a67502db9e5b9f32ac6d7b5f1bdcacb50ff1"} Oct 31 04:45:43.671113 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:43.671052 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-kc5qw" podStartSLOduration=1.671033161 podStartE2EDuration="1.671033161s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:45:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:45:43.667558235 +0000 UTC m=+2491.381892872" watchObservedRunningTime="2026-10-31 04:45:43.671033161 +0000 UTC m=+2491.385367778" Oct 31 04:45:43.766415 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:43.766313 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-kc5qw" Oct 31 04:45:44.768808 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:44.768763 4376 patch_prober.go:29] interesting pod/e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-kc5qw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" start-of-body= Oct 31 04:45:44.769213 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:44.768842 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-kc5qw" podUID="19a7ea83-1d36-494a-ab52-f24535365ff9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" Oct 31 04:45:45.766570 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:45.766532 4376 patch_prober.go:29] interesting pod/e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-kc5qw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" start-of-body= Oct 31 04:45:45.766770 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:45.766650 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-kc5qw" podUID="19a7ea83-1d36-494a-ab52-f24535365ff9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" Oct 31 04:45:45.815897 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:45.815866 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-55cfq6jfg"] Oct 31 04:45:46.768730 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:46.768692 4376 patch_prober.go:29] interesting pod/e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-kc5qw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" start-of-body= Oct 31 04:45:46.769040 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:46.769006 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-kc5qw" podUID="19a7ea83-1d36-494a-ab52-f24535365ff9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" Oct 31 04:45:47.767000 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:47.766965 4376 patch_prober.go:29] interesting pod/e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-kc5qw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" start-of-body= Oct 31 04:45:47.767413 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:47.767020 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-kc5qw" podUID="19a7ea83-1d36-494a-ab52-f24535365ff9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" Oct 31 04:45:48.766874 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:48.766829 4376 patch_prober.go:29] interesting pod/e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-kc5qw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" start-of-body= Oct 31 04:45:48.767062 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:48.766897 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-kc5qw" podUID="19a7ea83-1d36-494a-ab52-f24535365ff9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" Oct 31 04:45:49.796352 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:49.796299 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-kc5qw" Oct 31 04:45:49.798547 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:49.798515 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-kc5qw" Oct 31 04:45:49.799745 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:49.799724 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-kc5qw" Oct 31 04:45:50.816922 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:50.816878 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-55cfq6jfg" podUID="7c143dfc-5cce-460b-b348-c5ed65ab662a" containerName="payload-processing" containerID="cri-o://c6da9cdf59c1208f7a633c9a1d4884ab5a9aacd256b7b92ad426b7ba2567427f" gracePeriod=25 Oct 31 04:45:51.085071 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:51.085011 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-55cfq6jfg" Oct 31 04:45:51.157174 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:51.157141 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c143dfc-5cce-460b-b348-c5ed65ab662a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c143dfc-5cce-460b-b348-c5ed65ab662a-plugins-config-volume\") pod \"7c143dfc-5cce-460b-b348-c5ed65ab662a\" (UID: \"7c143dfc-5cce-460b-b348-c5ed65ab662a\") " Oct 31 04:45:51.157311 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:51.157300 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c143dfc-5cce-460b-b348-c5ed65ab662a-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7c143dfc-5cce-460b-b348-c5ed65ab662a-routing-overlay-volume\") pod \"7c143dfc-5cce-460b-b348-c5ed65ab662a\" (UID: \"7c143dfc-5cce-460b-b348-c5ed65ab662a\") " Oct 31 04:45:51.157469 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:51.157451 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c143dfc-5cce-460b-b348-c5ed65ab662a-provider-credentials\" (UniqueName: \"kubernetes.io/projected/7c143dfc-5cce-460b-b348-c5ed65ab662a-provider-credentials\") pod \"7c143dfc-5cce-460b-b348-c5ed65ab662a\" (UID: \"7c143dfc-5cce-460b-b348-c5ed65ab662a\") " Oct 31 04:45:51.157541 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:51.157521 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c143dfc-5cce-460b-b348-c5ed65ab662a-kube-api-access-svq5k\" (UniqueName: \"kubernetes.io/projected/7c143dfc-5cce-460b-b348-c5ed65ab662a-kube-api-access-svq5k\") pod \"7c143dfc-5cce-460b-b348-c5ed65ab662a\" (UID: \"7c143dfc-5cce-460b-b348-c5ed65ab662a\") " Oct 31 04:45:51.157635 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:51.157615 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c143dfc-5cce-460b-b348-c5ed65ab662a-plugins-config-volume" pod "7c143dfc-5cce-460b-b348-c5ed65ab662a" (UID: "7c143dfc-5cce-460b-b348-c5ed65ab662a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:45:51.157701 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:51.157642 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c143dfc-5cce-460b-b348-c5ed65ab662a-routing-overlay-volume" pod "7c143dfc-5cce-460b-b348-c5ed65ab662a" (UID: "7c143dfc-5cce-460b-b348-c5ed65ab662a"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:45:51.157947 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:51.157925 4376 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7c143dfc-5cce-460b-b348-c5ed65ab662a-routing-overlay-volume\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:45:51.157947 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:51.157947 4376 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c143dfc-5cce-460b-b348-c5ed65ab662a-plugins-config-volume\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:45:51.159640 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:51.159618 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c143dfc-5cce-460b-b348-c5ed65ab662a-provider-credentials" pod "7c143dfc-5cce-460b-b348-c5ed65ab662a" (UID: "7c143dfc-5cce-460b-b348-c5ed65ab662a"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:45:51.159827 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:51.159802 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c143dfc-5cce-460b-b348-c5ed65ab662a-kube-api-access-svq5k" pod "7c143dfc-5cce-460b-b348-c5ed65ab662a" (UID: "7c143dfc-5cce-460b-b348-c5ed65ab662a"). InnerVolumeSpecName "kube-api-access-svq5k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:45:51.259754 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:51.259723 4376 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7c143dfc-5cce-460b-b348-c5ed65ab662a-provider-credentials\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:45:51.259754 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:51.259749 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-svq5k\" (UniqueName: \"kubernetes.io/projected/7c143dfc-5cce-460b-b348-c5ed65ab662a-kube-api-access-svq5k\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:45:51.714271 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:51.714236 4376 generic.go:356] "Generic (PLEG): container finished" podID="7c143dfc-5cce-460b-b348-c5ed65ab662a" containerID="c6da9cdf59c1208f7a633c9a1d4884ab5a9aacd256b7b92ad426b7ba2567427f" exitCode=0 Oct 31 04:45:51.714471 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:51.714282 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-55cfq6jfg" Oct 31 04:45:51.714471 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:51.714343 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-55cfq6jfg" event={"ID":"7c143dfc-5cce-460b-b348-c5ed65ab662a","Type":"ContainerDied","Data":"c6da9cdf59c1208f7a633c9a1d4884ab5a9aacd256b7b92ad426b7ba2567427f"} Oct 31 04:45:51.714471 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:51.714394 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-55cfq6jfg" event={"ID":"7c143dfc-5cce-460b-b348-c5ed65ab662a","Type":"ContainerDied","Data":"a3bd5e9117cac169884a39965ec3cc8ebc3b3b0eb0e6b0e0f2644ded07e37c4f"} Oct 31 04:45:51.714471 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:51.714421 4376 scope.go:122] "RemoveContainer" containerID="c6da9cdf59c1208f7a633c9a1d4884ab5a9aacd256b7b92ad426b7ba2567427f" Oct 31 04:45:51.725121 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:51.725102 4376 scope.go:122] "RemoveContainer" containerID="c6da9cdf59c1208f7a633c9a1d4884ab5a9aacd256b7b92ad426b7ba2567427f" Oct 31 04:45:51.725414 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:45:51.725395 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6da9cdf59c1208f7a633c9a1d4884ab5a9aacd256b7b92ad426b7ba2567427f\": container with ID starting with c6da9cdf59c1208f7a633c9a1d4884ab5a9aacd256b7b92ad426b7ba2567427f not found: ID does not exist" containerID="c6da9cdf59c1208f7a633c9a1d4884ab5a9aacd256b7b92ad426b7ba2567427f" Oct 31 04:45:51.725489 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:51.725425 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6da9cdf59c1208f7a633c9a1d4884ab5a9aacd256b7b92ad426b7ba2567427f"} err="failed to get container status \"c6da9cdf59c1208f7a633c9a1d4884ab5a9aacd256b7b92ad426b7ba2567427f\": rpc error: code = NotFound desc = could not find container \"c6da9cdf59c1208f7a633c9a1d4884ab5a9aacd256b7b92ad426b7ba2567427f\": container with ID starting with c6da9cdf59c1208f7a633c9a1d4884ab5a9aacd256b7b92ad426b7ba2567427f not found: ID does not exist" Oct 31 04:45:51.769627 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:51.769601 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-55cfq6jfg"] Oct 31 04:45:51.781207 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:51.781185 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-55cfq6jfg"] Oct 31 04:45:52.445183 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:52.445151 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c143dfc-5cce-460b-b348-c5ed65ab662a" path="/var/lib/kubelet/pods/7c143dfc-5cce-460b-b348-c5ed65ab662a/volumes" Oct 31 04:45:59.773522 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:45:59.773484 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 04:46:11.000959 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:46:11.000922 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 04:46:41.003510 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:46:41.003474 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 04:46:41.203906 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:46:41.203866 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 04:46:55.235627 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:46:55.235551 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-np274"] Oct 31 04:46:55.236165 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:46:55.235789 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-np274" podUID="0de4ffc1-c510-4022-b7be-3f8f13360f76" containerName="istio-proxy" containerID="cri-o://218edb5fe0fbcf9808d1435b2dd347951d228dba57f508800d6e58852999807e" gracePeriod=30 Oct 31 04:47:00.449229 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:47:00.448627 4376 generic.go:356] "Generic (PLEG): container finished" podID="0de4ffc1-c510-4022-b7be-3f8f13360f76" containerID="218edb5fe0fbcf9808d1435b2dd347951d228dba57f508800d6e58852999807e" exitCode=0 Oct 31 04:47:00.451934 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:47:00.451903 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-np274" event={"ID":"0de4ffc1-c510-4022-b7be-3f8f13360f76","Type":"ContainerDied","Data":"218edb5fe0fbcf9808d1435b2dd347951d228dba57f508800d6e58852999807e"} Oct 31 04:47:00.527515 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:47:00.527492 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-np274" Oct 31 04:47:00.600305 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:47:00.600277 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0de4ffc1-c510-4022-b7be-3f8f13360f76-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0de4ffc1-c510-4022-b7be-3f8f13360f76-workload-socket\") pod \"0de4ffc1-c510-4022-b7be-3f8f13360f76\" (UID: \"0de4ffc1-c510-4022-b7be-3f8f13360f76\") " Oct 31 04:47:00.600546 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:47:00.600379 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0de4ffc1-c510-4022-b7be-3f8f13360f76-kube-api-access-7xf8b\" (UniqueName: \"kubernetes.io/projected/0de4ffc1-c510-4022-b7be-3f8f13360f76-kube-api-access-7xf8b\") pod \"0de4ffc1-c510-4022-b7be-3f8f13360f76\" (UID: \"0de4ffc1-c510-4022-b7be-3f8f13360f76\") " Oct 31 04:47:00.600546 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:47:00.600438 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0de4ffc1-c510-4022-b7be-3f8f13360f76-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0de4ffc1-c510-4022-b7be-3f8f13360f76-istio-podinfo\") pod \"0de4ffc1-c510-4022-b7be-3f8f13360f76\" (UID: \"0de4ffc1-c510-4022-b7be-3f8f13360f76\") " Oct 31 04:47:00.600546 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:47:00.600467 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0de4ffc1-c510-4022-b7be-3f8f13360f76-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0de4ffc1-c510-4022-b7be-3f8f13360f76-istiod-ca-cert\") pod \"0de4ffc1-c510-4022-b7be-3f8f13360f76\" (UID: \"0de4ffc1-c510-4022-b7be-3f8f13360f76\") " Oct 31 04:47:00.600736 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:47:00.600552 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0de4ffc1-c510-4022-b7be-3f8f13360f76-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0de4ffc1-c510-4022-b7be-3f8f13360f76-istio-envoy\") pod \"0de4ffc1-c510-4022-b7be-3f8f13360f76\" (UID: \"0de4ffc1-c510-4022-b7be-3f8f13360f76\") " Oct 31 04:47:00.600736 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:47:00.600585 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0de4ffc1-c510-4022-b7be-3f8f13360f76-workload-socket" pod "0de4ffc1-c510-4022-b7be-3f8f13360f76" (UID: "0de4ffc1-c510-4022-b7be-3f8f13360f76"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:47:00.600736 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:47:00.600598 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0de4ffc1-c510-4022-b7be-3f8f13360f76-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0de4ffc1-c510-4022-b7be-3f8f13360f76-workload-certs\") pod \"0de4ffc1-c510-4022-b7be-3f8f13360f76\" (UID: \"0de4ffc1-c510-4022-b7be-3f8f13360f76\") " Oct 31 04:47:00.600736 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:47:00.600637 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0de4ffc1-c510-4022-b7be-3f8f13360f76-istio-token\" (UniqueName: \"kubernetes.io/projected/0de4ffc1-c510-4022-b7be-3f8f13360f76-istio-token\") pod \"0de4ffc1-c510-4022-b7be-3f8f13360f76\" (UID: \"0de4ffc1-c510-4022-b7be-3f8f13360f76\") " Oct 31 04:47:00.600952 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:47:00.600762 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0de4ffc1-c510-4022-b7be-3f8f13360f76-workload-certs" pod "0de4ffc1-c510-4022-b7be-3f8f13360f76" (UID: "0de4ffc1-c510-4022-b7be-3f8f13360f76"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:47:00.600952 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:47:00.600778 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0de4ffc1-c510-4022-b7be-3f8f13360f76-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0de4ffc1-c510-4022-b7be-3f8f13360f76-credential-socket\") pod \"0de4ffc1-c510-4022-b7be-3f8f13360f76\" (UID: \"0de4ffc1-c510-4022-b7be-3f8f13360f76\") " Oct 31 04:47:00.600952 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:47:00.600856 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0de4ffc1-c510-4022-b7be-3f8f13360f76-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0de4ffc1-c510-4022-b7be-3f8f13360f76-istio-data\") pod \"0de4ffc1-c510-4022-b7be-3f8f13360f76\" (UID: \"0de4ffc1-c510-4022-b7be-3f8f13360f76\") " Oct 31 04:47:00.601293 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:47:00.601266 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0de4ffc1-c510-4022-b7be-3f8f13360f76-istiod-ca-cert" pod "0de4ffc1-c510-4022-b7be-3f8f13360f76" (UID: "0de4ffc1-c510-4022-b7be-3f8f13360f76"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:47:00.602056 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:47:00.602034 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0de4ffc1-c510-4022-b7be-3f8f13360f76-credential-socket" pod "0de4ffc1-c510-4022-b7be-3f8f13360f76" (UID: "0de4ffc1-c510-4022-b7be-3f8f13360f76"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:47:00.602172 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:47:00.602044 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0de4ffc1-c510-4022-b7be-3f8f13360f76-istio-data" pod "0de4ffc1-c510-4022-b7be-3f8f13360f76" (UID: "0de4ffc1-c510-4022-b7be-3f8f13360f76"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:47:00.602262 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:47:00.602042 4376 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0de4ffc1-c510-4022-b7be-3f8f13360f76-workload-socket\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:47:00.602418 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:47:00.602400 4376 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0de4ffc1-c510-4022-b7be-3f8f13360f76-workload-certs\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:47:00.603747 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:47:00.603717 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0de4ffc1-c510-4022-b7be-3f8f13360f76-istio-token" pod "0de4ffc1-c510-4022-b7be-3f8f13360f76" (UID: "0de4ffc1-c510-4022-b7be-3f8f13360f76"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:47:00.609471 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:47:00.609443 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0de4ffc1-c510-4022-b7be-3f8f13360f76-istio-envoy" pod "0de4ffc1-c510-4022-b7be-3f8f13360f76" (UID: "0de4ffc1-c510-4022-b7be-3f8f13360f76"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:47:00.609579 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:47:00.609453 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0de4ffc1-c510-4022-b7be-3f8f13360f76-istio-podinfo" pod "0de4ffc1-c510-4022-b7be-3f8f13360f76" (UID: "0de4ffc1-c510-4022-b7be-3f8f13360f76"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 04:47:00.609579 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:47:00.609493 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0de4ffc1-c510-4022-b7be-3f8f13360f76-kube-api-access-7xf8b" pod "0de4ffc1-c510-4022-b7be-3f8f13360f76" (UID: "0de4ffc1-c510-4022-b7be-3f8f13360f76"). InnerVolumeSpecName "kube-api-access-7xf8b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:47:00.703947 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:47:00.703915 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7xf8b\" (UniqueName: \"kubernetes.io/projected/0de4ffc1-c510-4022-b7be-3f8f13360f76-kube-api-access-7xf8b\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:47:00.703947 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:47:00.703946 4376 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0de4ffc1-c510-4022-b7be-3f8f13360f76-istio-podinfo\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:47:00.704144 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:47:00.703957 4376 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0de4ffc1-c510-4022-b7be-3f8f13360f76-istiod-ca-cert\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:47:00.704144 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:47:00.703966 4376 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0de4ffc1-c510-4022-b7be-3f8f13360f76-istio-envoy\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:47:00.704144 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:47:00.703974 4376 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0de4ffc1-c510-4022-b7be-3f8f13360f76-istio-token\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:47:00.704144 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:47:00.703984 4376 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0de4ffc1-c510-4022-b7be-3f8f13360f76-credential-socket\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:47:00.704144 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:47:00.703992 4376 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0de4ffc1-c510-4022-b7be-3f8f13360f76-istio-data\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:47:01.456979 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:47:01.456948 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-np274" Oct 31 04:47:01.456979 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:47:01.456973 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-np274" event={"ID":"0de4ffc1-c510-4022-b7be-3f8f13360f76","Type":"ContainerDied","Data":"4d14ca7e5d16d3e77c692c07da3e3bd77db11f1f3b84e1460e1680b078dc5b22"} Oct 31 04:47:01.457548 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:47:01.457020 4376 scope.go:122] "RemoveContainer" containerID="218edb5fe0fbcf9808d1435b2dd347951d228dba57f508800d6e58852999807e" Oct 31 04:47:01.527304 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:47:01.527276 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-np274"] Oct 31 04:47:01.541182 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:47:01.541162 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-np274"] Oct 31 04:47:02.445448 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:47:02.445418 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0de4ffc1-c510-4022-b7be-3f8f13360f76" path="/var/lib/kubelet/pods/0de4ffc1-c510-4022-b7be-3f8f13360f76/volumes" Oct 31 04:47:10.999759 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:47:10.999723 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 04:47:18.227412 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:47:18.227375 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-b679dtfm4"] Oct 31 04:47:21.451558 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:47:21.451519 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 04:47:23.228024 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:47:23.227987 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-b679dtfm4" podUID="1110d483-5331-4844-8cff-c42f3b132678" containerName="payload-processing" containerID="cri-o://5161f219e78d8ccf736ecb8ef9538e840310aee402b5f3652fcdfe1b8f84769b" gracePeriod=25 Oct 31 04:47:23.490517 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:47:23.490455 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-b679dtfm4" Oct 31 04:47:23.619182 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:47:23.619151 4376 generic.go:356] "Generic (PLEG): container finished" podID="1110d483-5331-4844-8cff-c42f3b132678" containerID="5161f219e78d8ccf736ecb8ef9538e840310aee402b5f3652fcdfe1b8f84769b" exitCode=0 Oct 31 04:47:23.619359 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:47:23.619196 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-b679dtfm4" Oct 31 04:47:23.619359 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:47:23.619233 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-b679dtfm4" event={"ID":"1110d483-5331-4844-8cff-c42f3b132678","Type":"ContainerDied","Data":"5161f219e78d8ccf736ecb8ef9538e840310aee402b5f3652fcdfe1b8f84769b"} Oct 31 04:47:23.619359 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:47:23.619286 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-b679dtfm4" event={"ID":"1110d483-5331-4844-8cff-c42f3b132678","Type":"ContainerDied","Data":"f2f6bc399d5795f0b6334dd726513541f72a3a3b443b7704b12dba28a09ace1d"} Oct 31 04:47:23.619359 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:47:23.619307 4376 scope.go:122] "RemoveContainer" containerID="5161f219e78d8ccf736ecb8ef9538e840310aee402b5f3652fcdfe1b8f84769b" Oct 31 04:47:23.628857 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:47:23.628840 4376 scope.go:122] "RemoveContainer" containerID="5161f219e78d8ccf736ecb8ef9538e840310aee402b5f3652fcdfe1b8f84769b" Oct 31 04:47:23.629132 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:47:23.629110 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5161f219e78d8ccf736ecb8ef9538e840310aee402b5f3652fcdfe1b8f84769b\": container with ID starting with 5161f219e78d8ccf736ecb8ef9538e840310aee402b5f3652fcdfe1b8f84769b not found: ID does not exist" containerID="5161f219e78d8ccf736ecb8ef9538e840310aee402b5f3652fcdfe1b8f84769b" Oct 31 04:47:23.629240 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:47:23.629137 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5161f219e78d8ccf736ecb8ef9538e840310aee402b5f3652fcdfe1b8f84769b"} err="failed to get container status \"5161f219e78d8ccf736ecb8ef9538e840310aee402b5f3652fcdfe1b8f84769b\": rpc error: code = NotFound desc = could not find container \"5161f219e78d8ccf736ecb8ef9538e840310aee402b5f3652fcdfe1b8f84769b\": container with ID starting with 5161f219e78d8ccf736ecb8ef9538e840310aee402b5f3652fcdfe1b8f84769b not found: ID does not exist" Oct 31 04:47:23.650637 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:47:23.650611 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1110d483-5331-4844-8cff-c42f3b132678-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1110d483-5331-4844-8cff-c42f3b132678-plugins-config-volume\") pod \"1110d483-5331-4844-8cff-c42f3b132678\" (UID: \"1110d483-5331-4844-8cff-c42f3b132678\") " Oct 31 04:47:23.650810 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:47:23.650700 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1110d483-5331-4844-8cff-c42f3b132678-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1110d483-5331-4844-8cff-c42f3b132678-routing-overlay-volume\") pod \"1110d483-5331-4844-8cff-c42f3b132678\" (UID: \"1110d483-5331-4844-8cff-c42f3b132678\") " Oct 31 04:47:23.650810 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:47:23.650801 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1110d483-5331-4844-8cff-c42f3b132678-provider-credentials\" (UniqueName: \"kubernetes.io/projected/1110d483-5331-4844-8cff-c42f3b132678-provider-credentials\") pod \"1110d483-5331-4844-8cff-c42f3b132678\" (UID: \"1110d483-5331-4844-8cff-c42f3b132678\") " Oct 31 04:47:23.650952 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:47:23.650937 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1110d483-5331-4844-8cff-c42f3b132678-kube-api-access-xqkjb\" (UniqueName: \"kubernetes.io/projected/1110d483-5331-4844-8cff-c42f3b132678-kube-api-access-xqkjb\") pod \"1110d483-5331-4844-8cff-c42f3b132678\" (UID: \"1110d483-5331-4844-8cff-c42f3b132678\") " Oct 31 04:47:23.651090 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:47:23.651068 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1110d483-5331-4844-8cff-c42f3b132678-plugins-config-volume" pod "1110d483-5331-4844-8cff-c42f3b132678" (UID: "1110d483-5331-4844-8cff-c42f3b132678"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:47:23.651142 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:47:23.651109 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1110d483-5331-4844-8cff-c42f3b132678-routing-overlay-volume" pod "1110d483-5331-4844-8cff-c42f3b132678" (UID: "1110d483-5331-4844-8cff-c42f3b132678"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:47:23.651503 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:47:23.651481 4376 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1110d483-5331-4844-8cff-c42f3b132678-plugins-config-volume\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:47:23.651503 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:47:23.651502 4376 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1110d483-5331-4844-8cff-c42f3b132678-routing-overlay-volume\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:47:23.652935 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:47:23.652907 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1110d483-5331-4844-8cff-c42f3b132678-provider-credentials" pod "1110d483-5331-4844-8cff-c42f3b132678" (UID: "1110d483-5331-4844-8cff-c42f3b132678"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:47:23.653040 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:47:23.652984 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1110d483-5331-4844-8cff-c42f3b132678-kube-api-access-xqkjb" pod "1110d483-5331-4844-8cff-c42f3b132678" (UID: "1110d483-5331-4844-8cff-c42f3b132678"). InnerVolumeSpecName "kube-api-access-xqkjb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:47:23.752638 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:47:23.752574 4376 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1110d483-5331-4844-8cff-c42f3b132678-provider-credentials\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:47:23.752638 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:47:23.752597 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xqkjb\" (UniqueName: \"kubernetes.io/projected/1110d483-5331-4844-8cff-c42f3b132678-kube-api-access-xqkjb\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:47:23.968412 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:47:23.968377 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-b679dtfm4"] Oct 31 04:47:23.971843 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:47:23.971815 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-b679dtfm4"] Oct 31 04:47:24.444791 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:47:24.444754 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1110d483-5331-4844-8cff-c42f3b132678" path="/var/lib/kubelet/pods/1110d483-5331-4844-8cff-c42f3b132678/volumes" Oct 31 04:47:39.074590 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:47:39.074554 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-78997b54j"] Oct 31 04:47:39.075312 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:47:39.075296 4376 cpu_manager.go:414] "containerMap: removing container" podUID="0de4ffc1-c510-4022-b7be-3f8f13360f76" containerName="istio-proxy" Oct 31 04:47:39.075386 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:47:39.075314 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0de4ffc1-c510-4022-b7be-3f8f13360f76" containerName="istio-proxy" Oct 31 04:47:39.075386 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:47:39.075345 4376 cpu_manager.go:414] "containerMap: removing container" podUID="7c143dfc-5cce-460b-b348-c5ed65ab662a" containerName="payload-processing" Oct 31 04:47:39.075386 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:47:39.075351 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c143dfc-5cce-460b-b348-c5ed65ab662a" containerName="payload-processing" Oct 31 04:47:39.075386 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:47:39.075381 4376 cpu_manager.go:414] "containerMap: removing container" podUID="2608ed82-e75d-41bf-8710-0faf0ce0ae08" containerName="cleanup" Oct 31 04:47:39.075386 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:47:39.075385 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2608ed82-e75d-41bf-8710-0faf0ce0ae08" containerName="cleanup" Oct 31 04:47:39.075549 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:47:39.075402 4376 cpu_manager.go:414] "containerMap: removing container" podUID="18cae0c8-c13a-4ebe-9c1d-3f87232d46fb" containerName="cleanup" Oct 31 04:47:39.075549 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:47:39.075407 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18cae0c8-c13a-4ebe-9c1d-3f87232d46fb" containerName="cleanup" Oct 31 04:47:39.075549 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:47:39.075417 4376 cpu_manager.go:414] "containerMap: removing container" podUID="fbee2356-f532-40d7-90b6-2572284f0b55" containerName="cleanup" Oct 31 04:47:39.075549 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:47:39.075421 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbee2356-f532-40d7-90b6-2572284f0b55" containerName="cleanup" Oct 31 04:47:39.075549 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:47:39.075427 4376 cpu_manager.go:414] "containerMap: removing container" podUID="1110d483-5331-4844-8cff-c42f3b132678" containerName="payload-processing" Oct 31 04:47:39.075549 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:47:39.075432 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1110d483-5331-4844-8cff-c42f3b132678" containerName="payload-processing" Oct 31 04:47:39.075549 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:47:39.075525 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="0de4ffc1-c510-4022-b7be-3f8f13360f76" containerName="istio-proxy" Oct 31 04:47:39.075549 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:47:39.075534 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c143dfc-5cce-460b-b348-c5ed65ab662a" containerName="payload-processing" Oct 31 04:47:39.075549 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:47:39.075540 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbee2356-f532-40d7-90b6-2572284f0b55" containerName="cleanup" Oct 31 04:47:39.075816 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:47:39.075553 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="1110d483-5331-4844-8cff-c42f3b132678" containerName="payload-processing" Oct 31 04:47:39.080638 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:47:39.080620 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-78997b54j" Oct 31 04:47:39.096301 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:47:39.096280 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-401761\"/\"openshift-service-ca.crt\"" Oct 31 04:47:39.096557 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:47:39.096286 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-401761\"/\"kube-root-ca.crt\"" Oct 31 04:47:39.100165 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:47:39.100141 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-78997b54j"] Oct 31 04:47:39.108365 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:47:39.108262 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-401761\"/\"agc-cred-512fbb167122\"" Oct 31 04:47:39.108365 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:47:39.108284 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-401761\"/\"payload-processing-external-model-plugins-e2e-extmod-a-401761\"" Oct 31 04:47:39.108365 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:47:39.108354 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-401761\"/\"payload-processing-external-model-e2e--201094b0-dockercfg-bbt5w\"" Oct 31 04:47:39.108610 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:47:39.108438 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-401761\"/\"routing-overlay\"" Oct 31 04:47:39.179854 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:47:39.179811 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4adf9b20-4ae7-4704-b299-8dbebd368bb9-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-401761-78997b54j\" (UID: \"4adf9b20-4ae7-4704-b299-8dbebd368bb9\") " pod="ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-78997b54j" Oct 31 04:47:39.180053 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:47:39.179897 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdshm\" (UniqueName: \"kubernetes.io/projected/4adf9b20-4ae7-4704-b299-8dbebd368bb9-kube-api-access-kdshm\") pod \"payload-processing-external-model-e2e-extmod-a-401761-78997b54j\" (UID: \"4adf9b20-4ae7-4704-b299-8dbebd368bb9\") " pod="ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-78997b54j" Oct 31 04:47:39.180053 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:47:39.180045 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4adf9b20-4ae7-4704-b299-8dbebd368bb9-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-401761-78997b54j\" (UID: \"4adf9b20-4ae7-4704-b299-8dbebd368bb9\") " pod="ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-78997b54j" Oct 31 04:47:39.180198 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:47:39.180095 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4adf9b20-4ae7-4704-b299-8dbebd368bb9-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-401761-78997b54j\" (UID: \"4adf9b20-4ae7-4704-b299-8dbebd368bb9\") " pod="ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-78997b54j" Oct 31 04:47:39.281647 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:47:39.281607 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4adf9b20-4ae7-4704-b299-8dbebd368bb9-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-401761-78997b54j\" (UID: \"4adf9b20-4ae7-4704-b299-8dbebd368bb9\") " pod="ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-78997b54j" Oct 31 04:47:39.288233 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:47:39.285530 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdshm\" (UniqueName: \"kubernetes.io/projected/4adf9b20-4ae7-4704-b299-8dbebd368bb9-kube-api-access-kdshm\") pod \"payload-processing-external-model-e2e-extmod-a-401761-78997b54j\" (UID: \"4adf9b20-4ae7-4704-b299-8dbebd368bb9\") " pod="ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-78997b54j" Oct 31 04:47:39.288233 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:47:39.285745 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4adf9b20-4ae7-4704-b299-8dbebd368bb9-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-401761-78997b54j\" (UID: \"4adf9b20-4ae7-4704-b299-8dbebd368bb9\") " pod="ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-78997b54j" Oct 31 04:47:39.288233 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:47:39.285816 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4adf9b20-4ae7-4704-b299-8dbebd368bb9-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-401761-78997b54j\" (UID: \"4adf9b20-4ae7-4704-b299-8dbebd368bb9\") " pod="ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-78997b54j" Oct 31 04:47:39.288233 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:47:39.286061 4376 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-401761/agc-cred-512fbb167122: secret "agc-cred-512fbb167122" not found Oct 31 04:47:39.288233 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:47:39.286079 4376 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-78997b54j: secret "agc-cred-512fbb167122" not found Oct 31 04:47:39.288233 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:47:39.286165 4376 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4adf9b20-4ae7-4704-b299-8dbebd368bb9-provider-credentials podName:4adf9b20-4ae7-4704-b299-8dbebd368bb9 nodeName:}" failed. No retries permitted until 2026-10-31 04:47:39.786143665 +0000 UTC m=+2607.500478259 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/4adf9b20-4ae7-4704-b299-8dbebd368bb9-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-401761-78997b54j" (UID: "4adf9b20-4ae7-4704-b299-8dbebd368bb9") : secret "agc-cred-512fbb167122" not found Oct 31 04:47:39.288233 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:47:39.283233 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4adf9b20-4ae7-4704-b299-8dbebd368bb9-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-401761-78997b54j\" (UID: \"4adf9b20-4ae7-4704-b299-8dbebd368bb9\") " pod="ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-78997b54j" Oct 31 04:47:39.288233 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:47:39.287906 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4adf9b20-4ae7-4704-b299-8dbebd368bb9-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-401761-78997b54j\" (UID: \"4adf9b20-4ae7-4704-b299-8dbebd368bb9\") " pod="ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-78997b54j" Oct 31 04:47:39.316873 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:47:39.316813 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdshm\" (UniqueName: \"kubernetes.io/projected/4adf9b20-4ae7-4704-b299-8dbebd368bb9-kube-api-access-kdshm\") pod \"payload-processing-external-model-e2e-extmod-a-401761-78997b54j\" (UID: \"4adf9b20-4ae7-4704-b299-8dbebd368bb9\") " pod="ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-78997b54j" Oct 31 04:47:39.549389 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:47:39.549356 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-78997b54j"] Oct 31 04:47:39.549747 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:47:39.549725 4376 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[provider-credentials], unattached volumes=[], failed to process volumes=[]: context canceled" pod="ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-78997b54j" podUID="4adf9b20-4ae7-4704-b299-8dbebd368bb9" Oct 31 04:47:39.739708 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:47:39.739678 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-78997b54j" Oct 31 04:47:39.748199 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:47:39.748178 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-78997b54j" Oct 31 04:47:39.794243 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:47:39.794215 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4adf9b20-4ae7-4704-b299-8dbebd368bb9-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4adf9b20-4ae7-4704-b299-8dbebd368bb9-routing-overlay-volume\") pod \"4adf9b20-4ae7-4704-b299-8dbebd368bb9\" (UID: \"4adf9b20-4ae7-4704-b299-8dbebd368bb9\") " Oct 31 04:47:39.794494 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:47:39.794470 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4adf9b20-4ae7-4704-b299-8dbebd368bb9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4adf9b20-4ae7-4704-b299-8dbebd368bb9-plugins-config-volume\") pod \"4adf9b20-4ae7-4704-b299-8dbebd368bb9\" (UID: \"4adf9b20-4ae7-4704-b299-8dbebd368bb9\") " Oct 31 04:47:39.794646 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:47:39.794579 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4adf9b20-4ae7-4704-b299-8dbebd368bb9-routing-overlay-volume" pod "4adf9b20-4ae7-4704-b299-8dbebd368bb9" (UID: "4adf9b20-4ae7-4704-b299-8dbebd368bb9"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:47:39.794646 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:47:39.794586 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4adf9b20-4ae7-4704-b299-8dbebd368bb9-kube-api-access-kdshm\" (UniqueName: \"kubernetes.io/projected/4adf9b20-4ae7-4704-b299-8dbebd368bb9-kube-api-access-kdshm\") pod \"4adf9b20-4ae7-4704-b299-8dbebd368bb9\" (UID: \"4adf9b20-4ae7-4704-b299-8dbebd368bb9\") " Oct 31 04:47:39.794883 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:47:39.794864 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4adf9b20-4ae7-4704-b299-8dbebd368bb9-plugins-config-volume" pod "4adf9b20-4ae7-4704-b299-8dbebd368bb9" (UID: "4adf9b20-4ae7-4704-b299-8dbebd368bb9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:47:39.795294 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:47:39.795266 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4adf9b20-4ae7-4704-b299-8dbebd368bb9-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-401761-78997b54j\" (UID: \"4adf9b20-4ae7-4704-b299-8dbebd368bb9\") " pod="ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-78997b54j" Oct 31 04:47:39.795432 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:47:39.795418 4376 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4adf9b20-4ae7-4704-b299-8dbebd368bb9-routing-overlay-volume\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:47:39.795489 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:47:39.795433 4376 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4adf9b20-4ae7-4704-b299-8dbebd368bb9-plugins-config-volume\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:47:39.795489 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:47:39.795444 4376 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-401761/agc-cred-512fbb167122: secret "agc-cred-512fbb167122" not found Oct 31 04:47:39.795489 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:47:39.795461 4376 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-78997b54j: secret "agc-cred-512fbb167122" not found Oct 31 04:47:39.795605 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:47:39.795515 4376 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4adf9b20-4ae7-4704-b299-8dbebd368bb9-provider-credentials podName:4adf9b20-4ae7-4704-b299-8dbebd368bb9 nodeName:}" failed. No retries permitted until 2026-10-31 04:47:40.795500731 +0000 UTC m=+2608.509835338 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/4adf9b20-4ae7-4704-b299-8dbebd368bb9-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-401761-78997b54j" (UID: "4adf9b20-4ae7-4704-b299-8dbebd368bb9") : secret "agc-cred-512fbb167122" not found Oct 31 04:47:39.797168 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:47:39.797147 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4adf9b20-4ae7-4704-b299-8dbebd368bb9-kube-api-access-kdshm" pod "4adf9b20-4ae7-4704-b299-8dbebd368bb9" (UID: "4adf9b20-4ae7-4704-b299-8dbebd368bb9"). InnerVolumeSpecName "kube-api-access-kdshm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:47:39.906389 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:47:39.896873 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdshm\" (UniqueName: \"kubernetes.io/projected/4adf9b20-4ae7-4704-b299-8dbebd368bb9-kube-api-access-kdshm\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:47:40.746945 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:47:40.746869 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-78997b54j" Oct 31 04:47:40.808024 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:47:40.807988 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4adf9b20-4ae7-4704-b299-8dbebd368bb9-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-401761-78997b54j\" (UID: \"4adf9b20-4ae7-4704-b299-8dbebd368bb9\") " pod="ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-78997b54j" Oct 31 04:47:40.808183 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:47:40.808136 4376 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-401761/agc-cred-512fbb167122: object "ai-tenant-e2e-extmod-a-401761"/"agc-cred-512fbb167122" not registered Oct 31 04:47:40.808183 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:47:40.808152 4376 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-78997b54j: object "ai-tenant-e2e-extmod-a-401761"/"agc-cred-512fbb167122" not registered Oct 31 04:47:40.808256 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:47:40.808212 4376 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4adf9b20-4ae7-4704-b299-8dbebd368bb9-provider-credentials podName:4adf9b20-4ae7-4704-b299-8dbebd368bb9 nodeName:}" failed. No retries permitted until 2026-10-31 04:47:42.808195302 +0000 UTC m=+2610.522529913 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/4adf9b20-4ae7-4704-b299-8dbebd368bb9-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-401761-78997b54j" (UID: "4adf9b20-4ae7-4704-b299-8dbebd368bb9") : object "ai-tenant-e2e-extmod-a-401761"/"agc-cred-512fbb167122" not registered Oct 31 04:47:40.831448 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:47:40.831365 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-78997b54j"] Oct 31 04:47:40.859790 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:47:40.859759 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-78997b54j"] Oct 31 04:47:40.909202 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:47:40.909173 4376 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4adf9b20-4ae7-4704-b299-8dbebd368bb9-provider-credentials\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:47:41.005594 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:47:41.005495 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 04:47:42.448509 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:47:42.448468 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4adf9b20-4ae7-4704-b299-8dbebd368bb9" path="/var/lib/kubelet/pods/4adf9b20-4ae7-4704-b299-8dbebd368bb9/volumes" Oct 31 04:47:55.653815 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:47:55.653780 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-766b99cdcf-m2gpm"] Oct 31 04:47:55.660565 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:47:55.660539 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-766b99cdcf-m2gpm" Oct 31 04:47:55.674254 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:47:55.674222 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-766b99cdcf-m2gpm"] Oct 31 04:47:55.679762 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:47:55.679737 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-edd2b7d908c7\"" Oct 31 04:47:55.679946 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:47:55.679827 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-r9vnq\"" Oct 31 04:47:55.679946 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:47:55.679942 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Oct 31 04:47:55.689531 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:47:55.689511 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Oct 31 04:47:55.689531 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:47:55.689523 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Oct 31 04:47:55.689671 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:47:55.689553 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Oct 31 04:47:55.734455 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:47:55.734426 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cf93b9cd-4bcd-49dd-8861-745aef0e9518-provider-credentials\") pod \"payload-processing-external-model-766b99cdcf-m2gpm\" (UID: \"cf93b9cd-4bcd-49dd-8861-745aef0e9518\") " pod="models-as-a-service/payload-processing-external-model-766b99cdcf-m2gpm" Oct 31 04:47:55.734592 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:47:55.734532 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf93b9cd-4bcd-49dd-8861-745aef0e9518-plugins-config-volume\") pod \"payload-processing-external-model-766b99cdcf-m2gpm\" (UID: \"cf93b9cd-4bcd-49dd-8861-745aef0e9518\") " pod="models-as-a-service/payload-processing-external-model-766b99cdcf-m2gpm" Oct 31 04:47:55.734633 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:47:55.734588 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gshpm\" (UniqueName: \"kubernetes.io/projected/cf93b9cd-4bcd-49dd-8861-745aef0e9518-kube-api-access-gshpm\") pod \"payload-processing-external-model-766b99cdcf-m2gpm\" (UID: \"cf93b9cd-4bcd-49dd-8861-745aef0e9518\") " pod="models-as-a-service/payload-processing-external-model-766b99cdcf-m2gpm" Oct 31 04:47:55.734724 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:47:55.734707 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cf93b9cd-4bcd-49dd-8861-745aef0e9518-routing-overlay-volume\") pod \"payload-processing-external-model-766b99cdcf-m2gpm\" (UID: \"cf93b9cd-4bcd-49dd-8861-745aef0e9518\") " pod="models-as-a-service/payload-processing-external-model-766b99cdcf-m2gpm" Oct 31 04:47:55.838207 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:47:55.838165 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cf93b9cd-4bcd-49dd-8861-745aef0e9518-routing-overlay-volume\") pod \"payload-processing-external-model-766b99cdcf-m2gpm\" (UID: \"cf93b9cd-4bcd-49dd-8861-745aef0e9518\") " pod="models-as-a-service/payload-processing-external-model-766b99cdcf-m2gpm" Oct 31 04:47:55.838551 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:47:55.838526 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cf93b9cd-4bcd-49dd-8861-745aef0e9518-provider-credentials\") pod \"payload-processing-external-model-766b99cdcf-m2gpm\" (UID: \"cf93b9cd-4bcd-49dd-8861-745aef0e9518\") " pod="models-as-a-service/payload-processing-external-model-766b99cdcf-m2gpm" Oct 31 04:47:55.838744 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:47:55.838730 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf93b9cd-4bcd-49dd-8861-745aef0e9518-plugins-config-volume\") pod \"payload-processing-external-model-766b99cdcf-m2gpm\" (UID: \"cf93b9cd-4bcd-49dd-8861-745aef0e9518\") " pod="models-as-a-service/payload-processing-external-model-766b99cdcf-m2gpm" Oct 31 04:47:55.838872 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:47:55.838859 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gshpm\" (UniqueName: \"kubernetes.io/projected/cf93b9cd-4bcd-49dd-8861-745aef0e9518-kube-api-access-gshpm\") pod \"payload-processing-external-model-766b99cdcf-m2gpm\" (UID: \"cf93b9cd-4bcd-49dd-8861-745aef0e9518\") " pod="models-as-a-service/payload-processing-external-model-766b99cdcf-m2gpm" Oct 31 04:47:55.840031 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:47:55.840007 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cf93b9cd-4bcd-49dd-8861-745aef0e9518-routing-overlay-volume\") pod \"payload-processing-external-model-766b99cdcf-m2gpm\" (UID: \"cf93b9cd-4bcd-49dd-8861-745aef0e9518\") " pod="models-as-a-service/payload-processing-external-model-766b99cdcf-m2gpm" Oct 31 04:47:55.848437 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:47:55.841521 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf93b9cd-4bcd-49dd-8861-745aef0e9518-plugins-config-volume\") pod \"payload-processing-external-model-766b99cdcf-m2gpm\" (UID: \"cf93b9cd-4bcd-49dd-8861-745aef0e9518\") " pod="models-as-a-service/payload-processing-external-model-766b99cdcf-m2gpm" Oct 31 04:47:55.893404 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:47:55.882578 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gshpm\" (UniqueName: \"kubernetes.io/projected/cf93b9cd-4bcd-49dd-8861-745aef0e9518-kube-api-access-gshpm\") pod \"payload-processing-external-model-766b99cdcf-m2gpm\" (UID: \"cf93b9cd-4bcd-49dd-8861-745aef0e9518\") " pod="models-as-a-service/payload-processing-external-model-766b99cdcf-m2gpm" Oct 31 04:47:55.894640 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:47:55.894611 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cf93b9cd-4bcd-49dd-8861-745aef0e9518-provider-credentials\") pod \"payload-processing-external-model-766b99cdcf-m2gpm\" (UID: \"cf93b9cd-4bcd-49dd-8861-745aef0e9518\") " pod="models-as-a-service/payload-processing-external-model-766b99cdcf-m2gpm" Oct 31 04:47:55.978516 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:47:55.978444 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-766b99cdcf-m2gpm" Oct 31 04:47:56.286783 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:47:56.286750 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf93b9cd_4bcd_49dd_8861_745aef0e9518.slice/crio-e18947b83149bc4c92f1cc54f08adadc0b520abbd21a57b0259186754e8daf3b WatchSource:0}: Error finding container e18947b83149bc4c92f1cc54f08adadc0b520abbd21a57b0259186754e8daf3b: Status 404 returned error can't find the container with id e18947b83149bc4c92f1cc54f08adadc0b520abbd21a57b0259186754e8daf3b Oct 31 04:47:56.288440 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:47:56.288417 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-766b99cdcf-m2gpm"] Oct 31 04:47:56.966890 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:47:56.966853 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-766b99cdcf-m2gpm" event={"ID":"cf93b9cd-4bcd-49dd-8861-745aef0e9518","Type":"ContainerStarted","Data":"42eca02da2609fb829432100e1a3d36682bae935fe1c09cbb753635d36814b1b"} Oct 31 04:47:56.967450 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:47:56.966911 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-766b99cdcf-m2gpm" event={"ID":"cf93b9cd-4bcd-49dd-8861-745aef0e9518","Type":"ContainerStarted","Data":"e18947b83149bc4c92f1cc54f08adadc0b520abbd21a57b0259186754e8daf3b"} Oct 31 04:47:56.967450 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:47:56.966940 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-766b99cdcf-m2gpm" Oct 31 04:47:57.010764 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:47:57.010717 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-766b99cdcf-m2gpm" podStartSLOduration=2.010704576 podStartE2EDuration="2.010704576s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:47:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:47:57.00783087 +0000 UTC m=+2624.722165524" watchObservedRunningTime="2026-10-31 04:47:57.010704576 +0000 UTC m=+2624.725039192" Oct 31 04:48:07.976763 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:07.976736 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-766b99cdcf-m2gpm" Oct 31 04:48:10.260855 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:10.260802 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-47c409-openshift-default-796cbb7889-drd46"] Oct 31 04:48:10.265828 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:10.265808 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-47c409-openshift-default-796cbb7889-drd46" Oct 31 04:48:10.281484 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:10.281464 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-47c409-openshift-default-dockercfg-thhmx\"" Oct 31 04:48:10.299620 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:10.299593 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-47c409-openshift-default-796cbb7889-drd46"] Oct 31 04:48:10.404755 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:10.404727 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6qv7\" (UniqueName: \"kubernetes.io/projected/d76a0ace-f020-430f-a26c-005231cb8be2-kube-api-access-k6qv7\") pod \"e2e-extmod-a-47c409-openshift-default-796cbb7889-drd46\" (UID: \"d76a0ace-f020-430f-a26c-005231cb8be2\") " pod="openshift-ingress/e2e-extmod-a-47c409-openshift-default-796cbb7889-drd46" Oct 31 04:48:10.404941 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:10.404767 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d76a0ace-f020-430f-a26c-005231cb8be2-istio-podinfo\") pod \"e2e-extmod-a-47c409-openshift-default-796cbb7889-drd46\" (UID: \"d76a0ace-f020-430f-a26c-005231cb8be2\") " pod="openshift-ingress/e2e-extmod-a-47c409-openshift-default-796cbb7889-drd46" Oct 31 04:48:10.404941 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:10.404904 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d76a0ace-f020-430f-a26c-005231cb8be2-istiod-ca-cert\") pod \"e2e-extmod-a-47c409-openshift-default-796cbb7889-drd46\" (UID: \"d76a0ace-f020-430f-a26c-005231cb8be2\") " pod="openshift-ingress/e2e-extmod-a-47c409-openshift-default-796cbb7889-drd46" Oct 31 04:48:10.405076 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:10.405002 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d76a0ace-f020-430f-a26c-005231cb8be2-workload-certs\") pod \"e2e-extmod-a-47c409-openshift-default-796cbb7889-drd46\" (UID: \"d76a0ace-f020-430f-a26c-005231cb8be2\") " pod="openshift-ingress/e2e-extmod-a-47c409-openshift-default-796cbb7889-drd46" Oct 31 04:48:10.405133 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:10.405092 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d76a0ace-f020-430f-a26c-005231cb8be2-credential-socket\") pod \"e2e-extmod-a-47c409-openshift-default-796cbb7889-drd46\" (UID: \"d76a0ace-f020-430f-a26c-005231cb8be2\") " pod="openshift-ingress/e2e-extmod-a-47c409-openshift-default-796cbb7889-drd46" Oct 31 04:48:10.405191 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:10.405158 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d76a0ace-f020-430f-a26c-005231cb8be2-istio-token\") pod \"e2e-extmod-a-47c409-openshift-default-796cbb7889-drd46\" (UID: \"d76a0ace-f020-430f-a26c-005231cb8be2\") " pod="openshift-ingress/e2e-extmod-a-47c409-openshift-default-796cbb7889-drd46" Oct 31 04:48:10.405191 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:10.405184 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d76a0ace-f020-430f-a26c-005231cb8be2-istio-envoy\") pod \"e2e-extmod-a-47c409-openshift-default-796cbb7889-drd46\" (UID: \"d76a0ace-f020-430f-a26c-005231cb8be2\") " pod="openshift-ingress/e2e-extmod-a-47c409-openshift-default-796cbb7889-drd46" Oct 31 04:48:10.405324 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:10.405311 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d76a0ace-f020-430f-a26c-005231cb8be2-workload-socket\") pod \"e2e-extmod-a-47c409-openshift-default-796cbb7889-drd46\" (UID: \"d76a0ace-f020-430f-a26c-005231cb8be2\") " pod="openshift-ingress/e2e-extmod-a-47c409-openshift-default-796cbb7889-drd46" Oct 31 04:48:10.405413 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:10.405354 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d76a0ace-f020-430f-a26c-005231cb8be2-istio-data\") pod \"e2e-extmod-a-47c409-openshift-default-796cbb7889-drd46\" (UID: \"d76a0ace-f020-430f-a26c-005231cb8be2\") " pod="openshift-ingress/e2e-extmod-a-47c409-openshift-default-796cbb7889-drd46" Oct 31 04:48:10.507254 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:10.507212 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d76a0ace-f020-430f-a26c-005231cb8be2-istiod-ca-cert\") pod \"e2e-extmod-a-47c409-openshift-default-796cbb7889-drd46\" (UID: \"d76a0ace-f020-430f-a26c-005231cb8be2\") " pod="openshift-ingress/e2e-extmod-a-47c409-openshift-default-796cbb7889-drd46" Oct 31 04:48:10.507470 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:10.507371 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d76a0ace-f020-430f-a26c-005231cb8be2-workload-certs\") pod \"e2e-extmod-a-47c409-openshift-default-796cbb7889-drd46\" (UID: \"d76a0ace-f020-430f-a26c-005231cb8be2\") " pod="openshift-ingress/e2e-extmod-a-47c409-openshift-default-796cbb7889-drd46" Oct 31 04:48:10.507589 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:10.507470 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d76a0ace-f020-430f-a26c-005231cb8be2-credential-socket\") pod \"e2e-extmod-a-47c409-openshift-default-796cbb7889-drd46\" (UID: \"d76a0ace-f020-430f-a26c-005231cb8be2\") " pod="openshift-ingress/e2e-extmod-a-47c409-openshift-default-796cbb7889-drd46" Oct 31 04:48:10.507589 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:10.507558 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d76a0ace-f020-430f-a26c-005231cb8be2-istio-token\") pod \"e2e-extmod-a-47c409-openshift-default-796cbb7889-drd46\" (UID: \"d76a0ace-f020-430f-a26c-005231cb8be2\") " pod="openshift-ingress/e2e-extmod-a-47c409-openshift-default-796cbb7889-drd46" Oct 31 04:48:10.507589 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:10.507583 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d76a0ace-f020-430f-a26c-005231cb8be2-istio-envoy\") pod \"e2e-extmod-a-47c409-openshift-default-796cbb7889-drd46\" (UID: \"d76a0ace-f020-430f-a26c-005231cb8be2\") " pod="openshift-ingress/e2e-extmod-a-47c409-openshift-default-796cbb7889-drd46" Oct 31 04:48:10.507912 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:10.507707 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d76a0ace-f020-430f-a26c-005231cb8be2-workload-socket\") pod \"e2e-extmod-a-47c409-openshift-default-796cbb7889-drd46\" (UID: \"d76a0ace-f020-430f-a26c-005231cb8be2\") " pod="openshift-ingress/e2e-extmod-a-47c409-openshift-default-796cbb7889-drd46" Oct 31 04:48:10.507912 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:10.507742 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d76a0ace-f020-430f-a26c-005231cb8be2-istio-data\") pod \"e2e-extmod-a-47c409-openshift-default-796cbb7889-drd46\" (UID: \"d76a0ace-f020-430f-a26c-005231cb8be2\") " pod="openshift-ingress/e2e-extmod-a-47c409-openshift-default-796cbb7889-drd46" Oct 31 04:48:10.507912 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:10.507783 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d76a0ace-f020-430f-a26c-005231cb8be2-workload-certs\") pod \"e2e-extmod-a-47c409-openshift-default-796cbb7889-drd46\" (UID: \"d76a0ace-f020-430f-a26c-005231cb8be2\") " pod="openshift-ingress/e2e-extmod-a-47c409-openshift-default-796cbb7889-drd46" Oct 31 04:48:10.507912 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:10.507819 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d76a0ace-f020-430f-a26c-005231cb8be2-credential-socket\") pod \"e2e-extmod-a-47c409-openshift-default-796cbb7889-drd46\" (UID: \"d76a0ace-f020-430f-a26c-005231cb8be2\") " pod="openshift-ingress/e2e-extmod-a-47c409-openshift-default-796cbb7889-drd46" Oct 31 04:48:10.507912 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:10.507872 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6qv7\" (UniqueName: \"kubernetes.io/projected/d76a0ace-f020-430f-a26c-005231cb8be2-kube-api-access-k6qv7\") pod \"e2e-extmod-a-47c409-openshift-default-796cbb7889-drd46\" (UID: \"d76a0ace-f020-430f-a26c-005231cb8be2\") " pod="openshift-ingress/e2e-extmod-a-47c409-openshift-default-796cbb7889-drd46" Oct 31 04:48:10.508195 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:10.507957 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d76a0ace-f020-430f-a26c-005231cb8be2-istio-podinfo\") pod \"e2e-extmod-a-47c409-openshift-default-796cbb7889-drd46\" (UID: \"d76a0ace-f020-430f-a26c-005231cb8be2\") " pod="openshift-ingress/e2e-extmod-a-47c409-openshift-default-796cbb7889-drd46" Oct 31 04:48:10.508195 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:10.507967 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d76a0ace-f020-430f-a26c-005231cb8be2-istiod-ca-cert\") pod \"e2e-extmod-a-47c409-openshift-default-796cbb7889-drd46\" (UID: \"d76a0ace-f020-430f-a26c-005231cb8be2\") " pod="openshift-ingress/e2e-extmod-a-47c409-openshift-default-796cbb7889-drd46" Oct 31 04:48:10.508195 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:10.508058 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d76a0ace-f020-430f-a26c-005231cb8be2-workload-socket\") pod \"e2e-extmod-a-47c409-openshift-default-796cbb7889-drd46\" (UID: \"d76a0ace-f020-430f-a26c-005231cb8be2\") " pod="openshift-ingress/e2e-extmod-a-47c409-openshift-default-796cbb7889-drd46" Oct 31 04:48:10.508195 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:10.508122 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d76a0ace-f020-430f-a26c-005231cb8be2-istio-data\") pod \"e2e-extmod-a-47c409-openshift-default-796cbb7889-drd46\" (UID: \"d76a0ace-f020-430f-a26c-005231cb8be2\") " pod="openshift-ingress/e2e-extmod-a-47c409-openshift-default-796cbb7889-drd46" Oct 31 04:48:10.510306 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:10.510280 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d76a0ace-f020-430f-a26c-005231cb8be2-istio-envoy\") pod \"e2e-extmod-a-47c409-openshift-default-796cbb7889-drd46\" (UID: \"d76a0ace-f020-430f-a26c-005231cb8be2\") " pod="openshift-ingress/e2e-extmod-a-47c409-openshift-default-796cbb7889-drd46" Oct 31 04:48:10.510455 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:10.510420 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d76a0ace-f020-430f-a26c-005231cb8be2-istio-podinfo\") pod \"e2e-extmod-a-47c409-openshift-default-796cbb7889-drd46\" (UID: \"d76a0ace-f020-430f-a26c-005231cb8be2\") " pod="openshift-ingress/e2e-extmod-a-47c409-openshift-default-796cbb7889-drd46" Oct 31 04:48:10.531565 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:10.531512 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d76a0ace-f020-430f-a26c-005231cb8be2-istio-token\") pod \"e2e-extmod-a-47c409-openshift-default-796cbb7889-drd46\" (UID: \"d76a0ace-f020-430f-a26c-005231cb8be2\") " pod="openshift-ingress/e2e-extmod-a-47c409-openshift-default-796cbb7889-drd46" Oct 31 04:48:10.531673 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:10.531616 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6qv7\" (UniqueName: \"kubernetes.io/projected/d76a0ace-f020-430f-a26c-005231cb8be2-kube-api-access-k6qv7\") pod \"e2e-extmod-a-47c409-openshift-default-796cbb7889-drd46\" (UID: \"d76a0ace-f020-430f-a26c-005231cb8be2\") " pod="openshift-ingress/e2e-extmod-a-47c409-openshift-default-796cbb7889-drd46" Oct 31 04:48:10.580572 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:10.580552 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-47c409-openshift-default-796cbb7889-drd46" Oct 31 04:48:10.895358 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:48:10.895294 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd76a0ace_f020_430f_a26c_005231cb8be2.slice/crio-fb19dbfbb11e68dd6f922602ead489bc79b40ff6540ee8cdeae5914558d4307a WatchSource:0}: Error finding container fb19dbfbb11e68dd6f922602ead489bc79b40ff6540ee8cdeae5914558d4307a: Status 404 returned error can't find the container with id fb19dbfbb11e68dd6f922602ead489bc79b40ff6540ee8cdeae5914558d4307a Oct 31 04:48:10.895841 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:10.895815 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-47c409-openshift-default-796cbb7889-drd46"] Oct 31 04:48:10.898276 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:10.898244 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:48:10.898344 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:10.898310 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:48:10.898385 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:10.898356 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:48:11.006018 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:48:11.005762 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 04:48:11.167785 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:11.167677 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-47c409-openshift-default-796cbb7889-drd46" event={"ID":"d76a0ace-f020-430f-a26c-005231cb8be2","Type":"ContainerStarted","Data":"1c8feda615d68ba3a4acdd1e22634785025a378fe79bec24ea294b989852fadd"} Oct 31 04:48:11.167785 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:11.167781 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-47c409-openshift-default-796cbb7889-drd46" event={"ID":"d76a0ace-f020-430f-a26c-005231cb8be2","Type":"ContainerStarted","Data":"fb19dbfbb11e68dd6f922602ead489bc79b40ff6540ee8cdeae5914558d4307a"} Oct 31 04:48:11.218500 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:11.218451 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-47c409-openshift-default-796cbb7889-drd46" podStartSLOduration=1.218436625 podStartE2EDuration="1.218436625s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:48:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:48:11.21536568 +0000 UTC m=+2638.929700310" watchObservedRunningTime="2026-10-31 04:48:11.218436625 +0000 UTC m=+2638.932771239" Oct 31 04:48:11.581520 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:11.581428 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-47c409-openshift-default-796cbb7889-drd46" Oct 31 04:48:11.583082 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:11.583059 4376 patch_prober.go:29] interesting pod/e2e-extmod-a-47c409-openshift-default-796cbb7889-drd46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body= Oct 31 04:48:11.583148 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:11.583121 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-47c409-openshift-default-796cbb7889-drd46" podUID="d76a0ace-f020-430f-a26c-005231cb8be2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" Oct 31 04:48:12.581406 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:12.581369 4376 patch_prober.go:29] interesting pod/e2e-extmod-a-47c409-openshift-default-796cbb7889-drd46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body= Oct 31 04:48:12.581619 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:12.581428 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-47c409-openshift-default-796cbb7889-drd46" podUID="d76a0ace-f020-430f-a26c-005231cb8be2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" Oct 31 04:48:13.581163 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:13.581132 4376 patch_prober.go:29] interesting pod/e2e-extmod-a-47c409-openshift-default-796cbb7889-drd46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body= Oct 31 04:48:13.581474 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:13.581181 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-47c409-openshift-default-796cbb7889-drd46" podUID="d76a0ace-f020-430f-a26c-005231cb8be2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" Oct 31 04:48:14.581414 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:14.581382 4376 patch_prober.go:29] interesting pod/e2e-extmod-a-47c409-openshift-default-796cbb7889-drd46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body= Oct 31 04:48:14.581834 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:14.581432 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-47c409-openshift-default-796cbb7889-drd46" podUID="d76a0ace-f020-430f-a26c-005231cb8be2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" Oct 31 04:48:14.688725 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:14.688693 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-xwqm7"] Oct 31 04:48:14.693822 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:14.693801 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-xwqm7" Oct 31 04:48:14.713768 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:14.713740 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-sqqs6\"" Oct 31 04:48:14.728447 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:14.728417 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-xwqm7"] Oct 31 04:48:14.804903 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:14.804868 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69c1b903-0719-4813-9e1e-11e87818ec69-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-xwqm7\" (UID: \"69c1b903-0719-4813-9e1e-11e87818ec69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-xwqm7" Oct 31 04:48:14.805094 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:14.804991 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69c1b903-0719-4813-9e1e-11e87818ec69-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-xwqm7\" (UID: \"69c1b903-0719-4813-9e1e-11e87818ec69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-xwqm7" Oct 31 04:48:14.805094 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:14.805018 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69c1b903-0719-4813-9e1e-11e87818ec69-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-xwqm7\" (UID: \"69c1b903-0719-4813-9e1e-11e87818ec69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-xwqm7" Oct 31 04:48:14.805094 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:14.805045 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69c1b903-0719-4813-9e1e-11e87818ec69-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-xwqm7\" (UID: \"69c1b903-0719-4813-9e1e-11e87818ec69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-xwqm7" Oct 31 04:48:14.805412 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:14.805390 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69c1b903-0719-4813-9e1e-11e87818ec69-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-xwqm7\" (UID: \"69c1b903-0719-4813-9e1e-11e87818ec69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-xwqm7" Oct 31 04:48:14.805489 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:14.805448 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69c1b903-0719-4813-9e1e-11e87818ec69-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-xwqm7\" (UID: \"69c1b903-0719-4813-9e1e-11e87818ec69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-xwqm7" Oct 31 04:48:14.805599 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:14.805568 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69c1b903-0719-4813-9e1e-11e87818ec69-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-xwqm7\" (UID: \"69c1b903-0719-4813-9e1e-11e87818ec69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-xwqm7" Oct 31 04:48:14.805742 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:14.805678 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7q7hv\" (UniqueName: \"kubernetes.io/projected/69c1b903-0719-4813-9e1e-11e87818ec69-kube-api-access-7q7hv\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-xwqm7\" (UID: \"69c1b903-0719-4813-9e1e-11e87818ec69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-xwqm7" Oct 31 04:48:14.805742 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:14.805710 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69c1b903-0719-4813-9e1e-11e87818ec69-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-xwqm7\" (UID: \"69c1b903-0719-4813-9e1e-11e87818ec69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-xwqm7" Oct 31 04:48:14.907314 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:14.907282 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69c1b903-0719-4813-9e1e-11e87818ec69-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-xwqm7\" (UID: \"69c1b903-0719-4813-9e1e-11e87818ec69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-xwqm7" Oct 31 04:48:14.907537 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:14.907386 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69c1b903-0719-4813-9e1e-11e87818ec69-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-xwqm7\" (UID: \"69c1b903-0719-4813-9e1e-11e87818ec69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-xwqm7" Oct 31 04:48:14.907537 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:14.907459 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69c1b903-0719-4813-9e1e-11e87818ec69-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-xwqm7\" (UID: \"69c1b903-0719-4813-9e1e-11e87818ec69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-xwqm7" Oct 31 04:48:14.907537 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:14.907482 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69c1b903-0719-4813-9e1e-11e87818ec69-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-xwqm7\" (UID: \"69c1b903-0719-4813-9e1e-11e87818ec69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-xwqm7" Oct 31 04:48:14.907537 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:14.907502 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69c1b903-0719-4813-9e1e-11e87818ec69-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-xwqm7\" (UID: \"69c1b903-0719-4813-9e1e-11e87818ec69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-xwqm7" Oct 31 04:48:14.907537 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:14.907524 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69c1b903-0719-4813-9e1e-11e87818ec69-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-xwqm7\" (UID: \"69c1b903-0719-4813-9e1e-11e87818ec69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-xwqm7" Oct 31 04:48:14.907864 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:14.907553 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69c1b903-0719-4813-9e1e-11e87818ec69-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-xwqm7\" (UID: \"69c1b903-0719-4813-9e1e-11e87818ec69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-xwqm7" Oct 31 04:48:14.907864 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:14.907651 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69c1b903-0719-4813-9e1e-11e87818ec69-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-xwqm7\" (UID: \"69c1b903-0719-4813-9e1e-11e87818ec69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-xwqm7" Oct 31 04:48:14.907864 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:14.907749 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7q7hv\" (UniqueName: \"kubernetes.io/projected/69c1b903-0719-4813-9e1e-11e87818ec69-kube-api-access-7q7hv\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-xwqm7\" (UID: \"69c1b903-0719-4813-9e1e-11e87818ec69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-xwqm7" Oct 31 04:48:14.907864 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:14.907852 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69c1b903-0719-4813-9e1e-11e87818ec69-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-xwqm7\" (UID: \"69c1b903-0719-4813-9e1e-11e87818ec69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-xwqm7" Oct 31 04:48:14.908392 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:14.908363 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69c1b903-0719-4813-9e1e-11e87818ec69-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-xwqm7\" (UID: \"69c1b903-0719-4813-9e1e-11e87818ec69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-xwqm7" Oct 31 04:48:14.908514 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:14.908437 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69c1b903-0719-4813-9e1e-11e87818ec69-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-xwqm7\" (UID: \"69c1b903-0719-4813-9e1e-11e87818ec69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-xwqm7" Oct 31 04:48:14.908514 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:14.908471 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69c1b903-0719-4813-9e1e-11e87818ec69-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-xwqm7\" (UID: \"69c1b903-0719-4813-9e1e-11e87818ec69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-xwqm7" Oct 31 04:48:14.908656 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:14.908625 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69c1b903-0719-4813-9e1e-11e87818ec69-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-xwqm7\" (UID: \"69c1b903-0719-4813-9e1e-11e87818ec69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-xwqm7" Oct 31 04:48:14.923120 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:14.923094 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69c1b903-0719-4813-9e1e-11e87818ec69-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-xwqm7\" (UID: \"69c1b903-0719-4813-9e1e-11e87818ec69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-xwqm7" Oct 31 04:48:14.924256 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:14.924234 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69c1b903-0719-4813-9e1e-11e87818ec69-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-xwqm7\" (UID: \"69c1b903-0719-4813-9e1e-11e87818ec69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-xwqm7" Oct 31 04:48:14.930967 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:14.930941 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69c1b903-0719-4813-9e1e-11e87818ec69-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-xwqm7\" (UID: \"69c1b903-0719-4813-9e1e-11e87818ec69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-xwqm7" Oct 31 04:48:14.931573 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:14.931555 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7q7hv\" (UniqueName: \"kubernetes.io/projected/69c1b903-0719-4813-9e1e-11e87818ec69-kube-api-access-7q7hv\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-xwqm7\" (UID: \"69c1b903-0719-4813-9e1e-11e87818ec69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-xwqm7" Oct 31 04:48:15.008566 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:15.008535 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-xwqm7" Oct 31 04:48:15.370995 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:15.370967 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-xwqm7"] Oct 31 04:48:15.371422 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:15.371220 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:48:15.371422 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:15.371295 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:48:15.371422 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:15.371345 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:48:15.583637 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:15.583593 4376 patch_prober.go:29] interesting pod/e2e-extmod-a-47c409-openshift-default-796cbb7889-drd46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body= Oct 31 04:48:15.583637 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:15.583638 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-47c409-openshift-default-796cbb7889-drd46" podUID="d76a0ace-f020-430f-a26c-005231cb8be2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" Oct 31 04:48:16.207858 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:16.207773 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-xwqm7" event={"ID":"69c1b903-0719-4813-9e1e-11e87818ec69","Type":"ContainerStarted","Data":"ff8b6101a553eebb447b95c7ef365f91f985170deab1f4fc7a18aeeb428a8e50"} Oct 31 04:48:16.208086 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:16.208069 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-xwqm7" event={"ID":"69c1b903-0719-4813-9e1e-11e87818ec69","Type":"ContainerStarted","Data":"8073b58d3d7901f5e70a386d8de5513664e81a103d3f9eeba253286871d67a76"} Oct 31 04:48:16.266422 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:16.266355 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-xwqm7" podStartSLOduration=2.266316583 podStartE2EDuration="2.266316583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:48:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:48:16.258499418 +0000 UTC m=+2643.972834059" watchObservedRunningTime="2026-10-31 04:48:16.266316583 +0000 UTC m=+2643.980651203" Oct 31 04:48:16.581642 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:16.581605 4376 patch_prober.go:29] interesting pod/e2e-extmod-a-47c409-openshift-default-796cbb7889-drd46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body= Oct 31 04:48:16.581933 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:16.581901 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-47c409-openshift-default-796cbb7889-drd46" podUID="d76a0ace-f020-430f-a26c-005231cb8be2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" Oct 31 04:48:17.009404 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:17.009362 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-xwqm7" Oct 31 04:48:17.011238 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:17.011212 4376 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-xwqm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body= Oct 31 04:48:17.011414 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:17.011260 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-xwqm7" podUID="69c1b903-0719-4813-9e1e-11e87818ec69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" Oct 31 04:48:17.586439 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:17.586409 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-47c409-openshift-default-796cbb7889-drd46" Oct 31 04:48:17.586776 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:17.586752 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-47c409-openshift-default-796cbb7889-drd46" Oct 31 04:48:17.587507 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:17.587489 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-47c409-openshift-default-796cbb7889-drd46" Oct 31 04:48:18.008972 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:18.008943 4376 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-xwqm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body= Oct 31 04:48:18.009151 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:18.008989 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-xwqm7" podUID="69c1b903-0719-4813-9e1e-11e87818ec69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" Oct 31 04:48:19.009813 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:19.009782 4376 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-xwqm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body= Oct 31 04:48:19.010240 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:19.009837 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-xwqm7" podUID="69c1b903-0719-4813-9e1e-11e87818ec69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" Oct 31 04:48:20.009012 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:20.008976 4376 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-xwqm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body= Oct 31 04:48:20.009191 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:20.009034 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-xwqm7" podUID="69c1b903-0719-4813-9e1e-11e87818ec69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" Oct 31 04:48:21.008974 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:21.008943 4376 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-xwqm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body= Oct 31 04:48:21.009388 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:21.008997 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-xwqm7" podUID="69c1b903-0719-4813-9e1e-11e87818ec69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" Oct 31 04:48:22.009420 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:22.009382 4376 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-xwqm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body= Oct 31 04:48:22.009945 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:22.009435 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-xwqm7" podUID="69c1b903-0719-4813-9e1e-11e87818ec69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" Oct 31 04:48:22.099287 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:22.099248 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 04:48:23.014889 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:23.014852 4376 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-xwqm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body= Oct 31 04:48:23.015524 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:23.015488 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-xwqm7" podUID="69c1b903-0719-4813-9e1e-11e87818ec69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" Oct 31 04:48:24.831989 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:24.831889 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-xwqm7" Oct 31 04:48:24.832633 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:24.832504 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-xwqm7" Oct 31 04:48:24.833747 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:24.833721 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-xwqm7" Oct 31 04:48:41.004182 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:48:41.004142 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 04:48:43.771395 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:43.771313 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-766b99cdcf-m2gpm"] Oct 31 04:48:44.804552 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:44.804519 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-nv4mk"] Oct 31 04:48:44.809479 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:44.809455 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-nv4mk" Oct 31 04:48:44.845870 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:44.845837 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-nv4mk"] Oct 31 04:48:44.953488 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:44.953450 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fd946a5-6c8c-4341-ae85-80f8cf13673f-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-nv4mk\" (UID: \"3fd946a5-6c8c-4341-ae85-80f8cf13673f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-nv4mk" Oct 31 04:48:44.953674 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:44.953561 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fd946a5-6c8c-4341-ae85-80f8cf13673f-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-nv4mk\" (UID: \"3fd946a5-6c8c-4341-ae85-80f8cf13673f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-nv4mk" Oct 31 04:48:44.953719 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:44.953679 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fd946a5-6c8c-4341-ae85-80f8cf13673f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-nv4mk\" (UID: \"3fd946a5-6c8c-4341-ae85-80f8cf13673f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-nv4mk" Oct 31 04:48:44.953719 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:44.953698 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fd946a5-6c8c-4341-ae85-80f8cf13673f-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-nv4mk\" (UID: \"3fd946a5-6c8c-4341-ae85-80f8cf13673f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-nv4mk" Oct 31 04:48:44.953787 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:44.953722 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fd946a5-6c8c-4341-ae85-80f8cf13673f-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-nv4mk\" (UID: \"3fd946a5-6c8c-4341-ae85-80f8cf13673f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-nv4mk" Oct 31 04:48:44.953845 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:44.953829 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fd946a5-6c8c-4341-ae85-80f8cf13673f-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-nv4mk\" (UID: \"3fd946a5-6c8c-4341-ae85-80f8cf13673f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-nv4mk" Oct 31 04:48:44.953894 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:44.953883 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fd946a5-6c8c-4341-ae85-80f8cf13673f-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-nv4mk\" (UID: \"3fd946a5-6c8c-4341-ae85-80f8cf13673f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-nv4mk" Oct 31 04:48:44.953982 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:44.953970 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fd946a5-6c8c-4341-ae85-80f8cf13673f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-nv4mk\" (UID: \"3fd946a5-6c8c-4341-ae85-80f8cf13673f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-nv4mk" Oct 31 04:48:44.954019 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:44.953997 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28gzr\" (UniqueName: \"kubernetes.io/projected/3fd946a5-6c8c-4341-ae85-80f8cf13673f-kube-api-access-28gzr\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-nv4mk\" (UID: \"3fd946a5-6c8c-4341-ae85-80f8cf13673f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-nv4mk" Oct 31 04:48:45.055089 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:45.055009 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fd946a5-6c8c-4341-ae85-80f8cf13673f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-nv4mk\" (UID: \"3fd946a5-6c8c-4341-ae85-80f8cf13673f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-nv4mk" Oct 31 04:48:45.055089 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:45.055043 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fd946a5-6c8c-4341-ae85-80f8cf13673f-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-nv4mk\" (UID: \"3fd946a5-6c8c-4341-ae85-80f8cf13673f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-nv4mk" Oct 31 04:48:45.055089 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:45.055064 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fd946a5-6c8c-4341-ae85-80f8cf13673f-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-nv4mk\" (UID: \"3fd946a5-6c8c-4341-ae85-80f8cf13673f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-nv4mk" Oct 31 04:48:45.055377 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:45.055116 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fd946a5-6c8c-4341-ae85-80f8cf13673f-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-nv4mk\" (UID: \"3fd946a5-6c8c-4341-ae85-80f8cf13673f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-nv4mk" Oct 31 04:48:45.055377 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:45.055167 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fd946a5-6c8c-4341-ae85-80f8cf13673f-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-nv4mk\" (UID: \"3fd946a5-6c8c-4341-ae85-80f8cf13673f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-nv4mk" Oct 31 04:48:45.055377 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:45.055253 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fd946a5-6c8c-4341-ae85-80f8cf13673f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-nv4mk\" (UID: \"3fd946a5-6c8c-4341-ae85-80f8cf13673f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-nv4mk" Oct 31 04:48:45.055377 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:45.055276 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28gzr\" (UniqueName: \"kubernetes.io/projected/3fd946a5-6c8c-4341-ae85-80f8cf13673f-kube-api-access-28gzr\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-nv4mk\" (UID: \"3fd946a5-6c8c-4341-ae85-80f8cf13673f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-nv4mk" Oct 31 04:48:45.055377 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:45.055318 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fd946a5-6c8c-4341-ae85-80f8cf13673f-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-nv4mk\" (UID: \"3fd946a5-6c8c-4341-ae85-80f8cf13673f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-nv4mk" Oct 31 04:48:45.055632 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:45.055430 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fd946a5-6c8c-4341-ae85-80f8cf13673f-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-nv4mk\" (UID: \"3fd946a5-6c8c-4341-ae85-80f8cf13673f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-nv4mk" Oct 31 04:48:45.055632 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:45.055591 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fd946a5-6c8c-4341-ae85-80f8cf13673f-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-nv4mk\" (UID: \"3fd946a5-6c8c-4341-ae85-80f8cf13673f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-nv4mk" Oct 31 04:48:45.055748 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:45.055635 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fd946a5-6c8c-4341-ae85-80f8cf13673f-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-nv4mk\" (UID: \"3fd946a5-6c8c-4341-ae85-80f8cf13673f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-nv4mk" Oct 31 04:48:45.055848 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:45.055824 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fd946a5-6c8c-4341-ae85-80f8cf13673f-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-nv4mk\" (UID: \"3fd946a5-6c8c-4341-ae85-80f8cf13673f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-nv4mk" Oct 31 04:48:45.055924 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:45.055898 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fd946a5-6c8c-4341-ae85-80f8cf13673f-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-nv4mk\" (UID: \"3fd946a5-6c8c-4341-ae85-80f8cf13673f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-nv4mk" Oct 31 04:48:45.055986 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:45.055935 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fd946a5-6c8c-4341-ae85-80f8cf13673f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-nv4mk\" (UID: \"3fd946a5-6c8c-4341-ae85-80f8cf13673f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-nv4mk" Oct 31 04:48:45.057579 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:45.057553 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fd946a5-6c8c-4341-ae85-80f8cf13673f-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-nv4mk\" (UID: \"3fd946a5-6c8c-4341-ae85-80f8cf13673f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-nv4mk" Oct 31 04:48:45.057857 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:45.057838 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fd946a5-6c8c-4341-ae85-80f8cf13673f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-nv4mk\" (UID: \"3fd946a5-6c8c-4341-ae85-80f8cf13673f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-nv4mk" Oct 31 04:48:45.078723 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:45.078701 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fd946a5-6c8c-4341-ae85-80f8cf13673f-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-nv4mk\" (UID: \"3fd946a5-6c8c-4341-ae85-80f8cf13673f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-nv4mk" Oct 31 04:48:45.078896 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:45.078877 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28gzr\" (UniqueName: \"kubernetes.io/projected/3fd946a5-6c8c-4341-ae85-80f8cf13673f-kube-api-access-28gzr\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-nv4mk\" (UID: \"3fd946a5-6c8c-4341-ae85-80f8cf13673f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-nv4mk" Oct 31 04:48:45.136487 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:45.136457 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-nv4mk" Oct 31 04:48:45.496499 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:48:45.496457 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fd946a5_6c8c_4341_ae85_80f8cf13673f.slice/crio-788facf93b30d00eb957e7ae4367b2ccfce123cd4abf3ea5aa9d1c08d2ae5693 WatchSource:0}: Error finding container 788facf93b30d00eb957e7ae4367b2ccfce123cd4abf3ea5aa9d1c08d2ae5693: Status 404 returned error can't find the container with id 788facf93b30d00eb957e7ae4367b2ccfce123cd4abf3ea5aa9d1c08d2ae5693 Oct 31 04:48:45.516246 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:45.516182 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:48:45.516545 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:45.516516 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:48:45.516735 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:45.516715 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:48:45.533991 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:45.533950 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-nv4mk"] Oct 31 04:48:45.564683 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:45.564590 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-nv4mk" event={"ID":"3fd946a5-6c8c-4341-ae85-80f8cf13673f","Type":"ContainerStarted","Data":"788facf93b30d00eb957e7ae4367b2ccfce123cd4abf3ea5aa9d1c08d2ae5693"} Oct 31 04:48:46.575614 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:46.575571 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-nv4mk" event={"ID":"3fd946a5-6c8c-4341-ae85-80f8cf13673f","Type":"ContainerStarted","Data":"241f209b922ec5ecd551d029023ed09ddfa009816e04267c35ed53eb4a6e57e8"} Oct 31 04:48:46.625286 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:46.625232 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-nv4mk" podStartSLOduration=2.625218147 podStartE2EDuration="2.625218147s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:48:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:48:46.621384229 +0000 UTC m=+2674.335718863" watchObservedRunningTime="2026-10-31 04:48:46.625218147 +0000 UTC m=+2674.339552833" Oct 31 04:48:47.142622 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:47.142586 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-nv4mk" Oct 31 04:48:47.165515 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:47.165477 4376 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-nv4mk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" start-of-body= Oct 31 04:48:47.165822 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:47.165788 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-nv4mk" podUID="3fd946a5-6c8c-4341-ae85-80f8cf13673f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" Oct 31 04:48:48.136987 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:48.136944 4376 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-nv4mk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" start-of-body= Oct 31 04:48:48.137495 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:48.137014 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-nv4mk" podUID="3fd946a5-6c8c-4341-ae85-80f8cf13673f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" Oct 31 04:48:48.772060 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:48.772011 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-766b99cdcf-m2gpm" podUID="cf93b9cd-4bcd-49dd-8861-745aef0e9518" containerName="payload-processing" containerID="cri-o://42eca02da2609fb829432100e1a3d36682bae935fe1c09cbb753635d36814b1b" gracePeriod=25 Oct 31 04:48:49.108179 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:49.108154 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-766b99cdcf-m2gpm" Oct 31 04:48:49.136963 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:49.136938 4376 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-nv4mk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" start-of-body= Oct 31 04:48:49.137108 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:49.136983 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-nv4mk" podUID="3fd946a5-6c8c-4341-ae85-80f8cf13673f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" Oct 31 04:48:49.201360 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:49.201318 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf93b9cd-4bcd-49dd-8861-745aef0e9518-kube-api-access-gshpm\" (UniqueName: \"kubernetes.io/projected/cf93b9cd-4bcd-49dd-8861-745aef0e9518-kube-api-access-gshpm\") pod \"cf93b9cd-4bcd-49dd-8861-745aef0e9518\" (UID: \"cf93b9cd-4bcd-49dd-8861-745aef0e9518\") " Oct 31 04:48:49.201532 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:49.201497 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf93b9cd-4bcd-49dd-8861-745aef0e9518-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cf93b9cd-4bcd-49dd-8861-745aef0e9518-routing-overlay-volume\") pod \"cf93b9cd-4bcd-49dd-8861-745aef0e9518\" (UID: \"cf93b9cd-4bcd-49dd-8861-745aef0e9518\") " Oct 31 04:48:49.201623 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:49.201611 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf93b9cd-4bcd-49dd-8861-745aef0e9518-provider-credentials\" (UniqueName: \"kubernetes.io/projected/cf93b9cd-4bcd-49dd-8861-745aef0e9518-provider-credentials\") pod \"cf93b9cd-4bcd-49dd-8861-745aef0e9518\" (UID: \"cf93b9cd-4bcd-49dd-8861-745aef0e9518\") " Oct 31 04:48:49.201660 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:49.201646 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf93b9cd-4bcd-49dd-8861-745aef0e9518-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf93b9cd-4bcd-49dd-8861-745aef0e9518-plugins-config-volume\") pod \"cf93b9cd-4bcd-49dd-8861-745aef0e9518\" (UID: \"cf93b9cd-4bcd-49dd-8861-745aef0e9518\") " Oct 31 04:48:49.201931 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:49.201896 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf93b9cd-4bcd-49dd-8861-745aef0e9518-routing-overlay-volume" pod "cf93b9cd-4bcd-49dd-8861-745aef0e9518" (UID: "cf93b9cd-4bcd-49dd-8861-745aef0e9518"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:48:49.202154 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:49.202134 4376 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cf93b9cd-4bcd-49dd-8861-745aef0e9518-routing-overlay-volume\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:48:49.202269 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:49.202158 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf93b9cd-4bcd-49dd-8861-745aef0e9518-plugins-config-volume" pod "cf93b9cd-4bcd-49dd-8861-745aef0e9518" (UID: "cf93b9cd-4bcd-49dd-8861-745aef0e9518"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:48:49.203642 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:49.203615 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf93b9cd-4bcd-49dd-8861-745aef0e9518-kube-api-access-gshpm" pod "cf93b9cd-4bcd-49dd-8861-745aef0e9518" (UID: "cf93b9cd-4bcd-49dd-8861-745aef0e9518"). InnerVolumeSpecName "kube-api-access-gshpm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:48:49.203763 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:49.203745 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf93b9cd-4bcd-49dd-8861-745aef0e9518-provider-credentials" pod "cf93b9cd-4bcd-49dd-8861-745aef0e9518" (UID: "cf93b9cd-4bcd-49dd-8861-745aef0e9518"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:48:49.303147 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:49.303110 4376 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cf93b9cd-4bcd-49dd-8861-745aef0e9518-provider-credentials\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:48:49.303147 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:49.303140 4376 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf93b9cd-4bcd-49dd-8861-745aef0e9518-plugins-config-volume\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:48:49.303147 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:49.303151 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gshpm\" (UniqueName: \"kubernetes.io/projected/cf93b9cd-4bcd-49dd-8861-745aef0e9518-kube-api-access-gshpm\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:48:49.668056 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:49.668020 4376 generic.go:356] "Generic (PLEG): container finished" podID="cf93b9cd-4bcd-49dd-8861-745aef0e9518" containerID="42eca02da2609fb829432100e1a3d36682bae935fe1c09cbb753635d36814b1b" exitCode=0 Oct 31 04:48:49.668267 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:49.668086 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-766b99cdcf-m2gpm" event={"ID":"cf93b9cd-4bcd-49dd-8861-745aef0e9518","Type":"ContainerDied","Data":"42eca02da2609fb829432100e1a3d36682bae935fe1c09cbb753635d36814b1b"} Oct 31 04:48:49.668267 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:49.668134 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-766b99cdcf-m2gpm" event={"ID":"cf93b9cd-4bcd-49dd-8861-745aef0e9518","Type":"ContainerDied","Data":"e18947b83149bc4c92f1cc54f08adadc0b520abbd21a57b0259186754e8daf3b"} Oct 31 04:48:49.668267 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:49.668162 4376 scope.go:122] "RemoveContainer" containerID="42eca02da2609fb829432100e1a3d36682bae935fe1c09cbb753635d36814b1b" Oct 31 04:48:49.668267 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:49.668089 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-766b99cdcf-m2gpm" Oct 31 04:48:49.679867 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:49.679847 4376 scope.go:122] "RemoveContainer" containerID="42eca02da2609fb829432100e1a3d36682bae935fe1c09cbb753635d36814b1b" Oct 31 04:48:49.680129 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:48:49.680112 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42eca02da2609fb829432100e1a3d36682bae935fe1c09cbb753635d36814b1b\": container with ID starting with 42eca02da2609fb829432100e1a3d36682bae935fe1c09cbb753635d36814b1b not found: ID does not exist" containerID="42eca02da2609fb829432100e1a3d36682bae935fe1c09cbb753635d36814b1b" Oct 31 04:48:49.680232 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:49.680137 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42eca02da2609fb829432100e1a3d36682bae935fe1c09cbb753635d36814b1b"} err="failed to get container status \"42eca02da2609fb829432100e1a3d36682bae935fe1c09cbb753635d36814b1b\": rpc error: code = NotFound desc = could not find container \"42eca02da2609fb829432100e1a3d36682bae935fe1c09cbb753635d36814b1b\": container with ID starting with 42eca02da2609fb829432100e1a3d36682bae935fe1c09cbb753635d36814b1b not found: ID does not exist" Oct 31 04:48:49.722556 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:49.722526 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-766b99cdcf-m2gpm"] Oct 31 04:48:49.735280 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:49.735238 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-766b99cdcf-m2gpm"] Oct 31 04:48:50.137146 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:50.137114 4376 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-nv4mk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" start-of-body= Oct 31 04:48:50.137659 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:50.137165 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-nv4mk" podUID="3fd946a5-6c8c-4341-ae85-80f8cf13673f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" Oct 31 04:48:50.469829 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:50.469747 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf93b9cd-4bcd-49dd-8861-745aef0e9518" path="/var/lib/kubelet/pods/cf93b9cd-4bcd-49dd-8861-745aef0e9518/volumes" Oct 31 04:48:51.153348 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:51.142746 4376 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-nv4mk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" start-of-body= Oct 31 04:48:51.153348 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:51.142819 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-nv4mk" podUID="3fd946a5-6c8c-4341-ae85-80f8cf13673f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" Oct 31 04:48:52.145353 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:52.138919 4376 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-nv4mk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" start-of-body= Oct 31 04:48:52.145353 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:52.139008 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-nv4mk" podUID="3fd946a5-6c8c-4341-ae85-80f8cf13673f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" Oct 31 04:48:53.137567 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:53.137524 4376 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-nv4mk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" start-of-body= Oct 31 04:48:53.137986 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:53.137591 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-nv4mk" podUID="3fd946a5-6c8c-4341-ae85-80f8cf13673f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" Oct 31 04:48:54.662048 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:54.662011 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-nv4mk" Oct 31 04:48:54.662605 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:54.662581 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-nv4mk" Oct 31 04:48:54.663369 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:54.663318 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-nv4mk" Oct 31 04:48:59.859077 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:59.859041 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lj6bl"] Oct 31 04:48:59.859894 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:59.859876 4376 cpu_manager.go:414] "containerMap: removing container" podUID="cf93b9cd-4bcd-49dd-8861-745aef0e9518" containerName="payload-processing" Oct 31 04:48:59.859894 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:59.859895 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf93b9cd-4bcd-49dd-8861-745aef0e9518" containerName="payload-processing" Oct 31 04:48:59.860034 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:59.860025 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf93b9cd-4bcd-49dd-8861-745aef0e9518" containerName="payload-processing" Oct 31 04:48:59.865197 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:59.865172 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lj6bl" Oct 31 04:48:59.907908 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:59.907876 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lj6bl"] Oct 31 04:48:59.975869 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:59.975834 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9435ec7-da60-49e4-9219-1a3cdb485843-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-lj6bl\" (UID: \"a9435ec7-da60-49e4-9219-1a3cdb485843\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lj6bl" Oct 31 04:48:59.976036 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:59.975888 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfvm9\" (UniqueName: \"kubernetes.io/projected/a9435ec7-da60-49e4-9219-1a3cdb485843-kube-api-access-wfvm9\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-lj6bl\" (UID: \"a9435ec7-da60-49e4-9219-1a3cdb485843\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lj6bl" Oct 31 04:48:59.976111 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:59.976035 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9435ec7-da60-49e4-9219-1a3cdb485843-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-lj6bl\" (UID: \"a9435ec7-da60-49e4-9219-1a3cdb485843\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lj6bl" Oct 31 04:48:59.976111 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:59.976102 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9435ec7-da60-49e4-9219-1a3cdb485843-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-lj6bl\" (UID: \"a9435ec7-da60-49e4-9219-1a3cdb485843\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lj6bl" Oct 31 04:48:59.976270 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:59.976254 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9435ec7-da60-49e4-9219-1a3cdb485843-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-lj6bl\" (UID: \"a9435ec7-da60-49e4-9219-1a3cdb485843\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lj6bl" Oct 31 04:48:59.976311 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:59.976281 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9435ec7-da60-49e4-9219-1a3cdb485843-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-lj6bl\" (UID: \"a9435ec7-da60-49e4-9219-1a3cdb485843\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lj6bl" Oct 31 04:48:59.976414 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:59.976402 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9435ec7-da60-49e4-9219-1a3cdb485843-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-lj6bl\" (UID: \"a9435ec7-da60-49e4-9219-1a3cdb485843\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lj6bl" Oct 31 04:48:59.976469 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:59.976455 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9435ec7-da60-49e4-9219-1a3cdb485843-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-lj6bl\" (UID: \"a9435ec7-da60-49e4-9219-1a3cdb485843\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lj6bl" Oct 31 04:48:59.976527 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:48:59.976512 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9435ec7-da60-49e4-9219-1a3cdb485843-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-lj6bl\" (UID: \"a9435ec7-da60-49e4-9219-1a3cdb485843\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lj6bl" Oct 31 04:49:00.078171 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:49:00.078135 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9435ec7-da60-49e4-9219-1a3cdb485843-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-lj6bl\" (UID: \"a9435ec7-da60-49e4-9219-1a3cdb485843\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lj6bl" Oct 31 04:49:00.079010 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:49:00.078849 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9435ec7-da60-49e4-9219-1a3cdb485843-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-lj6bl\" (UID: \"a9435ec7-da60-49e4-9219-1a3cdb485843\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lj6bl" Oct 31 04:49:00.079261 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:49:00.079238 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfvm9\" (UniqueName: \"kubernetes.io/projected/a9435ec7-da60-49e4-9219-1a3cdb485843-kube-api-access-wfvm9\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-lj6bl\" (UID: \"a9435ec7-da60-49e4-9219-1a3cdb485843\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lj6bl" Oct 31 04:49:00.079871 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:49:00.079848 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9435ec7-da60-49e4-9219-1a3cdb485843-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-lj6bl\" (UID: \"a9435ec7-da60-49e4-9219-1a3cdb485843\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lj6bl" Oct 31 04:49:00.080590 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:49:00.080569 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9435ec7-da60-49e4-9219-1a3cdb485843-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-lj6bl\" (UID: \"a9435ec7-da60-49e4-9219-1a3cdb485843\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lj6bl" Oct 31 04:49:00.081279 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:49:00.081157 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9435ec7-da60-49e4-9219-1a3cdb485843-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-lj6bl\" (UID: \"a9435ec7-da60-49e4-9219-1a3cdb485843\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lj6bl" Oct 31 04:49:00.081636 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:49:00.081613 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9435ec7-da60-49e4-9219-1a3cdb485843-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-lj6bl\" (UID: \"a9435ec7-da60-49e4-9219-1a3cdb485843\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lj6bl" Oct 31 04:49:00.081952 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:49:00.081930 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9435ec7-da60-49e4-9219-1a3cdb485843-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-lj6bl\" (UID: \"a9435ec7-da60-49e4-9219-1a3cdb485843\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lj6bl" Oct 31 04:49:00.082477 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:49:00.082455 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9435ec7-da60-49e4-9219-1a3cdb485843-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-lj6bl\" (UID: \"a9435ec7-da60-49e4-9219-1a3cdb485843\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lj6bl" Oct 31 04:49:00.082749 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:49:00.082718 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9435ec7-da60-49e4-9219-1a3cdb485843-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-lj6bl\" (UID: \"a9435ec7-da60-49e4-9219-1a3cdb485843\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lj6bl" Oct 31 04:49:00.083455 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:49:00.083435 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9435ec7-da60-49e4-9219-1a3cdb485843-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-lj6bl\" (UID: \"a9435ec7-da60-49e4-9219-1a3cdb485843\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lj6bl" Oct 31 04:49:00.084294 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:49:00.084191 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9435ec7-da60-49e4-9219-1a3cdb485843-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-lj6bl\" (UID: \"a9435ec7-da60-49e4-9219-1a3cdb485843\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lj6bl" Oct 31 04:49:00.084461 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:49:00.084269 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9435ec7-da60-49e4-9219-1a3cdb485843-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-lj6bl\" (UID: \"a9435ec7-da60-49e4-9219-1a3cdb485843\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lj6bl" Oct 31 04:49:00.084929 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:49:00.084828 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9435ec7-da60-49e4-9219-1a3cdb485843-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-lj6bl\" (UID: \"a9435ec7-da60-49e4-9219-1a3cdb485843\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lj6bl" Oct 31 04:49:00.098371 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:49:00.093151 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9435ec7-da60-49e4-9219-1a3cdb485843-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-lj6bl\" (UID: \"a9435ec7-da60-49e4-9219-1a3cdb485843\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lj6bl" Oct 31 04:49:00.098371 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:49:00.097370 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9435ec7-da60-49e4-9219-1a3cdb485843-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-lj6bl\" (UID: \"a9435ec7-da60-49e4-9219-1a3cdb485843\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lj6bl" Oct 31 04:49:00.105066 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:49:00.104495 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfvm9\" (UniqueName: \"kubernetes.io/projected/a9435ec7-da60-49e4-9219-1a3cdb485843-kube-api-access-wfvm9\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-lj6bl\" (UID: \"a9435ec7-da60-49e4-9219-1a3cdb485843\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lj6bl" Oct 31 04:49:00.105722 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:49:00.105695 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9435ec7-da60-49e4-9219-1a3cdb485843-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-lj6bl\" (UID: \"a9435ec7-da60-49e4-9219-1a3cdb485843\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lj6bl" Oct 31 04:49:00.183705 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:49:00.183677 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lj6bl" Oct 31 04:49:00.501471 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:49:00.501440 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9435ec7_da60_49e4_9219_1a3cdb485843.slice/crio-81605d6611605b7eeb8b5f1bce40c39a11e24ab1dc30937509249d733ab5a669 WatchSource:0}: Error finding container 81605d6611605b7eeb8b5f1bce40c39a11e24ab1dc30937509249d733ab5a669: Status 404 returned error can't find the container with id 81605d6611605b7eeb8b5f1bce40c39a11e24ab1dc30937509249d733ab5a669 Oct 31 04:49:00.503966 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:49:00.503944 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lj6bl"] Oct 31 04:49:00.504408 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:49:00.504377 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:49:00.504496 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:49:00.504476 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:49:00.504561 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:49:00.504524 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:49:00.761723 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:49:00.761640 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lj6bl" event={"ID":"a9435ec7-da60-49e4-9219-1a3cdb485843","Type":"ContainerStarted","Data":"a297566bb97bdba22ad788cfe14bbca8cb77f6c5f4e5c83358f4bda5fad0a4e9"} Oct 31 04:49:00.761723 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:49:00.761702 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lj6bl" event={"ID":"a9435ec7-da60-49e4-9219-1a3cdb485843","Type":"ContainerStarted","Data":"81605d6611605b7eeb8b5f1bce40c39a11e24ab1dc30937509249d733ab5a669"} Oct 31 04:49:00.812131 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:49:00.812069 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lj6bl" podStartSLOduration=1.812051241 podStartE2EDuration="1.812051241s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:48:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:49:00.80734391 +0000 UTC m=+2688.521678558" watchObservedRunningTime="2026-10-31 04:49:00.812051241 +0000 UTC m=+2688.526385860" Oct 31 04:49:01.188185 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:49:01.186081 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lj6bl" Oct 31 04:49:01.192352 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:49:01.190081 4376 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-lj6bl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" start-of-body= Oct 31 04:49:01.192352 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:49:01.190136 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lj6bl" podUID="a9435ec7-da60-49e4-9219-1a3cdb485843" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" Oct 31 04:49:02.184272 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:49:02.184232 4376 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-lj6bl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" start-of-body= Oct 31 04:49:02.184493 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:49:02.184290 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lj6bl" podUID="a9435ec7-da60-49e4-9219-1a3cdb485843" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" Oct 31 04:49:03.186352 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:49:03.184299 4376 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-lj6bl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" start-of-body= Oct 31 04:49:03.186352 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:49:03.184398 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lj6bl" podUID="a9435ec7-da60-49e4-9219-1a3cdb485843" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" Oct 31 04:49:04.191354 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:49:04.184383 4376 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-lj6bl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" start-of-body= Oct 31 04:49:04.191354 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:49:04.184457 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lj6bl" podUID="a9435ec7-da60-49e4-9219-1a3cdb485843" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" Oct 31 04:49:05.184631 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:49:05.184589 4376 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-lj6bl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" start-of-body= Oct 31 04:49:05.184823 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:49:05.184655 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lj6bl" podUID="a9435ec7-da60-49e4-9219-1a3cdb485843" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" Oct 31 04:49:06.184535 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:49:06.184501 4376 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-lj6bl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" start-of-body= Oct 31 04:49:06.184948 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:49:06.184553 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lj6bl" podUID="a9435ec7-da60-49e4-9219-1a3cdb485843" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" Oct 31 04:49:07.189744 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:49:07.189701 4376 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-lj6bl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" start-of-body= Oct 31 04:49:07.190263 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:49:07.189783 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lj6bl" podUID="a9435ec7-da60-49e4-9219-1a3cdb485843" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" Oct 31 04:49:08.185043 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:49:08.185004 4376 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-lj6bl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" start-of-body= Oct 31 04:49:08.185244 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:49:08.185081 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lj6bl" podUID="a9435ec7-da60-49e4-9219-1a3cdb485843" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" Oct 31 04:49:09.833520 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:49:09.833455 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lj6bl" Oct 31 04:49:09.834616 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:49:09.834588 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lj6bl" Oct 31 04:49:09.835671 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:49:09.835640 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lj6bl" Oct 31 04:49:11.003951 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:49:11.003907 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 04:49:41.002543 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:49:41.002504 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 04:49:46.090018 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:49:46.089974 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-8973e2-openshift-default-6bd79597f7-pdlqd"] Oct 31 04:49:46.099317 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:49:46.099282 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-8973e2-openshift-default-6bd79597f7-pdlqd" Oct 31 04:49:46.118145 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:49:46.118112 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-8973e2-openshift-default-6bd79597f7-pdlqd"] Oct 31 04:49:46.124854 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:49:46.124826 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-8973e2-openshift-default-dockercfg-lwl8l\"" Oct 31 04:49:46.218236 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:49:46.218198 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69b79bb1-ccf1-4322-a0bd-1a307906c83a-workload-socket\") pod \"e2e-extmod-a-8973e2-openshift-default-6bd79597f7-pdlqd\" (UID: \"69b79bb1-ccf1-4322-a0bd-1a307906c83a\") " pod="openshift-ingress/e2e-extmod-a-8973e2-openshift-default-6bd79597f7-pdlqd" Oct 31 04:49:46.218236 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:49:46.218242 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69b79bb1-ccf1-4322-a0bd-1a307906c83a-istio-data\") pod \"e2e-extmod-a-8973e2-openshift-default-6bd79597f7-pdlqd\" (UID: \"69b79bb1-ccf1-4322-a0bd-1a307906c83a\") " pod="openshift-ingress/e2e-extmod-a-8973e2-openshift-default-6bd79597f7-pdlqd" Oct 31 04:49:46.218478 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:49:46.218361 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69b79bb1-ccf1-4322-a0bd-1a307906c83a-istio-podinfo\") pod \"e2e-extmod-a-8973e2-openshift-default-6bd79597f7-pdlqd\" (UID: \"69b79bb1-ccf1-4322-a0bd-1a307906c83a\") " pod="openshift-ingress/e2e-extmod-a-8973e2-openshift-default-6bd79597f7-pdlqd" Oct 31 04:49:46.218478 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:49:46.218398 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69b79bb1-ccf1-4322-a0bd-1a307906c83a-istio-envoy\") pod \"e2e-extmod-a-8973e2-openshift-default-6bd79597f7-pdlqd\" (UID: \"69b79bb1-ccf1-4322-a0bd-1a307906c83a\") " pod="openshift-ingress/e2e-extmod-a-8973e2-openshift-default-6bd79597f7-pdlqd" Oct 31 04:49:46.218559 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:49:46.218539 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69b79bb1-ccf1-4322-a0bd-1a307906c83a-istiod-ca-cert\") pod \"e2e-extmod-a-8973e2-openshift-default-6bd79597f7-pdlqd\" (UID: \"69b79bb1-ccf1-4322-a0bd-1a307906c83a\") " pod="openshift-ingress/e2e-extmod-a-8973e2-openshift-default-6bd79597f7-pdlqd" Oct 31 04:49:46.218637 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:49:46.218623 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69b79bb1-ccf1-4322-a0bd-1a307906c83a-istio-token\") pod \"e2e-extmod-a-8973e2-openshift-default-6bd79597f7-pdlqd\" (UID: \"69b79bb1-ccf1-4322-a0bd-1a307906c83a\") " pod="openshift-ingress/e2e-extmod-a-8973e2-openshift-default-6bd79597f7-pdlqd" Oct 31 04:49:46.218714 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:49:46.218703 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69b79bb1-ccf1-4322-a0bd-1a307906c83a-workload-certs\") pod \"e2e-extmod-a-8973e2-openshift-default-6bd79597f7-pdlqd\" (UID: \"69b79bb1-ccf1-4322-a0bd-1a307906c83a\") " pod="openshift-ingress/e2e-extmod-a-8973e2-openshift-default-6bd79597f7-pdlqd" Oct 31 04:49:46.218792 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:49:46.218779 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69b79bb1-ccf1-4322-a0bd-1a307906c83a-credential-socket\") pod \"e2e-extmod-a-8973e2-openshift-default-6bd79597f7-pdlqd\" (UID: \"69b79bb1-ccf1-4322-a0bd-1a307906c83a\") " pod="openshift-ingress/e2e-extmod-a-8973e2-openshift-default-6bd79597f7-pdlqd" Oct 31 04:49:46.218828 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:49:46.218800 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6pbm\" (UniqueName: \"kubernetes.io/projected/69b79bb1-ccf1-4322-a0bd-1a307906c83a-kube-api-access-m6pbm\") pod \"e2e-extmod-a-8973e2-openshift-default-6bd79597f7-pdlqd\" (UID: \"69b79bb1-ccf1-4322-a0bd-1a307906c83a\") " pod="openshift-ingress/e2e-extmod-a-8973e2-openshift-default-6bd79597f7-pdlqd" Oct 31 04:49:46.320260 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:49:46.320163 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69b79bb1-ccf1-4322-a0bd-1a307906c83a-workload-certs\") pod \"e2e-extmod-a-8973e2-openshift-default-6bd79597f7-pdlqd\" (UID: \"69b79bb1-ccf1-4322-a0bd-1a307906c83a\") " pod="openshift-ingress/e2e-extmod-a-8973e2-openshift-default-6bd79597f7-pdlqd" Oct 31 04:49:46.320470 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:49:46.320377 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69b79bb1-ccf1-4322-a0bd-1a307906c83a-credential-socket\") pod \"e2e-extmod-a-8973e2-openshift-default-6bd79597f7-pdlqd\" (UID: \"69b79bb1-ccf1-4322-a0bd-1a307906c83a\") " pod="openshift-ingress/e2e-extmod-a-8973e2-openshift-default-6bd79597f7-pdlqd" Oct 31 04:49:46.320470 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:49:46.320398 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6pbm\" (UniqueName: \"kubernetes.io/projected/69b79bb1-ccf1-4322-a0bd-1a307906c83a-kube-api-access-m6pbm\") pod \"e2e-extmod-a-8973e2-openshift-default-6bd79597f7-pdlqd\" (UID: \"69b79bb1-ccf1-4322-a0bd-1a307906c83a\") " pod="openshift-ingress/e2e-extmod-a-8973e2-openshift-default-6bd79597f7-pdlqd" Oct 31 04:49:46.320470 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:49:46.320462 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69b79bb1-ccf1-4322-a0bd-1a307906c83a-workload-socket\") pod \"e2e-extmod-a-8973e2-openshift-default-6bd79597f7-pdlqd\" (UID: \"69b79bb1-ccf1-4322-a0bd-1a307906c83a\") " pod="openshift-ingress/e2e-extmod-a-8973e2-openshift-default-6bd79597f7-pdlqd" Oct 31 04:49:46.320665 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:49:46.320480 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69b79bb1-ccf1-4322-a0bd-1a307906c83a-istio-data\") pod \"e2e-extmod-a-8973e2-openshift-default-6bd79597f7-pdlqd\" (UID: \"69b79bb1-ccf1-4322-a0bd-1a307906c83a\") " pod="openshift-ingress/e2e-extmod-a-8973e2-openshift-default-6bd79597f7-pdlqd" Oct 31 04:49:46.320665 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:49:46.320528 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69b79bb1-ccf1-4322-a0bd-1a307906c83a-istio-podinfo\") pod \"e2e-extmod-a-8973e2-openshift-default-6bd79597f7-pdlqd\" (UID: \"69b79bb1-ccf1-4322-a0bd-1a307906c83a\") " pod="openshift-ingress/e2e-extmod-a-8973e2-openshift-default-6bd79597f7-pdlqd" Oct 31 04:49:46.320665 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:49:46.320531 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69b79bb1-ccf1-4322-a0bd-1a307906c83a-workload-certs\") pod \"e2e-extmod-a-8973e2-openshift-default-6bd79597f7-pdlqd\" (UID: \"69b79bb1-ccf1-4322-a0bd-1a307906c83a\") " pod="openshift-ingress/e2e-extmod-a-8973e2-openshift-default-6bd79597f7-pdlqd" Oct 31 04:49:46.320665 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:49:46.320562 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69b79bb1-ccf1-4322-a0bd-1a307906c83a-istio-envoy\") pod \"e2e-extmod-a-8973e2-openshift-default-6bd79597f7-pdlqd\" (UID: \"69b79bb1-ccf1-4322-a0bd-1a307906c83a\") " pod="openshift-ingress/e2e-extmod-a-8973e2-openshift-default-6bd79597f7-pdlqd" Oct 31 04:49:46.320893 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:49:46.320683 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69b79bb1-ccf1-4322-a0bd-1a307906c83a-istiod-ca-cert\") pod \"e2e-extmod-a-8973e2-openshift-default-6bd79597f7-pdlqd\" (UID: \"69b79bb1-ccf1-4322-a0bd-1a307906c83a\") " pod="openshift-ingress/e2e-extmod-a-8973e2-openshift-default-6bd79597f7-pdlqd" Oct 31 04:49:46.320893 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:49:46.320790 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69b79bb1-ccf1-4322-a0bd-1a307906c83a-istio-token\") pod \"e2e-extmod-a-8973e2-openshift-default-6bd79597f7-pdlqd\" (UID: \"69b79bb1-ccf1-4322-a0bd-1a307906c83a\") " pod="openshift-ingress/e2e-extmod-a-8973e2-openshift-default-6bd79597f7-pdlqd" Oct 31 04:49:46.320893 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:49:46.320789 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69b79bb1-ccf1-4322-a0bd-1a307906c83a-credential-socket\") pod \"e2e-extmod-a-8973e2-openshift-default-6bd79597f7-pdlqd\" (UID: \"69b79bb1-ccf1-4322-a0bd-1a307906c83a\") " pod="openshift-ingress/e2e-extmod-a-8973e2-openshift-default-6bd79597f7-pdlqd" Oct 31 04:49:46.321036 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:49:46.320964 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69b79bb1-ccf1-4322-a0bd-1a307906c83a-workload-socket\") pod \"e2e-extmod-a-8973e2-openshift-default-6bd79597f7-pdlqd\" (UID: \"69b79bb1-ccf1-4322-a0bd-1a307906c83a\") " pod="openshift-ingress/e2e-extmod-a-8973e2-openshift-default-6bd79597f7-pdlqd" Oct 31 04:49:46.321074 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:49:46.321024 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69b79bb1-ccf1-4322-a0bd-1a307906c83a-istio-data\") pod \"e2e-extmod-a-8973e2-openshift-default-6bd79597f7-pdlqd\" (UID: \"69b79bb1-ccf1-4322-a0bd-1a307906c83a\") " pod="openshift-ingress/e2e-extmod-a-8973e2-openshift-default-6bd79597f7-pdlqd" Oct 31 04:49:46.321649 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:49:46.321620 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69b79bb1-ccf1-4322-a0bd-1a307906c83a-istiod-ca-cert\") pod \"e2e-extmod-a-8973e2-openshift-default-6bd79597f7-pdlqd\" (UID: \"69b79bb1-ccf1-4322-a0bd-1a307906c83a\") " pod="openshift-ingress/e2e-extmod-a-8973e2-openshift-default-6bd79597f7-pdlqd" Oct 31 04:49:46.322963 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:49:46.322946 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69b79bb1-ccf1-4322-a0bd-1a307906c83a-istio-envoy\") pod \"e2e-extmod-a-8973e2-openshift-default-6bd79597f7-pdlqd\" (UID: \"69b79bb1-ccf1-4322-a0bd-1a307906c83a\") " pod="openshift-ingress/e2e-extmod-a-8973e2-openshift-default-6bd79597f7-pdlqd" Oct 31 04:49:46.323209 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:49:46.323187 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69b79bb1-ccf1-4322-a0bd-1a307906c83a-istio-podinfo\") pod \"e2e-extmod-a-8973e2-openshift-default-6bd79597f7-pdlqd\" (UID: \"69b79bb1-ccf1-4322-a0bd-1a307906c83a\") " pod="openshift-ingress/e2e-extmod-a-8973e2-openshift-default-6bd79597f7-pdlqd" Oct 31 04:49:46.344306 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:49:46.344241 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69b79bb1-ccf1-4322-a0bd-1a307906c83a-istio-token\") pod \"e2e-extmod-a-8973e2-openshift-default-6bd79597f7-pdlqd\" (UID: \"69b79bb1-ccf1-4322-a0bd-1a307906c83a\") " pod="openshift-ingress/e2e-extmod-a-8973e2-openshift-default-6bd79597f7-pdlqd" Oct 31 04:49:46.344442 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:49:46.344311 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6pbm\" (UniqueName: \"kubernetes.io/projected/69b79bb1-ccf1-4322-a0bd-1a307906c83a-kube-api-access-m6pbm\") pod \"e2e-extmod-a-8973e2-openshift-default-6bd79597f7-pdlqd\" (UID: \"69b79bb1-ccf1-4322-a0bd-1a307906c83a\") " pod="openshift-ingress/e2e-extmod-a-8973e2-openshift-default-6bd79597f7-pdlqd" Oct 31 04:49:46.416410 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:49:46.416378 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-8973e2-openshift-default-6bd79597f7-pdlqd" Oct 31 04:49:46.733517 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:49:46.733483 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69b79bb1_ccf1_4322_a0bd_1a307906c83a.slice/crio-858291b244b1069df9bf6cf6e3c042c0ed0b2f012258e03d5047303d8b35ef47 WatchSource:0}: Error finding container 858291b244b1069df9bf6cf6e3c042c0ed0b2f012258e03d5047303d8b35ef47: Status 404 returned error can't find the container with id 858291b244b1069df9bf6cf6e3c042c0ed0b2f012258e03d5047303d8b35ef47 Oct 31 04:49:46.734971 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:49:46.734939 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-8973e2-openshift-default-6bd79597f7-pdlqd"] Oct 31 04:49:46.736501 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:49:46.736466 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:49:46.736590 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:49:46.736536 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:49:46.736590 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:49:46.736565 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:49:47.310085 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:49:47.310049 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-8973e2-openshift-default-6bd79597f7-pdlqd" event={"ID":"69b79bb1-ccf1-4322-a0bd-1a307906c83a","Type":"ContainerStarted","Data":"4d4fa08898e985e3990b3c6be18814e244301190358a2585b835da704a133ab7"} Oct 31 04:49:47.310515 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:49:47.310100 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-8973e2-openshift-default-6bd79597f7-pdlqd" event={"ID":"69b79bb1-ccf1-4322-a0bd-1a307906c83a","Type":"ContainerStarted","Data":"858291b244b1069df9bf6cf6e3c042c0ed0b2f012258e03d5047303d8b35ef47"} Oct 31 04:49:47.360649 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:49:47.360589 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-8973e2-openshift-default-6bd79597f7-pdlqd" podStartSLOduration=1.36057247 podStartE2EDuration="1.36057247s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:49:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:49:47.356242446 +0000 UTC m=+2735.070577080" watchObservedRunningTime="2026-10-31 04:49:47.36057247 +0000 UTC m=+2735.074907086" Oct 31 04:49:47.416839 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:49:47.416808 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-8973e2-openshift-default-6bd79597f7-pdlqd" Oct 31 04:49:47.418383 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:49:47.418352 4376 patch_prober.go:29] interesting pod/e2e-extmod-a-8973e2-openshift-default-6bd79597f7-pdlqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" start-of-body= Oct 31 04:49:47.418522 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:49:47.418401 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-8973e2-openshift-default-6bd79597f7-pdlqd" podUID="69b79bb1-ccf1-4322-a0bd-1a307906c83a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" Oct 31 04:49:48.417708 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:49:48.417667 4376 patch_prober.go:29] interesting pod/e2e-extmod-a-8973e2-openshift-default-6bd79597f7-pdlqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" start-of-body= Oct 31 04:49:48.418123 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:49:48.417735 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-8973e2-openshift-default-6bd79597f7-pdlqd" podUID="69b79bb1-ccf1-4322-a0bd-1a307906c83a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" Oct 31 04:49:49.417286 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:49:49.417246 4376 patch_prober.go:29] interesting pod/e2e-extmod-a-8973e2-openshift-default-6bd79597f7-pdlqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" start-of-body= Oct 31 04:49:49.417495 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:49:49.417304 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-8973e2-openshift-default-6bd79597f7-pdlqd" podUID="69b79bb1-ccf1-4322-a0bd-1a307906c83a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" Oct 31 04:49:50.426359 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:49:50.417734 4376 patch_prober.go:29] interesting pod/e2e-extmod-a-8973e2-openshift-default-6bd79597f7-pdlqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" start-of-body= Oct 31 04:49:50.426359 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:49:50.417837 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-8973e2-openshift-default-6bd79597f7-pdlqd" podUID="69b79bb1-ccf1-4322-a0bd-1a307906c83a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" Oct 31 04:49:51.416880 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:49:51.416843 4376 patch_prober.go:29] interesting pod/e2e-extmod-a-8973e2-openshift-default-6bd79597f7-pdlqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" start-of-body= Oct 31 04:49:51.417073 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:49:51.416910 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-8973e2-openshift-default-6bd79597f7-pdlqd" podUID="69b79bb1-ccf1-4322-a0bd-1a307906c83a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" Oct 31 04:49:52.417216 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:49:52.417182 4376 patch_prober.go:29] interesting pod/e2e-extmod-a-8973e2-openshift-default-6bd79597f7-pdlqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" start-of-body= Oct 31 04:49:52.417645 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:49:52.417231 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-8973e2-openshift-default-6bd79597f7-pdlqd" podUID="69b79bb1-ccf1-4322-a0bd-1a307906c83a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" Oct 31 04:49:53.449660 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:49:53.449622 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-8973e2-openshift-default-6bd79597f7-pdlqd" Oct 31 04:49:53.450621 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:49:53.450595 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-8973e2-openshift-default-6bd79597f7-pdlqd" Oct 31 04:49:53.453676 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:49:53.453612 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-8973e2-openshift-default-6bd79597f7-pdlqd" Oct 31 04:49:54.548684 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:49:54.548650 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 04:50:11.001127 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:50:11.001086 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 04:50:14.036566 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:50:14.036531 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b29b642e-cn6jt"] Oct 31 04:50:14.048686 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:50:14.048654 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b29b642e-cn6jt"] Oct 31 04:50:14.445172 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:50:14.445134 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4dcd3fe-2bd0-43c9-8e61-12003a99f3c8" path="/var/lib/kubelet/pods/f4dcd3fe-2bd0-43c9-8e61-12003a99f3c8/volumes" Oct 31 04:50:41.001186 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:50:41.001141 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 04:50:52.563064 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:50:52.563028 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 04:51:11.001392 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:51:11.001354 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 04:51:16.638987 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:51:16.638950 4376 scope.go:122] "RemoveContainer" containerID="4ff405c0a49e86d320cc0c0013ec6ebaff403a669e12eccefd3a06772019652e" Oct 31 04:51:41.001767 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:51:41.001722 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 04:52:11.001815 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:52:11.001775 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 04:52:15.664809 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:52:15.664776 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 04:52:41.004568 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:52:41.004529 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 04:53:08.740868 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:53:08.740834 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 04:53:11.014455 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:53:11.014421 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 04:53:41.004906 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:53:41.004865 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 04:54:11.002950 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:54:11.002912 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 04:54:13.219040 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:54:13.218994 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 04:54:28.870828 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:54:28.870746 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-lvnb4"] Oct 31 04:54:28.871273 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:54:28.871029 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-lvnb4" podUID="6346bb15-5ba9-4c29-b74b-3e937ba64963" containerName="istio-proxy" containerID="cri-o://d5a316e75c816b9c855bfb28ffc126afaf4070eb4cd27ae985624dd1aa548ee2" gracePeriod=30 Oct 31 04:54:32.894412 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:54:32.894380 4376 patch_prober.go:29] interesting pod/e2e-shared-b-w2-031745-openshift-default-7f667bc645-lvnb4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 04:54:32.894839 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:54:32.894458 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-lvnb4" podUID="6346bb15-5ba9-4c29-b74b-3e937ba64963" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 04:54:34.145655 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:54:34.145596 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-lvnb4" Oct 31 04:54:34.256705 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:54:34.256669 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6346bb15-5ba9-4c29-b74b-3e937ba64963-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6346bb15-5ba9-4c29-b74b-3e937ba64963-credential-socket\") pod \"6346bb15-5ba9-4c29-b74b-3e937ba64963\" (UID: \"6346bb15-5ba9-4c29-b74b-3e937ba64963\") " Oct 31 04:54:34.256976 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:54:34.256723 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6346bb15-5ba9-4c29-b74b-3e937ba64963-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6346bb15-5ba9-4c29-b74b-3e937ba64963-istio-envoy\") pod \"6346bb15-5ba9-4c29-b74b-3e937ba64963\" (UID: \"6346bb15-5ba9-4c29-b74b-3e937ba64963\") " Oct 31 04:54:34.256976 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:54:34.256788 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6346bb15-5ba9-4c29-b74b-3e937ba64963-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6346bb15-5ba9-4c29-b74b-3e937ba64963-istiod-ca-cert\") pod \"6346bb15-5ba9-4c29-b74b-3e937ba64963\" (UID: \"6346bb15-5ba9-4c29-b74b-3e937ba64963\") " Oct 31 04:54:34.257117 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:54:34.257003 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6346bb15-5ba9-4c29-b74b-3e937ba64963-credential-socket" pod "6346bb15-5ba9-4c29-b74b-3e937ba64963" (UID: "6346bb15-5ba9-4c29-b74b-3e937ba64963"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:54:34.257117 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:54:34.257029 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6346bb15-5ba9-4c29-b74b-3e937ba64963-kube-api-access-sq2qs\" (UniqueName: \"kubernetes.io/projected/6346bb15-5ba9-4c29-b74b-3e937ba64963-kube-api-access-sq2qs\") pod \"6346bb15-5ba9-4c29-b74b-3e937ba64963\" (UID: \"6346bb15-5ba9-4c29-b74b-3e937ba64963\") " Oct 31 04:54:34.257234 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:54:34.257208 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6346bb15-5ba9-4c29-b74b-3e937ba64963-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6346bb15-5ba9-4c29-b74b-3e937ba64963-workload-certs\") pod \"6346bb15-5ba9-4c29-b74b-3e937ba64963\" (UID: \"6346bb15-5ba9-4c29-b74b-3e937ba64963\") " Oct 31 04:54:34.257308 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:54:34.257265 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6346bb15-5ba9-4c29-b74b-3e937ba64963-istio-token\" (UniqueName: \"kubernetes.io/projected/6346bb15-5ba9-4c29-b74b-3e937ba64963-istio-token\") pod \"6346bb15-5ba9-4c29-b74b-3e937ba64963\" (UID: \"6346bb15-5ba9-4c29-b74b-3e937ba64963\") " Oct 31 04:54:34.257308 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:54:34.257300 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6346bb15-5ba9-4c29-b74b-3e937ba64963-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6346bb15-5ba9-4c29-b74b-3e937ba64963-istio-data\") pod \"6346bb15-5ba9-4c29-b74b-3e937ba64963\" (UID: \"6346bb15-5ba9-4c29-b74b-3e937ba64963\") " Oct 31 04:54:34.257519 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:54:34.257318 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6346bb15-5ba9-4c29-b74b-3e937ba64963-istiod-ca-cert" pod "6346bb15-5ba9-4c29-b74b-3e937ba64963" (UID: "6346bb15-5ba9-4c29-b74b-3e937ba64963"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:54:34.257519 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:54:34.257400 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6346bb15-5ba9-4c29-b74b-3e937ba64963-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6346bb15-5ba9-4c29-b74b-3e937ba64963-istio-podinfo\") pod \"6346bb15-5ba9-4c29-b74b-3e937ba64963\" (UID: \"6346bb15-5ba9-4c29-b74b-3e937ba64963\") " Oct 31 04:54:34.257519 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:54:34.257435 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6346bb15-5ba9-4c29-b74b-3e937ba64963-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6346bb15-5ba9-4c29-b74b-3e937ba64963-workload-socket\") pod \"6346bb15-5ba9-4c29-b74b-3e937ba64963\" (UID: \"6346bb15-5ba9-4c29-b74b-3e937ba64963\") " Oct 31 04:54:34.257915 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:54:34.257881 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6346bb15-5ba9-4c29-b74b-3e937ba64963-workload-socket" pod "6346bb15-5ba9-4c29-b74b-3e937ba64963" (UID: "6346bb15-5ba9-4c29-b74b-3e937ba64963"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:54:34.257915 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:54:34.257898 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6346bb15-5ba9-4c29-b74b-3e937ba64963-istio-data" pod "6346bb15-5ba9-4c29-b74b-3e937ba64963" (UID: "6346bb15-5ba9-4c29-b74b-3e937ba64963"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:54:34.258089 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:54:34.257958 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6346bb15-5ba9-4c29-b74b-3e937ba64963-workload-certs" pod "6346bb15-5ba9-4c29-b74b-3e937ba64963" (UID: "6346bb15-5ba9-4c29-b74b-3e937ba64963"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:54:34.258299 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:54:34.258277 4376 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6346bb15-5ba9-4c29-b74b-3e937ba64963-istiod-ca-cert\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:54:34.258425 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:54:34.258304 4376 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6346bb15-5ba9-4c29-b74b-3e937ba64963-workload-certs\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:54:34.258425 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:54:34.258320 4376 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6346bb15-5ba9-4c29-b74b-3e937ba64963-istio-data\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:54:34.258425 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:54:34.258349 4376 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6346bb15-5ba9-4c29-b74b-3e937ba64963-workload-socket\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:54:34.258425 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:54:34.258367 4376 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6346bb15-5ba9-4c29-b74b-3e937ba64963-credential-socket\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:54:34.259897 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:54:34.259868 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6346bb15-5ba9-4c29-b74b-3e937ba64963-istio-token" pod "6346bb15-5ba9-4c29-b74b-3e937ba64963" (UID: "6346bb15-5ba9-4c29-b74b-3e937ba64963"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:54:34.260750 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:54:34.260716 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6346bb15-5ba9-4c29-b74b-3e937ba64963-istio-envoy" pod "6346bb15-5ba9-4c29-b74b-3e937ba64963" (UID: "6346bb15-5ba9-4c29-b74b-3e937ba64963"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:54:34.260883 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:54:34.260837 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6346bb15-5ba9-4c29-b74b-3e937ba64963-kube-api-access-sq2qs" pod "6346bb15-5ba9-4c29-b74b-3e937ba64963" (UID: "6346bb15-5ba9-4c29-b74b-3e937ba64963"). InnerVolumeSpecName "kube-api-access-sq2qs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:54:34.260956 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:54:34.260875 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6346bb15-5ba9-4c29-b74b-3e937ba64963-istio-podinfo" pod "6346bb15-5ba9-4c29-b74b-3e937ba64963" (UID: "6346bb15-5ba9-4c29-b74b-3e937ba64963"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 04:54:34.359590 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:54:34.359559 4376 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6346bb15-5ba9-4c29-b74b-3e937ba64963-istio-token\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:54:34.359590 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:54:34.359587 4376 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6346bb15-5ba9-4c29-b74b-3e937ba64963-istio-podinfo\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:54:34.359790 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:54:34.359600 4376 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6346bb15-5ba9-4c29-b74b-3e937ba64963-istio-envoy\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:54:34.359790 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:54:34.359613 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sq2qs\" (UniqueName: \"kubernetes.io/projected/6346bb15-5ba9-4c29-b74b-3e937ba64963-kube-api-access-sq2qs\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:54:34.561775 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:54:34.561741 4376 generic.go:356] "Generic (PLEG): container finished" podID="6346bb15-5ba9-4c29-b74b-3e937ba64963" containerID="d5a316e75c816b9c855bfb28ffc126afaf4070eb4cd27ae985624dd1aa548ee2" exitCode=0 Oct 31 04:54:34.561942 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:54:34.561785 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-lvnb4" Oct 31 04:54:34.561942 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:54:34.561806 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-lvnb4" event={"ID":"6346bb15-5ba9-4c29-b74b-3e937ba64963","Type":"ContainerDied","Data":"d5a316e75c816b9c855bfb28ffc126afaf4070eb4cd27ae985624dd1aa548ee2"} Oct 31 04:54:34.561942 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:54:34.561852 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-lvnb4" event={"ID":"6346bb15-5ba9-4c29-b74b-3e937ba64963","Type":"ContainerDied","Data":"76ca0efe825d7c34006833a57ee39305fc5516a3159c8d509d339283fd741eee"} Oct 31 04:54:34.561942 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:54:34.561881 4376 scope.go:122] "RemoveContainer" containerID="d5a316e75c816b9c855bfb28ffc126afaf4070eb4cd27ae985624dd1aa548ee2" Oct 31 04:54:34.572555 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:54:34.572536 4376 scope.go:122] "RemoveContainer" containerID="d5a316e75c816b9c855bfb28ffc126afaf4070eb4cd27ae985624dd1aa548ee2" Oct 31 04:54:34.572898 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:54:34.572869 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5a316e75c816b9c855bfb28ffc126afaf4070eb4cd27ae985624dd1aa548ee2\": container with ID starting with d5a316e75c816b9c855bfb28ffc126afaf4070eb4cd27ae985624dd1aa548ee2 not found: ID does not exist" containerID="d5a316e75c816b9c855bfb28ffc126afaf4070eb4cd27ae985624dd1aa548ee2" Oct 31 04:54:34.572993 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:54:34.572905 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5a316e75c816b9c855bfb28ffc126afaf4070eb4cd27ae985624dd1aa548ee2"} err="failed to get container status \"d5a316e75c816b9c855bfb28ffc126afaf4070eb4cd27ae985624dd1aa548ee2\": rpc error: code = NotFound desc = could not find container \"d5a316e75c816b9c855bfb28ffc126afaf4070eb4cd27ae985624dd1aa548ee2\": container with ID starting with d5a316e75c816b9c855bfb28ffc126afaf4070eb4cd27ae985624dd1aa548ee2 not found: ID does not exist" Oct 31 04:54:34.618738 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:54:34.618713 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-lvnb4"] Oct 31 04:54:34.631255 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:54:34.631224 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-lvnb4"] Oct 31 04:54:36.445647 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:54:36.445616 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6346bb15-5ba9-4c29-b74b-3e937ba64963" path="/var/lib/kubelet/pods/6346bb15-5ba9-4c29-b74b-3e937ba64963/volumes" Oct 31 04:54:41.001367 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:54:41.001305 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 04:54:45.284787 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:54:45.284749 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-64c9jc4hg"] Oct 31 04:54:45.285625 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:54:45.285608 4376 cpu_manager.go:414] "containerMap: removing container" podUID="6346bb15-5ba9-4c29-b74b-3e937ba64963" containerName="istio-proxy" Oct 31 04:54:45.285678 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:54:45.285627 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6346bb15-5ba9-4c29-b74b-3e937ba64963" containerName="istio-proxy" Oct 31 04:54:45.285791 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:54:45.285779 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="6346bb15-5ba9-4c29-b74b-3e937ba64963" containerName="istio-proxy" Oct 31 04:54:45.289569 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:54:45.289553 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-64c9jc4hg" Oct 31 04:54:45.309222 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:54:45.309194 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-64c9jc4hg"] Oct 31 04:54:45.317158 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:54:45.317133 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-567ca6\"/\"agc-cred-fcc1f4ac9c30\"" Oct 31 04:54:45.317287 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:54:45.317234 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-567ca6\"/\"payload-processing-external-model-plugins-e2e-extmod-b-567ca6\"" Oct 31 04:54:45.317287 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:54:45.317278 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-567ca6\"/\"kube-root-ca.crt\"" Oct 31 04:54:45.320288 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:54:45.320270 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-567ca6\"/\"routing-overlay\"" Oct 31 04:54:45.320288 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:54:45.320282 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-567ca6\"/\"payload-processing-external-model-e2e--a8f34224-dockercfg-gc8kx\"" Oct 31 04:54:45.320479 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:54:45.320391 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-567ca6\"/\"openshift-service-ca.crt\"" Oct 31 04:54:45.332511 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:54:45.332485 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6e65a42-e762-4a09-9fae-f65ccac885fd-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-567ca6-64c9jc4hg\" (UID: \"c6e65a42-e762-4a09-9fae-f65ccac885fd\") " pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-64c9jc4hg" Oct 31 04:54:45.332604 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:54:45.332528 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwm86\" (UniqueName: \"kubernetes.io/projected/c6e65a42-e762-4a09-9fae-f65ccac885fd-kube-api-access-fwm86\") pod \"payload-processing-external-model-e2e-extmod-b-567ca6-64c9jc4hg\" (UID: \"c6e65a42-e762-4a09-9fae-f65ccac885fd\") " pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-64c9jc4hg" Oct 31 04:54:45.332663 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:54:45.332644 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c6e65a42-e762-4a09-9fae-f65ccac885fd-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-567ca6-64c9jc4hg\" (UID: \"c6e65a42-e762-4a09-9fae-f65ccac885fd\") " pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-64c9jc4hg" Oct 31 04:54:45.332749 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:54:45.332734 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c6e65a42-e762-4a09-9fae-f65ccac885fd-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-567ca6-64c9jc4hg\" (UID: \"c6e65a42-e762-4a09-9fae-f65ccac885fd\") " pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-64c9jc4hg" Oct 31 04:54:45.434516 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:54:45.434481 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6e65a42-e762-4a09-9fae-f65ccac885fd-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-567ca6-64c9jc4hg\" (UID: \"c6e65a42-e762-4a09-9fae-f65ccac885fd\") " pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-64c9jc4hg" Oct 31 04:54:45.434673 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:54:45.434544 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwm86\" (UniqueName: \"kubernetes.io/projected/c6e65a42-e762-4a09-9fae-f65ccac885fd-kube-api-access-fwm86\") pod \"payload-processing-external-model-e2e-extmod-b-567ca6-64c9jc4hg\" (UID: \"c6e65a42-e762-4a09-9fae-f65ccac885fd\") " pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-64c9jc4hg" Oct 31 04:54:45.434673 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:54:45.434599 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c6e65a42-e762-4a09-9fae-f65ccac885fd-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-567ca6-64c9jc4hg\" (UID: \"c6e65a42-e762-4a09-9fae-f65ccac885fd\") " pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-64c9jc4hg" Oct 31 04:54:45.434806 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:54:45.434684 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c6e65a42-e762-4a09-9fae-f65ccac885fd-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-567ca6-64c9jc4hg\" (UID: \"c6e65a42-e762-4a09-9fae-f65ccac885fd\") " pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-64c9jc4hg" Oct 31 04:54:45.435440 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:54:45.435412 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c6e65a42-e762-4a09-9fae-f65ccac885fd-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-567ca6-64c9jc4hg\" (UID: \"c6e65a42-e762-4a09-9fae-f65ccac885fd\") " pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-64c9jc4hg" Oct 31 04:54:45.435574 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:54:45.435451 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6e65a42-e762-4a09-9fae-f65ccac885fd-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-567ca6-64c9jc4hg\" (UID: \"c6e65a42-e762-4a09-9fae-f65ccac885fd\") " pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-64c9jc4hg" Oct 31 04:54:45.437582 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:54:45.437555 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c6e65a42-e762-4a09-9fae-f65ccac885fd-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-567ca6-64c9jc4hg\" (UID: \"c6e65a42-e762-4a09-9fae-f65ccac885fd\") " pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-64c9jc4hg" Oct 31 04:54:45.458240 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:54:45.458215 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwm86\" (UniqueName: \"kubernetes.io/projected/c6e65a42-e762-4a09-9fae-f65ccac885fd-kube-api-access-fwm86\") pod \"payload-processing-external-model-e2e-extmod-b-567ca6-64c9jc4hg\" (UID: \"c6e65a42-e762-4a09-9fae-f65ccac885fd\") " pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-64c9jc4hg" Oct 31 04:54:45.604012 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:54:45.603944 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-64c9jc4hg" Oct 31 04:54:45.907955 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:54:45.907928 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6e65a42_e762_4a09_9fae_f65ccac885fd.slice/crio-821c55376a656592efa892e2d320c57f7f4dab5c999356b5b5131af05d4f5ed6 WatchSource:0}: Error finding container 821c55376a656592efa892e2d320c57f7f4dab5c999356b5b5131af05d4f5ed6: Status 404 returned error can't find the container with id 821c55376a656592efa892e2d320c57f7f4dab5c999356b5b5131af05d4f5ed6 Oct 31 04:54:45.910022 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:54:45.909986 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-64c9jc4hg"] Oct 31 04:54:46.712621 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:54:46.712579 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-64c9jc4hg" event={"ID":"c6e65a42-e762-4a09-9fae-f65ccac885fd","Type":"ContainerStarted","Data":"ff36668d3895c4b2d5ec802736b93e9eb7cb32300437e1da74ea795e8f498302"} Oct 31 04:54:46.713116 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:54:46.712640 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-64c9jc4hg" event={"ID":"c6e65a42-e762-4a09-9fae-f65ccac885fd","Type":"ContainerStarted","Data":"821c55376a656592efa892e2d320c57f7f4dab5c999356b5b5131af05d4f5ed6"} Oct 31 04:54:46.713116 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:54:46.712661 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-64c9jc4hg" Oct 31 04:54:46.756022 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:54:46.755962 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-64c9jc4hg" podStartSLOduration=1.755950054 podStartE2EDuration="1.755950054s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:54:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:54:46.753882633 +0000 UTC m=+3034.468217268" watchObservedRunningTime="2026-10-31 04:54:46.755950054 +0000 UTC m=+3034.470284671" Oct 31 04:54:57.722895 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:54:57.722870 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-64c9jc4hg" Oct 31 04:55:08.563718 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:55:08.563677 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-64c9jc4hg"] Oct 31 04:55:11.000472 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:55:11.000439 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 04:55:13.564828 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:55:13.564758 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-64c9jc4hg" podUID="c6e65a42-e762-4a09-9fae-f65ccac885fd" containerName="payload-processing" containerID="cri-o://ff36668d3895c4b2d5ec802736b93e9eb7cb32300437e1da74ea795e8f498302" gracePeriod=25 Oct 31 04:55:13.849318 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:55:13.849292 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-64c9jc4hg" Oct 31 04:55:13.907830 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:55:13.907800 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6e65a42-e762-4a09-9fae-f65ccac885fd-kube-api-access-fwm86\" (UniqueName: \"kubernetes.io/projected/c6e65a42-e762-4a09-9fae-f65ccac885fd-kube-api-access-fwm86\") pod \"c6e65a42-e762-4a09-9fae-f65ccac885fd\" (UID: \"c6e65a42-e762-4a09-9fae-f65ccac885fd\") " Oct 31 04:55:13.907990 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:55:13.907940 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6e65a42-e762-4a09-9fae-f65ccac885fd-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c6e65a42-e762-4a09-9fae-f65ccac885fd-routing-overlay-volume\") pod \"c6e65a42-e762-4a09-9fae-f65ccac885fd\" (UID: \"c6e65a42-e762-4a09-9fae-f65ccac885fd\") " Oct 31 04:55:13.908300 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:55:13.908067 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6e65a42-e762-4a09-9fae-f65ccac885fd-provider-credentials\" (UniqueName: \"kubernetes.io/projected/c6e65a42-e762-4a09-9fae-f65ccac885fd-provider-credentials\") pod \"c6e65a42-e762-4a09-9fae-f65ccac885fd\" (UID: \"c6e65a42-e762-4a09-9fae-f65ccac885fd\") " Oct 31 04:55:13.908466 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:55:13.908349 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6e65a42-e762-4a09-9fae-f65ccac885fd-routing-overlay-volume" pod "c6e65a42-e762-4a09-9fae-f65ccac885fd" (UID: "c6e65a42-e762-4a09-9fae-f65ccac885fd"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:55:13.908836 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:55:13.908799 4376 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c6e65a42-e762-4a09-9fae-f65ccac885fd-routing-overlay-volume\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:55:13.910914 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:55:13.910872 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6e65a42-e762-4a09-9fae-f65ccac885fd-kube-api-access-fwm86" pod "c6e65a42-e762-4a09-9fae-f65ccac885fd" (UID: "c6e65a42-e762-4a09-9fae-f65ccac885fd"). InnerVolumeSpecName "kube-api-access-fwm86". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:55:13.911757 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:55:13.911730 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6e65a42-e762-4a09-9fae-f65ccac885fd-provider-credentials" pod "c6e65a42-e762-4a09-9fae-f65ccac885fd" (UID: "c6e65a42-e762-4a09-9fae-f65ccac885fd"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:55:13.943172 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:55:13.943148 4376 generic.go:356] "Generic (PLEG): container finished" podID="c6e65a42-e762-4a09-9fae-f65ccac885fd" containerID="ff36668d3895c4b2d5ec802736b93e9eb7cb32300437e1da74ea795e8f498302" exitCode=0 Oct 31 04:55:13.943300 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:55:13.943195 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-64c9jc4hg" Oct 31 04:55:13.943300 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:55:13.943242 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-64c9jc4hg" event={"ID":"c6e65a42-e762-4a09-9fae-f65ccac885fd","Type":"ContainerDied","Data":"ff36668d3895c4b2d5ec802736b93e9eb7cb32300437e1da74ea795e8f498302"} Oct 31 04:55:13.943300 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:55:13.943284 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-64c9jc4hg" event={"ID":"c6e65a42-e762-4a09-9fae-f65ccac885fd","Type":"ContainerDied","Data":"821c55376a656592efa892e2d320c57f7f4dab5c999356b5b5131af05d4f5ed6"} Oct 31 04:55:13.943507 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:55:13.943310 4376 scope.go:122] "RemoveContainer" containerID="ff36668d3895c4b2d5ec802736b93e9eb7cb32300437e1da74ea795e8f498302" Oct 31 04:55:13.954234 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:55:13.954218 4376 scope.go:122] "RemoveContainer" containerID="ff36668d3895c4b2d5ec802736b93e9eb7cb32300437e1da74ea795e8f498302" Oct 31 04:55:13.954502 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:55:13.954474 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff36668d3895c4b2d5ec802736b93e9eb7cb32300437e1da74ea795e8f498302\": container with ID starting with ff36668d3895c4b2d5ec802736b93e9eb7cb32300437e1da74ea795e8f498302 not found: ID does not exist" containerID="ff36668d3895c4b2d5ec802736b93e9eb7cb32300437e1da74ea795e8f498302" Oct 31 04:55:13.954565 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:55:13.954513 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff36668d3895c4b2d5ec802736b93e9eb7cb32300437e1da74ea795e8f498302"} err="failed to get container status \"ff36668d3895c4b2d5ec802736b93e9eb7cb32300437e1da74ea795e8f498302\": rpc error: code = NotFound desc = could not find container \"ff36668d3895c4b2d5ec802736b93e9eb7cb32300437e1da74ea795e8f498302\": container with ID starting with ff36668d3895c4b2d5ec802736b93e9eb7cb32300437e1da74ea795e8f498302 not found: ID does not exist" Oct 31 04:55:14.009751 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:55:14.009725 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6e65a42-e762-4a09-9fae-f65ccac885fd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6e65a42-e762-4a09-9fae-f65ccac885fd-plugins-config-volume\") pod \"c6e65a42-e762-4a09-9fae-f65ccac885fd\" (UID: \"c6e65a42-e762-4a09-9fae-f65ccac885fd\") " Oct 31 04:55:14.010056 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:55:14.010037 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6e65a42-e762-4a09-9fae-f65ccac885fd-plugins-config-volume" pod "c6e65a42-e762-4a09-9fae-f65ccac885fd" (UID: "c6e65a42-e762-4a09-9fae-f65ccac885fd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:55:14.010276 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:55:14.010264 4376 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c6e65a42-e762-4a09-9fae-f65ccac885fd-provider-credentials\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:55:14.010314 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:55:14.010280 4376 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6e65a42-e762-4a09-9fae-f65ccac885fd-plugins-config-volume\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:55:14.010314 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:55:14.010290 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwm86\" (UniqueName: \"kubernetes.io/projected/c6e65a42-e762-4a09-9fae-f65ccac885fd-kube-api-access-fwm86\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 04:55:14.295469 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:55:14.295428 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-64c9jc4hg"] Oct 31 04:55:14.305760 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:55:14.305734 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-64c9jc4hg"] Oct 31 04:55:14.444244 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:55:14.444219 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6e65a42-e762-4a09-9fae-f65ccac885fd" path="/var/lib/kubelet/pods/c6e65a42-e762-4a09-9fae-f65ccac885fd/volumes" Oct 31 04:55:15.044618 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:55:15.044587 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-04d111-openshift-default-cd549dc77-f9kwz"] Oct 31 04:55:15.045359 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:55:15.045340 4376 cpu_manager.go:414] "containerMap: removing container" podUID="c6e65a42-e762-4a09-9fae-f65ccac885fd" containerName="payload-processing" Oct 31 04:55:15.045359 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:55:15.045357 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6e65a42-e762-4a09-9fae-f65ccac885fd" containerName="payload-processing" Oct 31 04:55:15.045492 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:55:15.045476 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6e65a42-e762-4a09-9fae-f65ccac885fd" containerName="payload-processing" Oct 31 04:55:15.049316 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:55:15.049293 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-04d111-openshift-default-cd549dc77-f9kwz" Oct 31 04:55:15.066848 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:55:15.066822 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-04d111-openshift-default-dockercfg-fxscp\"" Oct 31 04:55:15.084515 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:55:15.084495 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-04d111-openshift-default-cd549dc77-f9kwz"] Oct 31 04:55:15.140257 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:55:15.140230 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96a130da-eaed-4765-905b-261822ce8014-istio-envoy\") pod \"e2e-inf-a-04d111-openshift-default-cd549dc77-f9kwz\" (UID: \"96a130da-eaed-4765-905b-261822ce8014\") " pod="openshift-ingress/e2e-inf-a-04d111-openshift-default-cd549dc77-f9kwz" Oct 31 04:55:15.140418 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:55:15.140267 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96a130da-eaed-4765-905b-261822ce8014-istio-podinfo\") pod \"e2e-inf-a-04d111-openshift-default-cd549dc77-f9kwz\" (UID: \"96a130da-eaed-4765-905b-261822ce8014\") " pod="openshift-ingress/e2e-inf-a-04d111-openshift-default-cd549dc77-f9kwz" Oct 31 04:55:15.140418 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:55:15.140373 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96a130da-eaed-4765-905b-261822ce8014-istiod-ca-cert\") pod \"e2e-inf-a-04d111-openshift-default-cd549dc77-f9kwz\" (UID: \"96a130da-eaed-4765-905b-261822ce8014\") " pod="openshift-ingress/e2e-inf-a-04d111-openshift-default-cd549dc77-f9kwz" Oct 31 04:55:15.140418 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:55:15.140390 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4xsf\" (UniqueName: \"kubernetes.io/projected/96a130da-eaed-4765-905b-261822ce8014-kube-api-access-d4xsf\") pod \"e2e-inf-a-04d111-openshift-default-cd549dc77-f9kwz\" (UID: \"96a130da-eaed-4765-905b-261822ce8014\") " pod="openshift-ingress/e2e-inf-a-04d111-openshift-default-cd549dc77-f9kwz" Oct 31 04:55:15.140418 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:55:15.140414 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96a130da-eaed-4765-905b-261822ce8014-workload-socket\") pod \"e2e-inf-a-04d111-openshift-default-cd549dc77-f9kwz\" (UID: \"96a130da-eaed-4765-905b-261822ce8014\") " pod="openshift-ingress/e2e-inf-a-04d111-openshift-default-cd549dc77-f9kwz" Oct 31 04:55:15.140603 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:55:15.140559 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96a130da-eaed-4765-905b-261822ce8014-istio-token\") pod \"e2e-inf-a-04d111-openshift-default-cd549dc77-f9kwz\" (UID: \"96a130da-eaed-4765-905b-261822ce8014\") " pod="openshift-ingress/e2e-inf-a-04d111-openshift-default-cd549dc77-f9kwz" Oct 31 04:55:15.140744 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:55:15.140729 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96a130da-eaed-4765-905b-261822ce8014-workload-certs\") pod \"e2e-inf-a-04d111-openshift-default-cd549dc77-f9kwz\" (UID: \"96a130da-eaed-4765-905b-261822ce8014\") " pod="openshift-ingress/e2e-inf-a-04d111-openshift-default-cd549dc77-f9kwz" Oct 31 04:55:15.140851 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:55:15.140831 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96a130da-eaed-4765-905b-261822ce8014-credential-socket\") pod \"e2e-inf-a-04d111-openshift-default-cd549dc77-f9kwz\" (UID: \"96a130da-eaed-4765-905b-261822ce8014\") " pod="openshift-ingress/e2e-inf-a-04d111-openshift-default-cd549dc77-f9kwz" Oct 31 04:55:15.140906 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:55:15.140895 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96a130da-eaed-4765-905b-261822ce8014-istio-data\") pod \"e2e-inf-a-04d111-openshift-default-cd549dc77-f9kwz\" (UID: \"96a130da-eaed-4765-905b-261822ce8014\") " pod="openshift-ingress/e2e-inf-a-04d111-openshift-default-cd549dc77-f9kwz" Oct 31 04:55:15.251360 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:55:15.248396 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96a130da-eaed-4765-905b-261822ce8014-credential-socket\") pod \"e2e-inf-a-04d111-openshift-default-cd549dc77-f9kwz\" (UID: \"96a130da-eaed-4765-905b-261822ce8014\") " pod="openshift-ingress/e2e-inf-a-04d111-openshift-default-cd549dc77-f9kwz" Oct 31 04:55:15.251360 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:55:15.248506 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96a130da-eaed-4765-905b-261822ce8014-istio-data\") pod \"e2e-inf-a-04d111-openshift-default-cd549dc77-f9kwz\" (UID: \"96a130da-eaed-4765-905b-261822ce8014\") " pod="openshift-ingress/e2e-inf-a-04d111-openshift-default-cd549dc77-f9kwz" Oct 31 04:55:15.251360 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:55:15.248585 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96a130da-eaed-4765-905b-261822ce8014-istio-envoy\") pod \"e2e-inf-a-04d111-openshift-default-cd549dc77-f9kwz\" (UID: \"96a130da-eaed-4765-905b-261822ce8014\") " pod="openshift-ingress/e2e-inf-a-04d111-openshift-default-cd549dc77-f9kwz" Oct 31 04:55:15.251360 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:55:15.248615 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96a130da-eaed-4765-905b-261822ce8014-istio-podinfo\") pod \"e2e-inf-a-04d111-openshift-default-cd549dc77-f9kwz\" (UID: \"96a130da-eaed-4765-905b-261822ce8014\") " pod="openshift-ingress/e2e-inf-a-04d111-openshift-default-cd549dc77-f9kwz" Oct 31 04:55:15.251360 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:55:15.248680 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96a130da-eaed-4765-905b-261822ce8014-istiod-ca-cert\") pod \"e2e-inf-a-04d111-openshift-default-cd549dc77-f9kwz\" (UID: \"96a130da-eaed-4765-905b-261822ce8014\") " pod="openshift-ingress/e2e-inf-a-04d111-openshift-default-cd549dc77-f9kwz" Oct 31 04:55:15.251360 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:55:15.248703 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4xsf\" (UniqueName: \"kubernetes.io/projected/96a130da-eaed-4765-905b-261822ce8014-kube-api-access-d4xsf\") pod \"e2e-inf-a-04d111-openshift-default-cd549dc77-f9kwz\" (UID: \"96a130da-eaed-4765-905b-261822ce8014\") " pod="openshift-ingress/e2e-inf-a-04d111-openshift-default-cd549dc77-f9kwz" Oct 31 04:55:15.251360 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:55:15.248739 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96a130da-eaed-4765-905b-261822ce8014-workload-socket\") pod \"e2e-inf-a-04d111-openshift-default-cd549dc77-f9kwz\" (UID: \"96a130da-eaed-4765-905b-261822ce8014\") " pod="openshift-ingress/e2e-inf-a-04d111-openshift-default-cd549dc77-f9kwz" Oct 31 04:55:15.251360 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:55:15.248819 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96a130da-eaed-4765-905b-261822ce8014-istio-token\") pod \"e2e-inf-a-04d111-openshift-default-cd549dc77-f9kwz\" (UID: \"96a130da-eaed-4765-905b-261822ce8014\") " pod="openshift-ingress/e2e-inf-a-04d111-openshift-default-cd549dc77-f9kwz" Oct 31 04:55:15.251360 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:55:15.249219 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96a130da-eaed-4765-905b-261822ce8014-workload-certs\") pod \"e2e-inf-a-04d111-openshift-default-cd549dc77-f9kwz\" (UID: \"96a130da-eaed-4765-905b-261822ce8014\") " pod="openshift-ingress/e2e-inf-a-04d111-openshift-default-cd549dc77-f9kwz" Oct 31 04:55:15.266351 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:55:15.261618 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96a130da-eaed-4765-905b-261822ce8014-workload-certs\") pod \"e2e-inf-a-04d111-openshift-default-cd549dc77-f9kwz\" (UID: \"96a130da-eaed-4765-905b-261822ce8014\") " pod="openshift-ingress/e2e-inf-a-04d111-openshift-default-cd549dc77-f9kwz" Oct 31 04:55:15.266351 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:55:15.261931 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96a130da-eaed-4765-905b-261822ce8014-credential-socket\") pod \"e2e-inf-a-04d111-openshift-default-cd549dc77-f9kwz\" (UID: \"96a130da-eaed-4765-905b-261822ce8014\") " pod="openshift-ingress/e2e-inf-a-04d111-openshift-default-cd549dc77-f9kwz" Oct 31 04:55:15.266351 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:55:15.262172 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96a130da-eaed-4765-905b-261822ce8014-istio-data\") pod \"e2e-inf-a-04d111-openshift-default-cd549dc77-f9kwz\" (UID: \"96a130da-eaed-4765-905b-261822ce8014\") " pod="openshift-ingress/e2e-inf-a-04d111-openshift-default-cd549dc77-f9kwz" Oct 31 04:55:15.266351 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:55:15.263634 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96a130da-eaed-4765-905b-261822ce8014-workload-socket\") pod \"e2e-inf-a-04d111-openshift-default-cd549dc77-f9kwz\" (UID: \"96a130da-eaed-4765-905b-261822ce8014\") " pod="openshift-ingress/e2e-inf-a-04d111-openshift-default-cd549dc77-f9kwz" Oct 31 04:55:15.284360 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:55:15.281762 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96a130da-eaed-4765-905b-261822ce8014-istiod-ca-cert\") pod \"e2e-inf-a-04d111-openshift-default-cd549dc77-f9kwz\" (UID: \"96a130da-eaed-4765-905b-261822ce8014\") " pod="openshift-ingress/e2e-inf-a-04d111-openshift-default-cd549dc77-f9kwz" Oct 31 04:55:15.313310 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:55:15.302555 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96a130da-eaed-4765-905b-261822ce8014-istio-envoy\") pod \"e2e-inf-a-04d111-openshift-default-cd549dc77-f9kwz\" (UID: \"96a130da-eaed-4765-905b-261822ce8014\") " pod="openshift-ingress/e2e-inf-a-04d111-openshift-default-cd549dc77-f9kwz" Oct 31 04:55:15.313310 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:55:15.303081 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96a130da-eaed-4765-905b-261822ce8014-istio-podinfo\") pod \"e2e-inf-a-04d111-openshift-default-cd549dc77-f9kwz\" (UID: \"96a130da-eaed-4765-905b-261822ce8014\") " pod="openshift-ingress/e2e-inf-a-04d111-openshift-default-cd549dc77-f9kwz" Oct 31 04:55:15.350353 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:55:15.342296 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4xsf\" (UniqueName: \"kubernetes.io/projected/96a130da-eaed-4765-905b-261822ce8014-kube-api-access-d4xsf\") pod \"e2e-inf-a-04d111-openshift-default-cd549dc77-f9kwz\" (UID: \"96a130da-eaed-4765-905b-261822ce8014\") " pod="openshift-ingress/e2e-inf-a-04d111-openshift-default-cd549dc77-f9kwz" Oct 31 04:55:15.359351 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:55:15.355951 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96a130da-eaed-4765-905b-261822ce8014-istio-token\") pod \"e2e-inf-a-04d111-openshift-default-cd549dc77-f9kwz\" (UID: \"96a130da-eaed-4765-905b-261822ce8014\") " pod="openshift-ingress/e2e-inf-a-04d111-openshift-default-cd549dc77-f9kwz" Oct 31 04:55:15.373211 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:55:15.371880 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-04d111-openshift-default-cd549dc77-f9kwz" Oct 31 04:55:15.725425 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:55:15.725394 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96a130da_eaed_4765_905b_261822ce8014.slice/crio-292a75c68458e8bffd4786e8afa7a806e37bca7c9de146ce8335e0b9a4e877a0 WatchSource:0}: Error finding container 292a75c68458e8bffd4786e8afa7a806e37bca7c9de146ce8335e0b9a4e877a0: Status 404 returned error can't find the container with id 292a75c68458e8bffd4786e8afa7a806e37bca7c9de146ce8335e0b9a4e877a0 Oct 31 04:55:15.727635 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:55:15.727614 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-04d111-openshift-default-cd549dc77-f9kwz"] Oct 31 04:55:15.728396 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:55:15.728364 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:55:15.728458 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:55:15.728427 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:55:15.728495 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:55:15.728480 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:55:15.961184 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:55:15.961154 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-04d111-openshift-default-cd549dc77-f9kwz" event={"ID":"96a130da-eaed-4765-905b-261822ce8014","Type":"ContainerStarted","Data":"1ff8d4c81bcb160a5fba84a6af6d911265299a4bfa28ad219feae5154305902f"} Oct 31 04:55:15.961377 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:55:15.961202 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-04d111-openshift-default-cd549dc77-f9kwz" event={"ID":"96a130da-eaed-4765-905b-261822ce8014","Type":"ContainerStarted","Data":"292a75c68458e8bffd4786e8afa7a806e37bca7c9de146ce8335e0b9a4e877a0"} Oct 31 04:55:16.011532 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:55:16.011448 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-04d111-openshift-default-cd549dc77-f9kwz" podStartSLOduration=1.011434551 podStartE2EDuration="1.011434551s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:55:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:55:16.006120256 +0000 UTC m=+3063.720454891" watchObservedRunningTime="2026-10-31 04:55:16.011434551 +0000 UTC m=+3063.725769248" Oct 31 04:55:16.372936 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:55:16.372904 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-04d111-openshift-default-cd549dc77-f9kwz" Oct 31 04:55:16.379074 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:55:16.379049 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-04d111-openshift-default-cd549dc77-f9kwz" Oct 31 04:55:16.968067 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:55:16.968037 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-04d111-openshift-default-cd549dc77-f9kwz" Oct 31 04:55:16.968926 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:55:16.968908 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-04d111-openshift-default-cd549dc77-f9kwz" Oct 31 04:55:24.416966 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:55:24.416933 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-99pz4"] Oct 31 04:55:24.423257 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:55:24.423239 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-99pz4" Oct 31 04:55:24.441720 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:55:24.441688 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-a04e4e-openshift-default-dockercfg-22x4n\"" Oct 31 04:55:24.467181 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:55:24.466650 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-99pz4"] Oct 31 04:55:24.501069 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:55:24.501040 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fa45335-eb01-48ef-bdd2-2a32785614d5-workload-certs\") pod \"e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-99pz4\" (UID: \"2fa45335-eb01-48ef-bdd2-2a32785614d5\") " pod="openshift-ingress/e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-99pz4" Oct 31 04:55:24.501227 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:55:24.501084 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fa45335-eb01-48ef-bdd2-2a32785614d5-istiod-ca-cert\") pod \"e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-99pz4\" (UID: \"2fa45335-eb01-48ef-bdd2-2a32785614d5\") " pod="openshift-ingress/e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-99pz4" Oct 31 04:55:24.501227 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:55:24.501210 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fa45335-eb01-48ef-bdd2-2a32785614d5-credential-socket\") pod \"e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-99pz4\" (UID: \"2fa45335-eb01-48ef-bdd2-2a32785614d5\") " pod="openshift-ingress/e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-99pz4" Oct 31 04:55:24.501367 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:55:24.501246 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fa45335-eb01-48ef-bdd2-2a32785614d5-istio-data\") pod \"e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-99pz4\" (UID: \"2fa45335-eb01-48ef-bdd2-2a32785614d5\") " pod="openshift-ingress/e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-99pz4" Oct 31 04:55:24.501367 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:55:24.501291 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fa45335-eb01-48ef-bdd2-2a32785614d5-istio-podinfo\") pod \"e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-99pz4\" (UID: \"2fa45335-eb01-48ef-bdd2-2a32785614d5\") " pod="openshift-ingress/e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-99pz4" Oct 31 04:55:24.501468 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:55:24.501433 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nc2r\" (UniqueName: \"kubernetes.io/projected/2fa45335-eb01-48ef-bdd2-2a32785614d5-kube-api-access-5nc2r\") pod \"e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-99pz4\" (UID: \"2fa45335-eb01-48ef-bdd2-2a32785614d5\") " pod="openshift-ingress/e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-99pz4" Oct 31 04:55:24.501518 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:55:24.501475 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fa45335-eb01-48ef-bdd2-2a32785614d5-istio-envoy\") pod \"e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-99pz4\" (UID: \"2fa45335-eb01-48ef-bdd2-2a32785614d5\") " pod="openshift-ingress/e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-99pz4" Oct 31 04:55:24.501661 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:55:24.501640 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2fa45335-eb01-48ef-bdd2-2a32785614d5-istio-token\") pod \"e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-99pz4\" (UID: \"2fa45335-eb01-48ef-bdd2-2a32785614d5\") " pod="openshift-ingress/e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-99pz4" Oct 31 04:55:24.501703 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:55:24.501683 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fa45335-eb01-48ef-bdd2-2a32785614d5-workload-socket\") pod \"e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-99pz4\" (UID: \"2fa45335-eb01-48ef-bdd2-2a32785614d5\") " pod="openshift-ingress/e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-99pz4" Oct 31 04:55:24.603143 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:55:24.603110 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nc2r\" (UniqueName: \"kubernetes.io/projected/2fa45335-eb01-48ef-bdd2-2a32785614d5-kube-api-access-5nc2r\") pod \"e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-99pz4\" (UID: \"2fa45335-eb01-48ef-bdd2-2a32785614d5\") " pod="openshift-ingress/e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-99pz4" Oct 31 04:55:24.603364 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:55:24.603149 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fa45335-eb01-48ef-bdd2-2a32785614d5-istio-envoy\") pod \"e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-99pz4\" (UID: \"2fa45335-eb01-48ef-bdd2-2a32785614d5\") " pod="openshift-ingress/e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-99pz4" Oct 31 04:55:24.603364 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:55:24.603220 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2fa45335-eb01-48ef-bdd2-2a32785614d5-istio-token\") pod \"e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-99pz4\" (UID: \"2fa45335-eb01-48ef-bdd2-2a32785614d5\") " pod="openshift-ingress/e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-99pz4" Oct 31 04:55:24.603364 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:55:24.603240 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fa45335-eb01-48ef-bdd2-2a32785614d5-workload-socket\") pod \"e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-99pz4\" (UID: \"2fa45335-eb01-48ef-bdd2-2a32785614d5\") " pod="openshift-ingress/e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-99pz4" Oct 31 04:55:24.603586 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:55:24.603389 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fa45335-eb01-48ef-bdd2-2a32785614d5-workload-certs\") pod \"e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-99pz4\" (UID: \"2fa45335-eb01-48ef-bdd2-2a32785614d5\") " pod="openshift-ingress/e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-99pz4" Oct 31 04:55:24.603586 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:55:24.603427 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fa45335-eb01-48ef-bdd2-2a32785614d5-istiod-ca-cert\") pod \"e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-99pz4\" (UID: \"2fa45335-eb01-48ef-bdd2-2a32785614d5\") " pod="openshift-ingress/e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-99pz4" Oct 31 04:55:24.603586 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:55:24.603485 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fa45335-eb01-48ef-bdd2-2a32785614d5-credential-socket\") pod \"e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-99pz4\" (UID: \"2fa45335-eb01-48ef-bdd2-2a32785614d5\") " pod="openshift-ingress/e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-99pz4" Oct 31 04:55:24.603586 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:55:24.603522 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fa45335-eb01-48ef-bdd2-2a32785614d5-istio-data\") pod \"e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-99pz4\" (UID: \"2fa45335-eb01-48ef-bdd2-2a32785614d5\") " pod="openshift-ingress/e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-99pz4" Oct 31 04:55:24.603813 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:55:24.603618 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fa45335-eb01-48ef-bdd2-2a32785614d5-istio-podinfo\") pod \"e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-99pz4\" (UID: \"2fa45335-eb01-48ef-bdd2-2a32785614d5\") " pod="openshift-ingress/e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-99pz4" Oct 31 04:55:24.603813 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:55:24.603750 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fa45335-eb01-48ef-bdd2-2a32785614d5-workload-socket\") pod \"e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-99pz4\" (UID: \"2fa45335-eb01-48ef-bdd2-2a32785614d5\") " pod="openshift-ingress/e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-99pz4" Oct 31 04:55:24.603813 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:55:24.603776 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fa45335-eb01-48ef-bdd2-2a32785614d5-workload-certs\") pod \"e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-99pz4\" (UID: \"2fa45335-eb01-48ef-bdd2-2a32785614d5\") " pod="openshift-ingress/e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-99pz4" Oct 31 04:55:24.603813 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:55:24.603799 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fa45335-eb01-48ef-bdd2-2a32785614d5-istio-data\") pod \"e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-99pz4\" (UID: \"2fa45335-eb01-48ef-bdd2-2a32785614d5\") " pod="openshift-ingress/e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-99pz4" Oct 31 04:55:24.604114 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:55:24.604088 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fa45335-eb01-48ef-bdd2-2a32785614d5-credential-socket\") pod \"e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-99pz4\" (UID: \"2fa45335-eb01-48ef-bdd2-2a32785614d5\") " pod="openshift-ingress/e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-99pz4" Oct 31 04:55:24.604218 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:55:24.604197 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fa45335-eb01-48ef-bdd2-2a32785614d5-istiod-ca-cert\") pod \"e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-99pz4\" (UID: \"2fa45335-eb01-48ef-bdd2-2a32785614d5\") " pod="openshift-ingress/e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-99pz4" Oct 31 04:55:24.605494 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:55:24.605469 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fa45335-eb01-48ef-bdd2-2a32785614d5-istio-envoy\") pod \"e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-99pz4\" (UID: \"2fa45335-eb01-48ef-bdd2-2a32785614d5\") " pod="openshift-ingress/e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-99pz4" Oct 31 04:55:24.605933 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:55:24.605917 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fa45335-eb01-48ef-bdd2-2a32785614d5-istio-podinfo\") pod \"e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-99pz4\" (UID: \"2fa45335-eb01-48ef-bdd2-2a32785614d5\") " pod="openshift-ingress/e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-99pz4" Oct 31 04:55:24.627975 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:55:24.627948 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nc2r\" (UniqueName: \"kubernetes.io/projected/2fa45335-eb01-48ef-bdd2-2a32785614d5-kube-api-access-5nc2r\") pod \"e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-99pz4\" (UID: \"2fa45335-eb01-48ef-bdd2-2a32785614d5\") " pod="openshift-ingress/e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-99pz4" Oct 31 04:55:24.628070 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:55:24.627949 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2fa45335-eb01-48ef-bdd2-2a32785614d5-istio-token\") pod \"e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-99pz4\" (UID: \"2fa45335-eb01-48ef-bdd2-2a32785614d5\") " pod="openshift-ingress/e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-99pz4" Oct 31 04:55:24.739383 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:55:24.739276 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-99pz4" Oct 31 04:55:25.056387 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:55:25.056356 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fa45335_eb01_48ef_bdd2_2a32785614d5.slice/crio-6b947bb817e26268dcd3dca981a709cb7189f4e7d63d1e1cac55eeca6794e802 WatchSource:0}: Error finding container 6b947bb817e26268dcd3dca981a709cb7189f4e7d63d1e1cac55eeca6794e802: Status 404 returned error can't find the container with id 6b947bb817e26268dcd3dca981a709cb7189f4e7d63d1e1cac55eeca6794e802 Oct 31 04:55:25.058429 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:55:25.058398 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-99pz4"] Oct 31 04:55:25.059504 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:55:25.059471 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:55:25.059593 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:55:25.059536 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:55:25.059593 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:55:25.059568 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:55:26.032621 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:55:26.032584 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-99pz4" event={"ID":"2fa45335-eb01-48ef-bdd2-2a32785614d5","Type":"ContainerStarted","Data":"85be22f43c677d624717f8f64f0de22274b17c67b6e946db24ce0084c6a66446"} Oct 31 04:55:26.033029 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:55:26.032640 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-99pz4" event={"ID":"2fa45335-eb01-48ef-bdd2-2a32785614d5","Type":"ContainerStarted","Data":"6b947bb817e26268dcd3dca981a709cb7189f4e7d63d1e1cac55eeca6794e802"} Oct 31 04:55:26.105207 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:55:26.105144 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-99pz4" podStartSLOduration=2.105125536 podStartE2EDuration="2.105125536s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:55:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:55:26.091921458 +0000 UTC m=+3073.806256098" watchObservedRunningTime="2026-10-31 04:55:26.105125536 +0000 UTC m=+3073.819460156" Oct 31 04:55:26.756353 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:55:26.748610 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-99pz4" Oct 31 04:55:26.756556 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:55:26.756425 4376 patch_prober.go:29] interesting pod/e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-99pz4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused" start-of-body= Oct 31 04:55:26.756556 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:55:26.756484 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-99pz4" podUID="2fa45335-eb01-48ef-bdd2-2a32785614d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused" Oct 31 04:55:27.760350 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:55:27.759703 4376 patch_prober.go:29] interesting pod/e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-99pz4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused" start-of-body= Oct 31 04:55:27.760350 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:55:27.759757 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-99pz4" podUID="2fa45335-eb01-48ef-bdd2-2a32785614d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused" Oct 31 04:55:28.754947 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:55:28.754909 4376 patch_prober.go:29] interesting pod/e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-99pz4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused" start-of-body= Oct 31 04:55:28.755231 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:55:28.755200 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-99pz4" podUID="2fa45335-eb01-48ef-bdd2-2a32785614d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused" Oct 31 04:55:29.742353 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:55:29.741640 4376 patch_prober.go:29] interesting pod/e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-99pz4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused" start-of-body= Oct 31 04:55:29.742353 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:55:29.741730 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-99pz4" podUID="2fa45335-eb01-48ef-bdd2-2a32785614d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused" Oct 31 04:55:30.756353 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:55:30.744539 4376 patch_prober.go:29] interesting pod/e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-99pz4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused" start-of-body= Oct 31 04:55:30.756353 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:55:30.744600 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-99pz4" podUID="2fa45335-eb01-48ef-bdd2-2a32785614d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused" Oct 31 04:55:31.745920 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:55:31.745891 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-99pz4" Oct 31 04:55:31.746381 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:55:31.746362 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-99pz4" Oct 31 04:55:31.746940 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:55:31.746922 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-99pz4" Oct 31 04:55:41.001822 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:55:41.001790 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 04:55:49.148958 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:55:49.111864 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 04:56:02.740454 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:56:02.740417 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-fb346b-openshift-default-744c56d4f9-mhkhz"] Oct 31 04:56:02.745200 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:56:02.745177 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-fb346b-openshift-default-744c56d4f9-mhkhz" Oct 31 04:56:02.760786 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:56:02.760759 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-fb346b-openshift-default-dockercfg-twtvp\"" Oct 31 04:56:02.779069 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:56:02.779038 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-fb346b-openshift-default-744c56d4f9-mhkhz"] Oct 31 04:56:02.873018 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:56:02.872960 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6c79800-45c4-4778-91ef-027ceba7bb8a-credential-socket\") pod \"e2e-inf-b-fb346b-openshift-default-744c56d4f9-mhkhz\" (UID: \"a6c79800-45c4-4778-91ef-027ceba7bb8a\") " pod="openshift-ingress/e2e-inf-b-fb346b-openshift-default-744c56d4f9-mhkhz" Oct 31 04:56:02.905456 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:56:02.905414 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6c79800-45c4-4778-91ef-027ceba7bb8a-istio-data\") pod \"e2e-inf-b-fb346b-openshift-default-744c56d4f9-mhkhz\" (UID: \"a6c79800-45c4-4778-91ef-027ceba7bb8a\") " pod="openshift-ingress/e2e-inf-b-fb346b-openshift-default-744c56d4f9-mhkhz" Oct 31 04:56:02.905656 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:56:02.905471 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6c79800-45c4-4778-91ef-027ceba7bb8a-istio-envoy\") pod \"e2e-inf-b-fb346b-openshift-default-744c56d4f9-mhkhz\" (UID: \"a6c79800-45c4-4778-91ef-027ceba7bb8a\") " pod="openshift-ingress/e2e-inf-b-fb346b-openshift-default-744c56d4f9-mhkhz" Oct 31 04:56:02.905656 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:56:02.905565 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6c79800-45c4-4778-91ef-027ceba7bb8a-istio-token\") pod \"e2e-inf-b-fb346b-openshift-default-744c56d4f9-mhkhz\" (UID: \"a6c79800-45c4-4778-91ef-027ceba7bb8a\") " pod="openshift-ingress/e2e-inf-b-fb346b-openshift-default-744c56d4f9-mhkhz" Oct 31 04:56:02.905656 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:56:02.905648 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6c79800-45c4-4778-91ef-027ceba7bb8a-workload-certs\") pod \"e2e-inf-b-fb346b-openshift-default-744c56d4f9-mhkhz\" (UID: \"a6c79800-45c4-4778-91ef-027ceba7bb8a\") " pod="openshift-ingress/e2e-inf-b-fb346b-openshift-default-744c56d4f9-mhkhz" Oct 31 04:56:02.905848 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:56:02.905742 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvbtj\" (UniqueName: \"kubernetes.io/projected/a6c79800-45c4-4778-91ef-027ceba7bb8a-kube-api-access-vvbtj\") pod \"e2e-inf-b-fb346b-openshift-default-744c56d4f9-mhkhz\" (UID: \"a6c79800-45c4-4778-91ef-027ceba7bb8a\") " pod="openshift-ingress/e2e-inf-b-fb346b-openshift-default-744c56d4f9-mhkhz" Oct 31 04:56:02.905950 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:56:02.905931 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6c79800-45c4-4778-91ef-027ceba7bb8a-istiod-ca-cert\") pod \"e2e-inf-b-fb346b-openshift-default-744c56d4f9-mhkhz\" (UID: \"a6c79800-45c4-4778-91ef-027ceba7bb8a\") " pod="openshift-ingress/e2e-inf-b-fb346b-openshift-default-744c56d4f9-mhkhz" Oct 31 04:56:02.906085 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:56:02.906069 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6c79800-45c4-4778-91ef-027ceba7bb8a-workload-socket\") pod \"e2e-inf-b-fb346b-openshift-default-744c56d4f9-mhkhz\" (UID: \"a6c79800-45c4-4778-91ef-027ceba7bb8a\") " pod="openshift-ingress/e2e-inf-b-fb346b-openshift-default-744c56d4f9-mhkhz" Oct 31 04:56:02.906152 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:56:02.906105 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6c79800-45c4-4778-91ef-027ceba7bb8a-istio-podinfo\") pod \"e2e-inf-b-fb346b-openshift-default-744c56d4f9-mhkhz\" (UID: \"a6c79800-45c4-4778-91ef-027ceba7bb8a\") " pod="openshift-ingress/e2e-inf-b-fb346b-openshift-default-744c56d4f9-mhkhz" Oct 31 04:56:03.017403 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:56:03.017173 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6c79800-45c4-4778-91ef-027ceba7bb8a-istiod-ca-cert\") pod \"e2e-inf-b-fb346b-openshift-default-744c56d4f9-mhkhz\" (UID: \"a6c79800-45c4-4778-91ef-027ceba7bb8a\") " pod="openshift-ingress/e2e-inf-b-fb346b-openshift-default-744c56d4f9-mhkhz" Oct 31 04:56:03.017403 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:56:03.017301 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6c79800-45c4-4778-91ef-027ceba7bb8a-workload-socket\") pod \"e2e-inf-b-fb346b-openshift-default-744c56d4f9-mhkhz\" (UID: \"a6c79800-45c4-4778-91ef-027ceba7bb8a\") " pod="openshift-ingress/e2e-inf-b-fb346b-openshift-default-744c56d4f9-mhkhz" Oct 31 04:56:03.017403 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:56:03.017349 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6c79800-45c4-4778-91ef-027ceba7bb8a-istio-podinfo\") pod \"e2e-inf-b-fb346b-openshift-default-744c56d4f9-mhkhz\" (UID: \"a6c79800-45c4-4778-91ef-027ceba7bb8a\") " pod="openshift-ingress/e2e-inf-b-fb346b-openshift-default-744c56d4f9-mhkhz" Oct 31 04:56:03.017713 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:56:03.017404 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6c79800-45c4-4778-91ef-027ceba7bb8a-credential-socket\") pod \"e2e-inf-b-fb346b-openshift-default-744c56d4f9-mhkhz\" (UID: \"a6c79800-45c4-4778-91ef-027ceba7bb8a\") " pod="openshift-ingress/e2e-inf-b-fb346b-openshift-default-744c56d4f9-mhkhz" Oct 31 04:56:03.017713 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:56:03.017429 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6c79800-45c4-4778-91ef-027ceba7bb8a-istio-data\") pod \"e2e-inf-b-fb346b-openshift-default-744c56d4f9-mhkhz\" (UID: \"a6c79800-45c4-4778-91ef-027ceba7bb8a\") " pod="openshift-ingress/e2e-inf-b-fb346b-openshift-default-744c56d4f9-mhkhz" Oct 31 04:56:03.017713 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:56:03.017454 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6c79800-45c4-4778-91ef-027ceba7bb8a-istio-envoy\") pod \"e2e-inf-b-fb346b-openshift-default-744c56d4f9-mhkhz\" (UID: \"a6c79800-45c4-4778-91ef-027ceba7bb8a\") " pod="openshift-ingress/e2e-inf-b-fb346b-openshift-default-744c56d4f9-mhkhz" Oct 31 04:56:03.017713 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:56:03.017597 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6c79800-45c4-4778-91ef-027ceba7bb8a-istio-token\") pod \"e2e-inf-b-fb346b-openshift-default-744c56d4f9-mhkhz\" (UID: \"a6c79800-45c4-4778-91ef-027ceba7bb8a\") " pod="openshift-ingress/e2e-inf-b-fb346b-openshift-default-744c56d4f9-mhkhz" Oct 31 04:56:03.017713 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:56:03.017685 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6c79800-45c4-4778-91ef-027ceba7bb8a-workload-certs\") pod \"e2e-inf-b-fb346b-openshift-default-744c56d4f9-mhkhz\" (UID: \"a6c79800-45c4-4778-91ef-027ceba7bb8a\") " pod="openshift-ingress/e2e-inf-b-fb346b-openshift-default-744c56d4f9-mhkhz" Oct 31 04:56:03.017974 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:56:03.017772 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvbtj\" (UniqueName: \"kubernetes.io/projected/a6c79800-45c4-4778-91ef-027ceba7bb8a-kube-api-access-vvbtj\") pod \"e2e-inf-b-fb346b-openshift-default-744c56d4f9-mhkhz\" (UID: \"a6c79800-45c4-4778-91ef-027ceba7bb8a\") " pod="openshift-ingress/e2e-inf-b-fb346b-openshift-default-744c56d4f9-mhkhz" Oct 31 04:56:03.043356 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:56:03.019264 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6c79800-45c4-4778-91ef-027ceba7bb8a-istiod-ca-cert\") pod \"e2e-inf-b-fb346b-openshift-default-744c56d4f9-mhkhz\" (UID: \"a6c79800-45c4-4778-91ef-027ceba7bb8a\") " pod="openshift-ingress/e2e-inf-b-fb346b-openshift-default-744c56d4f9-mhkhz" Oct 31 04:56:03.043356 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:56:03.020591 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6c79800-45c4-4778-91ef-027ceba7bb8a-workload-certs\") pod \"e2e-inf-b-fb346b-openshift-default-744c56d4f9-mhkhz\" (UID: \"a6c79800-45c4-4778-91ef-027ceba7bb8a\") " pod="openshift-ingress/e2e-inf-b-fb346b-openshift-default-744c56d4f9-mhkhz" Oct 31 04:56:03.043356 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:56:03.021130 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6c79800-45c4-4778-91ef-027ceba7bb8a-credential-socket\") pod \"e2e-inf-b-fb346b-openshift-default-744c56d4f9-mhkhz\" (UID: \"a6c79800-45c4-4778-91ef-027ceba7bb8a\") " pod="openshift-ingress/e2e-inf-b-fb346b-openshift-default-744c56d4f9-mhkhz" Oct 31 04:56:03.043356 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:56:03.021401 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6c79800-45c4-4778-91ef-027ceba7bb8a-istio-data\") pod \"e2e-inf-b-fb346b-openshift-default-744c56d4f9-mhkhz\" (UID: \"a6c79800-45c4-4778-91ef-027ceba7bb8a\") " pod="openshift-ingress/e2e-inf-b-fb346b-openshift-default-744c56d4f9-mhkhz" Oct 31 04:56:03.043356 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:56:03.023601 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6c79800-45c4-4778-91ef-027ceba7bb8a-istio-podinfo\") pod \"e2e-inf-b-fb346b-openshift-default-744c56d4f9-mhkhz\" (UID: \"a6c79800-45c4-4778-91ef-027ceba7bb8a\") " pod="openshift-ingress/e2e-inf-b-fb346b-openshift-default-744c56d4f9-mhkhz" Oct 31 04:56:03.043356 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:56:03.035577 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6c79800-45c4-4778-91ef-027ceba7bb8a-workload-socket\") pod \"e2e-inf-b-fb346b-openshift-default-744c56d4f9-mhkhz\" (UID: \"a6c79800-45c4-4778-91ef-027ceba7bb8a\") " pod="openshift-ingress/e2e-inf-b-fb346b-openshift-default-744c56d4f9-mhkhz" Oct 31 04:56:03.057682 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:56:03.057648 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6c79800-45c4-4778-91ef-027ceba7bb8a-istio-token\") pod \"e2e-inf-b-fb346b-openshift-default-744c56d4f9-mhkhz\" (UID: \"a6c79800-45c4-4778-91ef-027ceba7bb8a\") " pod="openshift-ingress/e2e-inf-b-fb346b-openshift-default-744c56d4f9-mhkhz" Oct 31 04:56:03.067112 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:56:03.067078 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvbtj\" (UniqueName: \"kubernetes.io/projected/a6c79800-45c4-4778-91ef-027ceba7bb8a-kube-api-access-vvbtj\") pod \"e2e-inf-b-fb346b-openshift-default-744c56d4f9-mhkhz\" (UID: \"a6c79800-45c4-4778-91ef-027ceba7bb8a\") " pod="openshift-ingress/e2e-inf-b-fb346b-openshift-default-744c56d4f9-mhkhz" Oct 31 04:56:03.068088 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:56:03.068038 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6c79800-45c4-4778-91ef-027ceba7bb8a-istio-envoy\") pod \"e2e-inf-b-fb346b-openshift-default-744c56d4f9-mhkhz\" (UID: \"a6c79800-45c4-4778-91ef-027ceba7bb8a\") " pod="openshift-ingress/e2e-inf-b-fb346b-openshift-default-744c56d4f9-mhkhz" Oct 31 04:56:03.363362 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:56:03.363242 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-fb346b-openshift-default-744c56d4f9-mhkhz" Oct 31 04:56:03.685794 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:56:03.685758 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6c79800_45c4_4778_91ef_027ceba7bb8a.slice/crio-e611feb531d0f329989673e2313b19071545068c4a39ee057fcee8e299584896 WatchSource:0}: Error finding container e611feb531d0f329989673e2313b19071545068c4a39ee057fcee8e299584896: Status 404 returned error can't find the container with id e611feb531d0f329989673e2313b19071545068c4a39ee057fcee8e299584896 Oct 31 04:56:03.687989 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:56:03.687968 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-fb346b-openshift-default-744c56d4f9-mhkhz"] Oct 31 04:56:03.688940 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:56:03.688904 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:56:03.689038 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:56:03.688988 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:56:03.689162 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:56:03.689036 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:56:04.489057 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:56:04.489017 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-fb346b-openshift-default-744c56d4f9-mhkhz" event={"ID":"a6c79800-45c4-4778-91ef-027ceba7bb8a","Type":"ContainerStarted","Data":"fb3ba42b1998b8d6c9d673a03e1c679051143e65e9356c84feacc830afa26e8b"} Oct 31 04:56:04.489566 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:56:04.489082 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-fb346b-openshift-default-744c56d4f9-mhkhz" event={"ID":"a6c79800-45c4-4778-91ef-027ceba7bb8a","Type":"ContainerStarted","Data":"e611feb531d0f329989673e2313b19071545068c4a39ee057fcee8e299584896"} Oct 31 04:56:04.548629 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:56:04.548579 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-fb346b-openshift-default-744c56d4f9-mhkhz" podStartSLOduration=2.548565868 podStartE2EDuration="2.548565868s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:56:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:56:04.547691755 +0000 UTC m=+3112.262026390" watchObservedRunningTime="2026-10-31 04:56:04.548565868 +0000 UTC m=+3112.262900484" Oct 31 04:56:05.363385 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:56:05.363324 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-fb346b-openshift-default-744c56d4f9-mhkhz" Oct 31 04:56:05.370008 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:56:05.369980 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-fb346b-openshift-default-744c56d4f9-mhkhz" Oct 31 04:56:05.495846 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:56:05.495820 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-fb346b-openshift-default-744c56d4f9-mhkhz" Oct 31 04:56:05.496800 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:56:05.496780 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-fb346b-openshift-default-744c56d4f9-mhkhz" Oct 31 04:56:11.001862 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:56:11.001827 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 04:56:41.002439 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:56:41.002403 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 04:56:53.148149 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:56:53.148110 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 04:57:03.009490 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:57:03.009438 4376 patch_prober.go:29] interesting pod/e2e-inf-a-04d111-openshift-default-cd549dc77-f9kwz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.215:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 04:57:03.009980 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:57:03.009503 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-04d111-openshift-default-cd549dc77-f9kwz" podUID="96a130da-eaed-4765-905b-261822ce8014" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.215:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 04:57:06.092521 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:57:06.092488 4376 generic.go:356] "Generic (PLEG): container finished" podID="96a130da-eaed-4765-905b-261822ce8014" containerID="1ff8d4c81bcb160a5fba84a6af6d911265299a4bfa28ad219feae5154305902f" exitCode=0 Oct 31 04:57:06.092966 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:57:06.092571 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-04d111-openshift-default-cd549dc77-f9kwz" event={"ID":"96a130da-eaed-4765-905b-261822ce8014","Type":"ContainerDied","Data":"1ff8d4c81bcb160a5fba84a6af6d911265299a4bfa28ad219feae5154305902f"} Oct 31 04:57:06.093028 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:57:06.093008 4376 scope.go:122] "RemoveContainer" containerID="1ff8d4c81bcb160a5fba84a6af6d911265299a4bfa28ad219feae5154305902f" Oct 31 04:57:06.094918 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:57:06.094881 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:57:06.095052 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:57:06.094974 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:57:06.095052 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:57:06.095023 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:57:07.102077 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:57:07.102039 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-04d111-openshift-default-cd549dc77-f9kwz" event={"ID":"96a130da-eaed-4765-905b-261822ce8014","Type":"ContainerStarted","Data":"7db18a134621d04c502edc286ab1e934c6242b6c8b8da9b42df1f2762948decf"} Oct 31 04:57:07.373390 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:57:07.373322 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-04d111-openshift-default-cd549dc77-f9kwz" Oct 31 04:57:07.373570 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:57:07.373544 4376 patch_prober.go:29] interesting pod/e2e-inf-a-04d111-openshift-default-cd549dc77-f9kwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused" start-of-body= Oct 31 04:57:07.373616 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:57:07.373597 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-04d111-openshift-default-cd549dc77-f9kwz" podUID="96a130da-eaed-4765-905b-261822ce8014" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused" Oct 31 04:57:08.373068 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:57:08.373033 4376 patch_prober.go:29] interesting pod/e2e-inf-a-04d111-openshift-default-cd549dc77-f9kwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused" start-of-body= Oct 31 04:57:08.373476 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:57:08.373086 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-04d111-openshift-default-cd549dc77-f9kwz" podUID="96a130da-eaed-4765-905b-261822ce8014" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused" Oct 31 04:57:09.373493 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:57:09.373459 4376 patch_prober.go:29] interesting pod/e2e-inf-a-04d111-openshift-default-cd549dc77-f9kwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused" start-of-body= Oct 31 04:57:09.373888 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:57:09.373510 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-04d111-openshift-default-cd549dc77-f9kwz" podUID="96a130da-eaed-4765-905b-261822ce8014" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused" Oct 31 04:57:10.373730 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:57:10.373680 4376 patch_prober.go:29] interesting pod/e2e-inf-a-04d111-openshift-default-cd549dc77-f9kwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused" start-of-body= Oct 31 04:57:10.374178 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:57:10.373764 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-04d111-openshift-default-cd549dc77-f9kwz" podUID="96a130da-eaed-4765-905b-261822ce8014" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused" Oct 31 04:57:11.005660 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:57:11.005625 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 04:57:11.372807 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:57:11.372767 4376 patch_prober.go:29] interesting pod/e2e-inf-a-04d111-openshift-default-cd549dc77-f9kwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused" start-of-body= Oct 31 04:57:11.373017 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:57:11.372842 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-04d111-openshift-default-cd549dc77-f9kwz" podUID="96a130da-eaed-4765-905b-261822ce8014" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused" Oct 31 04:57:12.125217 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:57:12.125177 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-47e9af-74ddc46b6c-dbghm"] Oct 31 04:57:12.130147 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:57:12.130125 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-47e9af-74ddc46b6c-dbghm" Oct 31 04:57:12.167651 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:57:12.167626 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-47e9af-dockercfg-xbpsf\"" Oct 31 04:57:12.173570 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:57:12.173528 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-47e9af-74ddc46b6c-dbghm"] Oct 31 04:57:12.206633 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:57:12.198755 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-47e9af\"" Oct 31 04:57:12.352353 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:57:12.327212 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mh4xq\" (UniqueName: \"kubernetes.io/projected/e1abe8c9-b4dc-4380-8495-a0ce56fb2734-kube-api-access-mh4xq\") pod \"payload-pre-processing-e2e-inf-b-47e9af-74ddc46b6c-dbghm\" (UID: \"e1abe8c9-b4dc-4380-8495-a0ce56fb2734\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-47e9af-74ddc46b6c-dbghm" Oct 31 04:57:12.398407 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:57:12.378308 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1abe8c9-b4dc-4380-8495-a0ce56fb2734-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-47e9af-74ddc46b6c-dbghm\" (UID: \"e1abe8c9-b4dc-4380-8495-a0ce56fb2734\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-47e9af-74ddc46b6c-dbghm" Oct 31 04:57:12.412646 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:57:12.375744 4376 patch_prober.go:29] interesting pod/e2e-inf-a-04d111-openshift-default-cd549dc77-f9kwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused" start-of-body= Oct 31 04:57:12.473549 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:57:12.473508 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-04d111-openshift-default-cd549dc77-f9kwz" podUID="96a130da-eaed-4765-905b-261822ce8014" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused" Oct 31 04:57:12.513925 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:57:12.513890 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1abe8c9-b4dc-4380-8495-a0ce56fb2734-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-47e9af-74ddc46b6c-dbghm\" (UID: \"e1abe8c9-b4dc-4380-8495-a0ce56fb2734\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-47e9af-74ddc46b6c-dbghm" Oct 31 04:57:12.514140 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:57:12.514121 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mh4xq\" (UniqueName: \"kubernetes.io/projected/e1abe8c9-b4dc-4380-8495-a0ce56fb2734-kube-api-access-mh4xq\") pod \"payload-pre-processing-e2e-inf-b-47e9af-74ddc46b6c-dbghm\" (UID: \"e1abe8c9-b4dc-4380-8495-a0ce56fb2734\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-47e9af-74ddc46b6c-dbghm" Oct 31 04:57:12.514758 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:57:12.514735 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1abe8c9-b4dc-4380-8495-a0ce56fb2734-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-47e9af-74ddc46b6c-dbghm\" (UID: \"e1abe8c9-b4dc-4380-8495-a0ce56fb2734\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-47e9af-74ddc46b6c-dbghm" Oct 31 04:57:12.565352 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:57:12.561966 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mh4xq\" (UniqueName: \"kubernetes.io/projected/e1abe8c9-b4dc-4380-8495-a0ce56fb2734-kube-api-access-mh4xq\") pod \"payload-pre-processing-e2e-inf-b-47e9af-74ddc46b6c-dbghm\" (UID: \"e1abe8c9-b4dc-4380-8495-a0ce56fb2734\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-47e9af-74ddc46b6c-dbghm" Oct 31 04:57:12.803792 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:57:12.803706 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-47e9af-dockercfg-xbpsf\"" Oct 31 04:57:12.809687 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:57:12.809662 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-47e9af-74ddc46b6c-dbghm" Oct 31 04:57:13.176410 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:57:13.176380 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1abe8c9_b4dc_4380_8495_a0ce56fb2734.slice/crio-0697e71030fc14dd9a279e7ec2b89a4b319d7d302c6c133bfb5e4ea350e3a536 WatchSource:0}: Error finding container 0697e71030fc14dd9a279e7ec2b89a4b319d7d302c6c133bfb5e4ea350e3a536: Status 404 returned error can't find the container with id 0697e71030fc14dd9a279e7ec2b89a4b319d7d302c6c133bfb5e4ea350e3a536 Oct 31 04:57:13.179127 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:57:13.179100 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-47e9af-74ddc46b6c-dbghm"] Oct 31 04:57:13.377869 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:57:13.377845 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-04d111-openshift-default-cd549dc77-f9kwz" Oct 31 04:57:13.378309 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:57:13.378294 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-04d111-openshift-default-cd549dc77-f9kwz" Oct 31 04:57:13.378923 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:57:13.378907 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-04d111-openshift-default-cd549dc77-f9kwz" Oct 31 04:57:14.155324 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:57:14.155290 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-47e9af-74ddc46b6c-dbghm" event={"ID":"e1abe8c9-b4dc-4380-8495-a0ce56fb2734","Type":"ContainerStarted","Data":"4f7ca525d6dac0927166bce87df58750e7de777702ea37d257a2204892174dd3"} Oct 31 04:57:14.155529 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:57:14.155374 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-47e9af-74ddc46b6c-dbghm" event={"ID":"e1abe8c9-b4dc-4380-8495-a0ce56fb2734","Type":"ContainerStarted","Data":"0697e71030fc14dd9a279e7ec2b89a4b319d7d302c6c133bfb5e4ea350e3a536"} Oct 31 04:57:14.155529 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:57:14.155398 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-47e9af-74ddc46b6c-dbghm" Oct 31 04:57:14.197206 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:57:14.197162 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-47e9af-74ddc46b6c-dbghm" podStartSLOduration=2.197149411 podStartE2EDuration="2.197149411s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:57:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:57:14.19608583 +0000 UTC m=+3181.910420468" watchObservedRunningTime="2026-10-31 04:57:14.197149411 +0000 UTC m=+3181.911484028" Oct 31 04:57:25.165478 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:57:25.165396 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-47e9af-74ddc46b6c-dbghm" Oct 31 04:57:28.128690 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:57:28.128654 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c1fe26-7c97f95bc7-knlhj"] Oct 31 04:57:28.133747 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:57:28.133726 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-c1fe26-7c97f95bc7-knlhj" Oct 31 04:57:28.156115 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:57:28.156086 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c1fe26-7c97f95bc7-knlhj"] Oct 31 04:57:28.163047 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:57:28.163021 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-c1fe26-dockercfg-wszb5\"" Oct 31 04:57:28.163200 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:57:28.163057 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-c1fe26\"" Oct 31 04:57:28.180553 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:57:28.180526 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c1fe26-7c889f576c-mptbr"] Oct 31 04:57:28.185438 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:57:28.185418 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c1fe26-7c889f576c-mptbr" Oct 31 04:57:28.193455 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:57:28.193425 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48d10907-ef2a-43b7-b481-746473b85ef6-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-c1fe26-7c889f576c-mptbr\" (UID: \"48d10907-ef2a-43b7-b481-746473b85ef6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c1fe26-7c889f576c-mptbr" Oct 31 04:57:28.193890 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:57:28.193862 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbrvw\" (UniqueName: \"kubernetes.io/projected/48d10907-ef2a-43b7-b481-746473b85ef6-kube-api-access-vbrvw\") pod \"payload-pre-processing-e2e-inf-b-c1fe26-7c889f576c-mptbr\" (UID: \"48d10907-ef2a-43b7-b481-746473b85ef6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c1fe26-7c889f576c-mptbr" Oct 31 04:57:28.194030 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:57:28.194012 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7x7k\" (UniqueName: \"kubernetes.io/projected/7d918c33-5fb2-4f29-a616-c29446e15ff2-kube-api-access-t7x7k\") pod \"payload-processing-e2e-inf-b-c1fe26-7c97f95bc7-knlhj\" (UID: \"7d918c33-5fb2-4f29-a616-c29446e15ff2\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c1fe26-7c97f95bc7-knlhj" Oct 31 04:57:28.194523 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:57:28.194502 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d918c33-5fb2-4f29-a616-c29446e15ff2-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-c1fe26-7c97f95bc7-knlhj\" (UID: \"7d918c33-5fb2-4f29-a616-c29446e15ff2\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c1fe26-7c97f95bc7-knlhj" Oct 31 04:57:28.210692 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:57:28.210665 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c1fe26-7c889f576c-mptbr"] Oct 31 04:57:28.336353 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:57:28.321825 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48d10907-ef2a-43b7-b481-746473b85ef6-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-c1fe26-7c889f576c-mptbr\" (UID: \"48d10907-ef2a-43b7-b481-746473b85ef6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c1fe26-7c889f576c-mptbr" Oct 31 04:57:28.336353 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:57:28.322141 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbrvw\" (UniqueName: \"kubernetes.io/projected/48d10907-ef2a-43b7-b481-746473b85ef6-kube-api-access-vbrvw\") pod \"payload-pre-processing-e2e-inf-b-c1fe26-7c889f576c-mptbr\" (UID: \"48d10907-ef2a-43b7-b481-746473b85ef6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c1fe26-7c889f576c-mptbr" Oct 31 04:57:28.336353 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:57:28.322177 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7x7k\" (UniqueName: \"kubernetes.io/projected/7d918c33-5fb2-4f29-a616-c29446e15ff2-kube-api-access-t7x7k\") pod \"payload-processing-e2e-inf-b-c1fe26-7c97f95bc7-knlhj\" (UID: \"7d918c33-5fb2-4f29-a616-c29446e15ff2\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c1fe26-7c97f95bc7-knlhj" Oct 31 04:57:28.336353 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:57:28.322224 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d918c33-5fb2-4f29-a616-c29446e15ff2-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-c1fe26-7c97f95bc7-knlhj\" (UID: \"7d918c33-5fb2-4f29-a616-c29446e15ff2\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c1fe26-7c97f95bc7-knlhj" Oct 31 04:57:28.336353 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:57:28.323255 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d918c33-5fb2-4f29-a616-c29446e15ff2-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-c1fe26-7c97f95bc7-knlhj\" (UID: \"7d918c33-5fb2-4f29-a616-c29446e15ff2\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c1fe26-7c97f95bc7-knlhj" Oct 31 04:57:28.369168 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:57:28.369134 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48d10907-ef2a-43b7-b481-746473b85ef6-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-c1fe26-7c889f576c-mptbr\" (UID: \"48d10907-ef2a-43b7-b481-746473b85ef6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c1fe26-7c889f576c-mptbr" Oct 31 04:57:28.476563 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:57:28.473271 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7x7k\" (UniqueName: \"kubernetes.io/projected/7d918c33-5fb2-4f29-a616-c29446e15ff2-kube-api-access-t7x7k\") pod \"payload-processing-e2e-inf-b-c1fe26-7c97f95bc7-knlhj\" (UID: \"7d918c33-5fb2-4f29-a616-c29446e15ff2\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c1fe26-7c97f95bc7-knlhj" Oct 31 04:57:28.502126 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:57:28.501216 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbrvw\" (UniqueName: \"kubernetes.io/projected/48d10907-ef2a-43b7-b481-746473b85ef6-kube-api-access-vbrvw\") pod \"payload-pre-processing-e2e-inf-b-c1fe26-7c889f576c-mptbr\" (UID: \"48d10907-ef2a-43b7-b481-746473b85ef6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c1fe26-7c889f576c-mptbr" Oct 31 04:57:28.518621 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:57:28.518586 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c1fe26-7c889f576c-mptbr" Oct 31 04:57:28.754696 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:57:28.754620 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-c1fe26-7c97f95bc7-knlhj" Oct 31 04:57:28.876310 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:57:28.876283 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c1fe26-7c889f576c-mptbr"] Oct 31 04:57:28.879162 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:57:28.879136 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48d10907_ef2a_43b7_b481_746473b85ef6.slice/crio-00699118d4af51c9f82f6b559a963358bb4a61fd80c0c4233c0d1d4e22ad6ba2 WatchSource:0}: Error finding container 00699118d4af51c9f82f6b559a963358bb4a61fd80c0c4233c0d1d4e22ad6ba2: Status 404 returned error can't find the container with id 00699118d4af51c9f82f6b559a963358bb4a61fd80c0c4233c0d1d4e22ad6ba2 Oct 31 04:57:29.059125 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:57:29.059098 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d918c33_5fb2_4f29_a616_c29446e15ff2.slice/crio-75ffc981ec704512f834ec3ab1a5aa289d544de37deb3237fcf90e366a05ca16 WatchSource:0}: Error finding container 75ffc981ec704512f834ec3ab1a5aa289d544de37deb3237fcf90e366a05ca16: Status 404 returned error can't find the container with id 75ffc981ec704512f834ec3ab1a5aa289d544de37deb3237fcf90e366a05ca16 Oct 31 04:57:29.061143 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:57:29.061118 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c1fe26-7c97f95bc7-knlhj"] Oct 31 04:57:29.503110 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:57:29.503079 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-c1fe26-7c97f95bc7-knlhj" event={"ID":"7d918c33-5fb2-4f29-a616-c29446e15ff2","Type":"ContainerStarted","Data":"d57b3926ea3879d4a9753c3e1bbbdaef248a656063ec0fb53012bb2b44d84522"} Oct 31 04:57:29.503563 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:57:29.503130 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-c1fe26-7c97f95bc7-knlhj" event={"ID":"7d918c33-5fb2-4f29-a616-c29446e15ff2","Type":"ContainerStarted","Data":"75ffc981ec704512f834ec3ab1a5aa289d544de37deb3237fcf90e366a05ca16"} Oct 31 04:57:29.503563 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:57:29.503281 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-c1fe26-7c97f95bc7-knlhj" Oct 31 04:57:29.504856 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:57:29.504832 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c1fe26-7c889f576c-mptbr" event={"ID":"48d10907-ef2a-43b7-b481-746473b85ef6","Type":"ContainerStarted","Data":"09b3e5cb1833e3454e5c12041f09e7d184ec2384206f12e976c9a309f8443dad"} Oct 31 04:57:29.504971 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:57:29.504873 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c1fe26-7c889f576c-mptbr" event={"ID":"48d10907-ef2a-43b7-b481-746473b85ef6","Type":"ContainerStarted","Data":"00699118d4af51c9f82f6b559a963358bb4a61fd80c0c4233c0d1d4e22ad6ba2"} Oct 31 04:57:29.504971 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:57:29.504910 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c1fe26-7c889f576c-mptbr" Oct 31 04:57:29.544667 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:57:29.544626 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-c1fe26-7c97f95bc7-knlhj" podStartSLOduration=1.544615071 podStartE2EDuration="1.544615071s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:57:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:57:29.544148999 +0000 UTC m=+3197.258483644" watchObservedRunningTime="2026-10-31 04:57:29.544615071 +0000 UTC m=+3197.258949689" Oct 31 04:57:29.587650 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:57:29.587605 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c1fe26-7c889f576c-mptbr" podStartSLOduration=1.587593974 podStartE2EDuration="1.587593974s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:57:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:57:29.584224088 +0000 UTC m=+3197.298558730" watchObservedRunningTime="2026-10-31 04:57:29.587593974 +0000 UTC m=+3197.301928590" Oct 31 04:57:40.515764 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:57:40.515736 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-c1fe26-7c97f95bc7-knlhj" Oct 31 04:57:40.516275 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:57:40.515848 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c1fe26-7c889f576c-mptbr" Oct 31 04:57:41.015564 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:57:41.015503 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 04:57:44.596353 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:57:44.592520 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 04:57:55.749801 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:57:55.749769 4376 generic.go:356] "Generic (PLEG): container finished" podID="a6c79800-45c4-4778-91ef-027ceba7bb8a" containerID="fb3ba42b1998b8d6c9d673a03e1c679051143e65e9356c84feacc830afa26e8b" exitCode=0 Oct 31 04:57:55.750260 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:57:55.749841 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-fb346b-openshift-default-744c56d4f9-mhkhz" event={"ID":"a6c79800-45c4-4778-91ef-027ceba7bb8a","Type":"ContainerDied","Data":"fb3ba42b1998b8d6c9d673a03e1c679051143e65e9356c84feacc830afa26e8b"} Oct 31 04:57:55.750260 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:57:55.750216 4376 scope.go:122] "RemoveContainer" containerID="fb3ba42b1998b8d6c9d673a03e1c679051143e65e9356c84feacc830afa26e8b" Oct 31 04:57:55.752157 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:57:55.752128 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:57:55.752237 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:57:55.752200 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:57:55.752280 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:57:55.752235 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:57:56.374065 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:57:56.374029 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-12c0bc-openshift-default-84bddf978c-ghhlp"] Oct 31 04:57:56.380213 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:57:56.380188 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-12c0bc-openshift-default-84bddf978c-ghhlp" Oct 31 04:57:56.406584 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:57:56.406553 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-12c0bc-openshift-default-84bddf978c-ghhlp"] Oct 31 04:57:56.420596 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:57:56.420570 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-12c0bc-openshift-default-dockercfg-cskdc\"" Oct 31 04:57:56.484752 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:57:56.484717 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bde4a1c-a591-4864-8276-2c3a74b99e70-istio-podinfo\") pod \"e2e-inf-b-12c0bc-openshift-default-84bddf978c-ghhlp\" (UID: \"7bde4a1c-a591-4864-8276-2c3a74b99e70\") " pod="openshift-ingress/e2e-inf-b-12c0bc-openshift-default-84bddf978c-ghhlp" Oct 31 04:57:56.484944 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:57:56.484857 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bde4a1c-a591-4864-8276-2c3a74b99e70-istiod-ca-cert\") pod \"e2e-inf-b-12c0bc-openshift-default-84bddf978c-ghhlp\" (UID: \"7bde4a1c-a591-4864-8276-2c3a74b99e70\") " pod="openshift-ingress/e2e-inf-b-12c0bc-openshift-default-84bddf978c-ghhlp" Oct 31 04:57:56.484944 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:57:56.484925 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bde4a1c-a591-4864-8276-2c3a74b99e70-istio-data\") pod \"e2e-inf-b-12c0bc-openshift-default-84bddf978c-ghhlp\" (UID: \"7bde4a1c-a591-4864-8276-2c3a74b99e70\") " pod="openshift-ingress/e2e-inf-b-12c0bc-openshift-default-84bddf978c-ghhlp" Oct 31 04:57:56.485099 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:57:56.485081 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7crvm\" (UniqueName: \"kubernetes.io/projected/7bde4a1c-a591-4864-8276-2c3a74b99e70-kube-api-access-7crvm\") pod \"e2e-inf-b-12c0bc-openshift-default-84bddf978c-ghhlp\" (UID: \"7bde4a1c-a591-4864-8276-2c3a74b99e70\") " pod="openshift-ingress/e2e-inf-b-12c0bc-openshift-default-84bddf978c-ghhlp" Oct 31 04:57:56.485163 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:57:56.485133 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bde4a1c-a591-4864-8276-2c3a74b99e70-credential-socket\") pod \"e2e-inf-b-12c0bc-openshift-default-84bddf978c-ghhlp\" (UID: \"7bde4a1c-a591-4864-8276-2c3a74b99e70\") " pod="openshift-ingress/e2e-inf-b-12c0bc-openshift-default-84bddf978c-ghhlp" Oct 31 04:57:56.485242 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:57:56.485216 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bde4a1c-a591-4864-8276-2c3a74b99e70-istio-token\") pod \"e2e-inf-b-12c0bc-openshift-default-84bddf978c-ghhlp\" (UID: \"7bde4a1c-a591-4864-8276-2c3a74b99e70\") " pod="openshift-ingress/e2e-inf-b-12c0bc-openshift-default-84bddf978c-ghhlp" Oct 31 04:57:56.485396 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:57:56.485373 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bde4a1c-a591-4864-8276-2c3a74b99e70-istio-envoy\") pod \"e2e-inf-b-12c0bc-openshift-default-84bddf978c-ghhlp\" (UID: \"7bde4a1c-a591-4864-8276-2c3a74b99e70\") " pod="openshift-ingress/e2e-inf-b-12c0bc-openshift-default-84bddf978c-ghhlp" Oct 31 04:57:56.485447 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:57:56.485429 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bde4a1c-a591-4864-8276-2c3a74b99e70-workload-socket\") pod \"e2e-inf-b-12c0bc-openshift-default-84bddf978c-ghhlp\" (UID: \"7bde4a1c-a591-4864-8276-2c3a74b99e70\") " pod="openshift-ingress/e2e-inf-b-12c0bc-openshift-default-84bddf978c-ghhlp" Oct 31 04:57:56.485692 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:57:56.485666 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bde4a1c-a591-4864-8276-2c3a74b99e70-workload-certs\") pod \"e2e-inf-b-12c0bc-openshift-default-84bddf978c-ghhlp\" (UID: \"7bde4a1c-a591-4864-8276-2c3a74b99e70\") " pod="openshift-ingress/e2e-inf-b-12c0bc-openshift-default-84bddf978c-ghhlp" Oct 31 04:57:56.600167 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:57:56.600131 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bde4a1c-a591-4864-8276-2c3a74b99e70-istiod-ca-cert\") pod \"e2e-inf-b-12c0bc-openshift-default-84bddf978c-ghhlp\" (UID: \"7bde4a1c-a591-4864-8276-2c3a74b99e70\") " pod="openshift-ingress/e2e-inf-b-12c0bc-openshift-default-84bddf978c-ghhlp" Oct 31 04:57:56.601317 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:57:56.601293 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bde4a1c-a591-4864-8276-2c3a74b99e70-istio-data\") pod \"e2e-inf-b-12c0bc-openshift-default-84bddf978c-ghhlp\" (UID: \"7bde4a1c-a591-4864-8276-2c3a74b99e70\") " pod="openshift-ingress/e2e-inf-b-12c0bc-openshift-default-84bddf978c-ghhlp" Oct 31 04:57:56.601971 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:57:56.601950 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7crvm\" (UniqueName: \"kubernetes.io/projected/7bde4a1c-a591-4864-8276-2c3a74b99e70-kube-api-access-7crvm\") pod \"e2e-inf-b-12c0bc-openshift-default-84bddf978c-ghhlp\" (UID: \"7bde4a1c-a591-4864-8276-2c3a74b99e70\") " pod="openshift-ingress/e2e-inf-b-12c0bc-openshift-default-84bddf978c-ghhlp" Oct 31 04:57:56.602171 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:57:56.602142 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bde4a1c-a591-4864-8276-2c3a74b99e70-credential-socket\") pod \"e2e-inf-b-12c0bc-openshift-default-84bddf978c-ghhlp\" (UID: \"7bde4a1c-a591-4864-8276-2c3a74b99e70\") " pod="openshift-ingress/e2e-inf-b-12c0bc-openshift-default-84bddf978c-ghhlp" Oct 31 04:57:56.602873 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:57:56.602854 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bde4a1c-a591-4864-8276-2c3a74b99e70-istio-token\") pod \"e2e-inf-b-12c0bc-openshift-default-84bddf978c-ghhlp\" (UID: \"7bde4a1c-a591-4864-8276-2c3a74b99e70\") " pod="openshift-ingress/e2e-inf-b-12c0bc-openshift-default-84bddf978c-ghhlp" Oct 31 04:57:56.603018 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:57:56.603003 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bde4a1c-a591-4864-8276-2c3a74b99e70-istio-envoy\") pod \"e2e-inf-b-12c0bc-openshift-default-84bddf978c-ghhlp\" (UID: \"7bde4a1c-a591-4864-8276-2c3a74b99e70\") " pod="openshift-ingress/e2e-inf-b-12c0bc-openshift-default-84bddf978c-ghhlp" Oct 31 04:57:56.603152 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:57:56.603138 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bde4a1c-a591-4864-8276-2c3a74b99e70-workload-socket\") pod \"e2e-inf-b-12c0bc-openshift-default-84bddf978c-ghhlp\" (UID: \"7bde4a1c-a591-4864-8276-2c3a74b99e70\") " pod="openshift-ingress/e2e-inf-b-12c0bc-openshift-default-84bddf978c-ghhlp" Oct 31 04:57:56.603493 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:57:56.603474 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bde4a1c-a591-4864-8276-2c3a74b99e70-workload-certs\") pod \"e2e-inf-b-12c0bc-openshift-default-84bddf978c-ghhlp\" (UID: \"7bde4a1c-a591-4864-8276-2c3a74b99e70\") " pod="openshift-ingress/e2e-inf-b-12c0bc-openshift-default-84bddf978c-ghhlp" Oct 31 04:57:56.603654 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:57:56.603640 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bde4a1c-a591-4864-8276-2c3a74b99e70-istio-podinfo\") pod \"e2e-inf-b-12c0bc-openshift-default-84bddf978c-ghhlp\" (UID: \"7bde4a1c-a591-4864-8276-2c3a74b99e70\") " pod="openshift-ingress/e2e-inf-b-12c0bc-openshift-default-84bddf978c-ghhlp" Oct 31 04:57:56.604816 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:57:56.604794 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bde4a1c-a591-4864-8276-2c3a74b99e70-workload-socket\") pod \"e2e-inf-b-12c0bc-openshift-default-84bddf978c-ghhlp\" (UID: \"7bde4a1c-a591-4864-8276-2c3a74b99e70\") " pod="openshift-ingress/e2e-inf-b-12c0bc-openshift-default-84bddf978c-ghhlp" Oct 31 04:57:56.605450 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:57:56.605152 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bde4a1c-a591-4864-8276-2c3a74b99e70-workload-certs\") pod \"e2e-inf-b-12c0bc-openshift-default-84bddf978c-ghhlp\" (UID: \"7bde4a1c-a591-4864-8276-2c3a74b99e70\") " pod="openshift-ingress/e2e-inf-b-12c0bc-openshift-default-84bddf978c-ghhlp" Oct 31 04:57:56.605586 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:57:56.601211 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bde4a1c-a591-4864-8276-2c3a74b99e70-istiod-ca-cert\") pod \"e2e-inf-b-12c0bc-openshift-default-84bddf978c-ghhlp\" (UID: \"7bde4a1c-a591-4864-8276-2c3a74b99e70\") " pod="openshift-ingress/e2e-inf-b-12c0bc-openshift-default-84bddf978c-ghhlp" Oct 31 04:57:56.605686 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:57:56.602802 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bde4a1c-a591-4864-8276-2c3a74b99e70-credential-socket\") pod \"e2e-inf-b-12c0bc-openshift-default-84bddf978c-ghhlp\" (UID: \"7bde4a1c-a591-4864-8276-2c3a74b99e70\") " pod="openshift-ingress/e2e-inf-b-12c0bc-openshift-default-84bddf978c-ghhlp" Oct 31 04:57:56.605785 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:57:56.601802 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bde4a1c-a591-4864-8276-2c3a74b99e70-istio-data\") pod \"e2e-inf-b-12c0bc-openshift-default-84bddf978c-ghhlp\" (UID: \"7bde4a1c-a591-4864-8276-2c3a74b99e70\") " pod="openshift-ingress/e2e-inf-b-12c0bc-openshift-default-84bddf978c-ghhlp" Oct 31 04:57:56.646423 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:57:56.634939 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bde4a1c-a591-4864-8276-2c3a74b99e70-istio-podinfo\") pod \"e2e-inf-b-12c0bc-openshift-default-84bddf978c-ghhlp\" (UID: \"7bde4a1c-a591-4864-8276-2c3a74b99e70\") " pod="openshift-ingress/e2e-inf-b-12c0bc-openshift-default-84bddf978c-ghhlp" Oct 31 04:57:56.672347 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:57:56.663809 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bde4a1c-a591-4864-8276-2c3a74b99e70-istio-envoy\") pod \"e2e-inf-b-12c0bc-openshift-default-84bddf978c-ghhlp\" (UID: \"7bde4a1c-a591-4864-8276-2c3a74b99e70\") " pod="openshift-ingress/e2e-inf-b-12c0bc-openshift-default-84bddf978c-ghhlp" Oct 31 04:57:56.673100 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:57:56.673044 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bde4a1c-a591-4864-8276-2c3a74b99e70-istio-token\") pod \"e2e-inf-b-12c0bc-openshift-default-84bddf978c-ghhlp\" (UID: \"7bde4a1c-a591-4864-8276-2c3a74b99e70\") " pod="openshift-ingress/e2e-inf-b-12c0bc-openshift-default-84bddf978c-ghhlp" Oct 31 04:57:56.675891 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:57:56.675839 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7crvm\" (UniqueName: \"kubernetes.io/projected/7bde4a1c-a591-4864-8276-2c3a74b99e70-kube-api-access-7crvm\") pod \"e2e-inf-b-12c0bc-openshift-default-84bddf978c-ghhlp\" (UID: \"7bde4a1c-a591-4864-8276-2c3a74b99e70\") " pod="openshift-ingress/e2e-inf-b-12c0bc-openshift-default-84bddf978c-ghhlp" Oct 31 04:57:56.699693 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:57:56.699479 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-12c0bc-openshift-default-84bddf978c-ghhlp" Oct 31 04:57:56.827464 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:57:56.826122 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-fb346b-openshift-default-744c56d4f9-mhkhz" event={"ID":"a6c79800-45c4-4778-91ef-027ceba7bb8a","Type":"ContainerStarted","Data":"883f4122c1c076a8437e00ec21b3a649c13dc38046b2ca718b01132dfb4b1418"} Oct 31 04:57:57.129033 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:57:57.128998 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bde4a1c_a591_4864_8276_2c3a74b99e70.slice/crio-2b8f3231fb96c5a5495462598e123caa61adfc6999017d3139d30613273b3cec WatchSource:0}: Error finding container 2b8f3231fb96c5a5495462598e123caa61adfc6999017d3139d30613273b3cec: Status 404 returned error can't find the container with id 2b8f3231fb96c5a5495462598e123caa61adfc6999017d3139d30613273b3cec Oct 31 04:57:57.131467 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:57:57.131444 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-12c0bc-openshift-default-84bddf978c-ghhlp"] Oct 31 04:57:57.132261 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:57:57.132226 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:57:57.132379 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:57:57.132344 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:57:57.132456 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:57:57.132397 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:57:57.369595 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:57:57.364468 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-fb346b-openshift-default-744c56d4f9-mhkhz" Oct 31 04:57:57.370363 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:57:57.370046 4376 patch_prober.go:29] interesting pod/e2e-inf-b-fb346b-openshift-default-744c56d4f9-mhkhz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" start-of-body= Oct 31 04:57:57.370363 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:57:57.370100 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-fb346b-openshift-default-744c56d4f9-mhkhz" podUID="a6c79800-45c4-4778-91ef-027ceba7bb8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" Oct 31 04:57:57.834645 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:57:57.834607 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-12c0bc-openshift-default-84bddf978c-ghhlp" event={"ID":"7bde4a1c-a591-4864-8276-2c3a74b99e70","Type":"ContainerStarted","Data":"24de4e330c1dfdcedac40896966a3533b0dcb513c23465a2cebc3c5507138eb8"} Oct 31 04:57:57.835124 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:57:57.834665 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-12c0bc-openshift-default-84bddf978c-ghhlp" event={"ID":"7bde4a1c-a591-4864-8276-2c3a74b99e70","Type":"ContainerStarted","Data":"2b8f3231fb96c5a5495462598e123caa61adfc6999017d3139d30613273b3cec"} Oct 31 04:57:57.885538 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:57:57.885488 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-12c0bc-openshift-default-84bddf978c-ghhlp" podStartSLOduration=1.8854740639999998 podStartE2EDuration="1.885474064s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:57:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:57:57.883814565 +0000 UTC m=+3225.598149184" watchObservedRunningTime="2026-10-31 04:57:57.885474064 +0000 UTC m=+3225.599808681" Oct 31 04:57:58.363593 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:57:58.363560 4376 patch_prober.go:29] interesting pod/e2e-inf-b-fb346b-openshift-default-744c56d4f9-mhkhz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" start-of-body= Oct 31 04:57:58.363771 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:57:58.363611 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-fb346b-openshift-default-744c56d4f9-mhkhz" podUID="a6c79800-45c4-4778-91ef-027ceba7bb8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" Oct 31 04:57:58.700270 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:57:58.700235 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-12c0bc-openshift-default-84bddf978c-ghhlp" Oct 31 04:57:58.706825 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:57:58.706795 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-12c0bc-openshift-default-84bddf978c-ghhlp" Oct 31 04:57:58.867022 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:57:58.866990 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-12c0bc-openshift-default-84bddf978c-ghhlp" Oct 31 04:57:58.868692 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:57:58.868665 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-12c0bc-openshift-default-84bddf978c-ghhlp" Oct 31 04:57:59.370672 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:57:59.370635 4376 patch_prober.go:29] interesting pod/e2e-inf-b-fb346b-openshift-default-744c56d4f9-mhkhz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" start-of-body= Oct 31 04:57:59.370988 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:57:59.370957 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-fb346b-openshift-default-744c56d4f9-mhkhz" podUID="a6c79800-45c4-4778-91ef-027ceba7bb8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" Oct 31 04:58:00.365376 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:58:00.364766 4376 patch_prober.go:29] interesting pod/e2e-inf-b-fb346b-openshift-default-744c56d4f9-mhkhz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" start-of-body= Oct 31 04:58:00.365376 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:58:00.364863 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-fb346b-openshift-default-744c56d4f9-mhkhz" podUID="a6c79800-45c4-4778-91ef-027ceba7bb8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" Oct 31 04:58:01.364553 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:58:01.364520 4376 patch_prober.go:29] interesting pod/e2e-inf-b-fb346b-openshift-default-744c56d4f9-mhkhz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" start-of-body= Oct 31 04:58:01.364757 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:58:01.364571 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-fb346b-openshift-default-744c56d4f9-mhkhz" podUID="a6c79800-45c4-4778-91ef-027ceba7bb8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" Oct 31 04:58:02.213248 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:58:02.213211 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-12c0bc-69fdd4b4ff-n6qfz"] Oct 31 04:58:02.218410 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:58:02.218390 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-12c0bc-69fdd4b4ff-n6qfz" Oct 31 04:58:02.245659 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:58:02.245628 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-12c0bc-69fdd4b4ff-n6qfz"] Oct 31 04:58:02.306353 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:58:02.287657 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-12c0bc-dockercfg-ls2c9\"" Oct 31 04:58:02.306353 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:58:02.287751 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-12c0bc\"" Oct 31 04:58:02.339872 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:58:02.339821 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b2dd0ee-880c-4538-b706-3412b77b7a53-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-12c0bc-69fdd4b4ff-n6qfz\" (UID: \"7b2dd0ee-880c-4538-b706-3412b77b7a53\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-12c0bc-69fdd4b4ff-n6qfz" Oct 31 04:58:02.340148 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:58:02.340122 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6brh\" (UniqueName: \"kubernetes.io/projected/7b2dd0ee-880c-4538-b706-3412b77b7a53-kube-api-access-g6brh\") pod \"payload-pre-processing-e2e-inf-b-12c0bc-69fdd4b4ff-n6qfz\" (UID: \"7b2dd0ee-880c-4538-b706-3412b77b7a53\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-12c0bc-69fdd4b4ff-n6qfz" Oct 31 04:58:02.382355 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:58:02.374963 4376 patch_prober.go:29] interesting pod/e2e-inf-b-fb346b-openshift-default-744c56d4f9-mhkhz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" start-of-body= Oct 31 04:58:02.382355 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:58:02.375018 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-fb346b-openshift-default-744c56d4f9-mhkhz" podUID="a6c79800-45c4-4778-91ef-027ceba7bb8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" Oct 31 04:58:02.454351 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:58:02.441302 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6brh\" (UniqueName: \"kubernetes.io/projected/7b2dd0ee-880c-4538-b706-3412b77b7a53-kube-api-access-g6brh\") pod \"payload-pre-processing-e2e-inf-b-12c0bc-69fdd4b4ff-n6qfz\" (UID: \"7b2dd0ee-880c-4538-b706-3412b77b7a53\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-12c0bc-69fdd4b4ff-n6qfz" Oct 31 04:58:02.454351 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:58:02.441490 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b2dd0ee-880c-4538-b706-3412b77b7a53-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-12c0bc-69fdd4b4ff-n6qfz\" (UID: \"7b2dd0ee-880c-4538-b706-3412b77b7a53\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-12c0bc-69fdd4b4ff-n6qfz" Oct 31 04:58:02.454351 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:58:02.442481 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b2dd0ee-880c-4538-b706-3412b77b7a53-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-12c0bc-69fdd4b4ff-n6qfz\" (UID: \"7b2dd0ee-880c-4538-b706-3412b77b7a53\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-12c0bc-69fdd4b4ff-n6qfz" Oct 31 04:58:02.518616 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:58:02.480890 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6brh\" (UniqueName: \"kubernetes.io/projected/7b2dd0ee-880c-4538-b706-3412b77b7a53-kube-api-access-g6brh\") pod \"payload-pre-processing-e2e-inf-b-12c0bc-69fdd4b4ff-n6qfz\" (UID: \"7b2dd0ee-880c-4538-b706-3412b77b7a53\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-12c0bc-69fdd4b4ff-n6qfz" Oct 31 04:58:02.545144 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:58:02.543661 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-12c0bc-69fdd4b4ff-n6qfz" Oct 31 04:58:02.884130 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:58:02.884096 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b2dd0ee_880c_4538_b706_3412b77b7a53.slice/crio-66f0d2b537fb1b958eeca84f5330f6f062efe43cdd118075f7524a71f2ce89dd WatchSource:0}: Error finding container 66f0d2b537fb1b958eeca84f5330f6f062efe43cdd118075f7524a71f2ce89dd: Status 404 returned error can't find the container with id 66f0d2b537fb1b958eeca84f5330f6f062efe43cdd118075f7524a71f2ce89dd Oct 31 04:58:02.884794 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:58:02.884759 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-12c0bc-69fdd4b4ff-n6qfz"] Oct 31 04:58:02.904744 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:58:02.904712 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-12c0bc-69fdd4b4ff-n6qfz" event={"ID":"7b2dd0ee-880c-4538-b706-3412b77b7a53","Type":"ContainerStarted","Data":"66f0d2b537fb1b958eeca84f5330f6f062efe43cdd118075f7524a71f2ce89dd"} Oct 31 04:58:03.455661 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:58:03.455627 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-fb346b-openshift-default-744c56d4f9-mhkhz" Oct 31 04:58:03.459395 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:58:03.457463 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-fb346b-openshift-default-744c56d4f9-mhkhz" Oct 31 04:58:03.459395 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:58:03.458653 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-fb346b-openshift-default-744c56d4f9-mhkhz" Oct 31 04:58:03.914820 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:58:03.914786 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-12c0bc-69fdd4b4ff-n6qfz" event={"ID":"7b2dd0ee-880c-4538-b706-3412b77b7a53","Type":"ContainerStarted","Data":"9975fba59982c4c92ecf05c9f0e39b77d40553c133dc3bd3ba89b4d4b81e0ab9"} Oct 31 04:58:03.914991 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:58:03.914867 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-12c0bc-69fdd4b4ff-n6qfz" Oct 31 04:58:03.956515 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:58:03.956469 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-12c0bc-69fdd4b4ff-n6qfz" podStartSLOduration=1.956456607 podStartE2EDuration="1.956456607s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:58:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:58:03.955711378 +0000 UTC m=+3231.670046015" watchObservedRunningTime="2026-10-31 04:58:03.956456607 +0000 UTC m=+3231.670791223" Oct 31 04:58:11.001267 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:58:11.001222 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 04:58:14.926264 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:58:14.926235 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-12c0bc-69fdd4b4ff-n6qfz" Oct 31 04:58:41.001527 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:58:41.001492 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 04:58:51.800715 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:58:51.800681 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 04:59:11.008818 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:59:11.008775 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 04:59:24.435261 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:59:24.435233 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-108ad8/test-model-108ad8-kserve-7c99b696dc-jh5fg"] Oct 31 04:59:24.440373 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:59:24.440340 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-108ad8/test-model-108ad8-kserve-7c99b696dc-jh5fg" Oct 31 04:59:24.463738 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:59:24.463708 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-108ad8/test-model-108ad8-kserve-7c99b696dc-jh5fg"] Oct 31 04:59:24.470764 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:59:24.470738 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-108ad8\"/\"test-model-108ad8-kserve-self-signed-certs\"" Oct 31 04:59:24.470764 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:59:24.470758 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-108ad8\"/\"openshift-service-ca.crt\"" Oct 31 04:59:24.470930 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:59:24.470824 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-108ad8\"/\"kube-root-ca.crt\"" Oct 31 04:59:24.470930 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:59:24.470873 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-108ad8\"/\"default-dockercfg-xrckb\"" Oct 31 04:59:24.584528 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:59:24.584491 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njvcp\" (UniqueName: \"kubernetes.io/projected/70f810de-025a-402f-8c1b-5d70a156789b-kube-api-access-njvcp\") pod \"test-model-108ad8-kserve-7c99b696dc-jh5fg\" (UID: \"70f810de-025a-402f-8c1b-5d70a156789b\") " pod="ai-tenant-e2e-inf-a-108ad8/test-model-108ad8-kserve-7c99b696dc-jh5fg" Oct 31 04:59:24.584751 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:59:24.584604 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70f810de-025a-402f-8c1b-5d70a156789b-dshm\") pod \"test-model-108ad8-kserve-7c99b696dc-jh5fg\" (UID: \"70f810de-025a-402f-8c1b-5d70a156789b\") " pod="ai-tenant-e2e-inf-a-108ad8/test-model-108ad8-kserve-7c99b696dc-jh5fg" Oct 31 04:59:24.584751 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:59:24.584736 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70f810de-025a-402f-8c1b-5d70a156789b-model-cache\") pod \"test-model-108ad8-kserve-7c99b696dc-jh5fg\" (UID: \"70f810de-025a-402f-8c1b-5d70a156789b\") " pod="ai-tenant-e2e-inf-a-108ad8/test-model-108ad8-kserve-7c99b696dc-jh5fg" Oct 31 04:59:24.584900 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:59:24.584870 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70f810de-025a-402f-8c1b-5d70a156789b-tmp-dir\") pod \"test-model-108ad8-kserve-7c99b696dc-jh5fg\" (UID: \"70f810de-025a-402f-8c1b-5d70a156789b\") " pod="ai-tenant-e2e-inf-a-108ad8/test-model-108ad8-kserve-7c99b696dc-jh5fg" Oct 31 04:59:24.585049 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:59:24.585020 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70f810de-025a-402f-8c1b-5d70a156789b-tls-certs\") pod \"test-model-108ad8-kserve-7c99b696dc-jh5fg\" (UID: \"70f810de-025a-402f-8c1b-5d70a156789b\") " pod="ai-tenant-e2e-inf-a-108ad8/test-model-108ad8-kserve-7c99b696dc-jh5fg" Oct 31 04:59:24.585185 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:59:24.585111 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70f810de-025a-402f-8c1b-5d70a156789b-home\") pod \"test-model-108ad8-kserve-7c99b696dc-jh5fg\" (UID: \"70f810de-025a-402f-8c1b-5d70a156789b\") " pod="ai-tenant-e2e-inf-a-108ad8/test-model-108ad8-kserve-7c99b696dc-jh5fg" Oct 31 04:59:24.700678 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:59:24.697836 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70f810de-025a-402f-8c1b-5d70a156789b-model-cache\") pod \"test-model-108ad8-kserve-7c99b696dc-jh5fg\" (UID: \"70f810de-025a-402f-8c1b-5d70a156789b\") " pod="ai-tenant-e2e-inf-a-108ad8/test-model-108ad8-kserve-7c99b696dc-jh5fg" Oct 31 04:59:24.700678 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:59:24.697917 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70f810de-025a-402f-8c1b-5d70a156789b-tmp-dir\") pod \"test-model-108ad8-kserve-7c99b696dc-jh5fg\" (UID: \"70f810de-025a-402f-8c1b-5d70a156789b\") " pod="ai-tenant-e2e-inf-a-108ad8/test-model-108ad8-kserve-7c99b696dc-jh5fg" Oct 31 04:59:24.700678 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:59:24.698067 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70f810de-025a-402f-8c1b-5d70a156789b-tls-certs\") pod \"test-model-108ad8-kserve-7c99b696dc-jh5fg\" (UID: \"70f810de-025a-402f-8c1b-5d70a156789b\") " pod="ai-tenant-e2e-inf-a-108ad8/test-model-108ad8-kserve-7c99b696dc-jh5fg" Oct 31 04:59:24.700678 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:59:24.698180 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70f810de-025a-402f-8c1b-5d70a156789b-home\") pod \"test-model-108ad8-kserve-7c99b696dc-jh5fg\" (UID: \"70f810de-025a-402f-8c1b-5d70a156789b\") " pod="ai-tenant-e2e-inf-a-108ad8/test-model-108ad8-kserve-7c99b696dc-jh5fg" Oct 31 04:59:24.700678 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:59:24.698262 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njvcp\" (UniqueName: \"kubernetes.io/projected/70f810de-025a-402f-8c1b-5d70a156789b-kube-api-access-njvcp\") pod \"test-model-108ad8-kserve-7c99b696dc-jh5fg\" (UID: \"70f810de-025a-402f-8c1b-5d70a156789b\") " pod="ai-tenant-e2e-inf-a-108ad8/test-model-108ad8-kserve-7c99b696dc-jh5fg" Oct 31 04:59:24.700678 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:59:24.698302 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70f810de-025a-402f-8c1b-5d70a156789b-dshm\") pod \"test-model-108ad8-kserve-7c99b696dc-jh5fg\" (UID: \"70f810de-025a-402f-8c1b-5d70a156789b\") " pod="ai-tenant-e2e-inf-a-108ad8/test-model-108ad8-kserve-7c99b696dc-jh5fg" Oct 31 04:59:24.711353 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:59:24.709938 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70f810de-025a-402f-8c1b-5d70a156789b-model-cache\") pod \"test-model-108ad8-kserve-7c99b696dc-jh5fg\" (UID: \"70f810de-025a-402f-8c1b-5d70a156789b\") " pod="ai-tenant-e2e-inf-a-108ad8/test-model-108ad8-kserve-7c99b696dc-jh5fg" Oct 31 04:59:24.711353 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:59:24.710202 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70f810de-025a-402f-8c1b-5d70a156789b-home\") pod \"test-model-108ad8-kserve-7c99b696dc-jh5fg\" (UID: \"70f810de-025a-402f-8c1b-5d70a156789b\") " pod="ai-tenant-e2e-inf-a-108ad8/test-model-108ad8-kserve-7c99b696dc-jh5fg" Oct 31 04:59:24.711353 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:59:24.710463 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70f810de-025a-402f-8c1b-5d70a156789b-tmp-dir\") pod \"test-model-108ad8-kserve-7c99b696dc-jh5fg\" (UID: \"70f810de-025a-402f-8c1b-5d70a156789b\") " pod="ai-tenant-e2e-inf-a-108ad8/test-model-108ad8-kserve-7c99b696dc-jh5fg" Oct 31 04:59:24.735161 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:59:24.735129 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70f810de-025a-402f-8c1b-5d70a156789b-dshm\") pod \"test-model-108ad8-kserve-7c99b696dc-jh5fg\" (UID: \"70f810de-025a-402f-8c1b-5d70a156789b\") " pod="ai-tenant-e2e-inf-a-108ad8/test-model-108ad8-kserve-7c99b696dc-jh5fg" Oct 31 04:59:24.754369 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:59:24.745997 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-njvcp\" (UniqueName: \"kubernetes.io/projected/70f810de-025a-402f-8c1b-5d70a156789b-kube-api-access-njvcp\") pod \"test-model-108ad8-kserve-7c99b696dc-jh5fg\" (UID: \"70f810de-025a-402f-8c1b-5d70a156789b\") " pod="ai-tenant-e2e-inf-a-108ad8/test-model-108ad8-kserve-7c99b696dc-jh5fg" Oct 31 04:59:24.762945 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:59:24.761300 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70f810de-025a-402f-8c1b-5d70a156789b-tls-certs\") pod \"test-model-108ad8-kserve-7c99b696dc-jh5fg\" (UID: \"70f810de-025a-402f-8c1b-5d70a156789b\") " pod="ai-tenant-e2e-inf-a-108ad8/test-model-108ad8-kserve-7c99b696dc-jh5fg" Oct 31 04:59:24.763651 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:59:24.763625 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-108ad8/test-model-108ad8-kserve-7c99b696dc-jh5fg" Oct 31 04:59:25.189739 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:59:25.189712 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod70f810de_025a_402f_8c1b_5d70a156789b.slice/crio-e868c8ebc5e34cd4a8f8eaf641e97569798f2eb36d594d97c496f98b701b82c6 WatchSource:0}: Error finding container e868c8ebc5e34cd4a8f8eaf641e97569798f2eb36d594d97c496f98b701b82c6: Status 404 returned error can't find the container with id e868c8ebc5e34cd4a8f8eaf641e97569798f2eb36d594d97c496f98b701b82c6 Oct 31 04:59:25.192870 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:59:25.192852 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-108ad8/test-model-108ad8-kserve-7c99b696dc-jh5fg"] Oct 31 04:59:25.797687 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:59:25.797645 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-108ad8/test-model-108ad8-kserve-7c99b696dc-jh5fg" event={"ID":"70f810de-025a-402f-8c1b-5d70a156789b","Type":"ContainerStarted","Data":"f9d2a3c8e0cff254e8d34de1a8c8ec77e644854505334f8f0d16df1a412183bb"} Oct 31 04:59:25.798077 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:59:25.797693 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-108ad8/test-model-108ad8-kserve-7c99b696dc-jh5fg" event={"ID":"70f810de-025a-402f-8c1b-5d70a156789b","Type":"ContainerStarted","Data":"e868c8ebc5e34cd4a8f8eaf641e97569798f2eb36d594d97c496f98b701b82c6"} Oct 31 04:59:25.842781 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:59:25.842727 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-108ad8/test-model-108ad8-kserve-7c99b696dc-jh5fg" podStartSLOduration=1.842714668 podStartE2EDuration="1.842714668s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:59:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:59:25.838787532 +0000 UTC m=+3313.553122150" watchObservedRunningTime="2026-10-31 04:59:25.842714668 +0000 UTC m=+3313.557049284" Oct 31 04:59:26.764488 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:59:26.764453 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-108ad8/test-model-108ad8-kserve-7c99b696dc-jh5fg" Oct 31 04:59:34.764986 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:59:34.764958 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-108ad8/test-model-108ad8-kserve-7c99b696dc-jh5fg" Oct 31 04:59:34.777801 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:59:34.777778 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-108ad8/test-model-108ad8-kserve-7c99b696dc-jh5fg" Oct 31 04:59:34.881323 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:59:34.881297 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-108ad8/test-model-108ad8-kserve-7c99b696dc-jh5fg" Oct 31 04:59:41.000997 ip-10-0-134-155 kubenswrapper[4376]: E1031 04:59:41.000958 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 04:59:47.756800 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:59:47.756756 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-47e9af/test-model-47e9af-kserve-5557db6747-hkddj"] Oct 31 04:59:47.763244 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:59:47.763217 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-47e9af/test-model-47e9af-kserve-5557db6747-hkddj" Oct 31 04:59:47.779798 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:59:47.779755 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-47e9af/test-model-47e9af-kserve-5557db6747-hkddj"] Oct 31 04:59:47.786175 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:59:47.786148 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-47e9af\"/\"kube-root-ca.crt\"" Oct 31 04:59:47.792087 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:59:47.792065 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-47e9af\"/\"openshift-service-ca.crt\"" Oct 31 04:59:47.792578 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:59:47.792090 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-47e9af\"/\"default-dockercfg-hwcgq\"" Oct 31 04:59:47.792578 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:59:47.792165 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-47e9af\"/\"test-model-47e9af-kserve-self-signed-certs\"" Oct 31 04:59:47.792946 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:59:47.792887 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdmjj\" (UniqueName: \"kubernetes.io/projected/981a91d9-8a9c-4d3e-a2fb-1e69d12624f8-kube-api-access-cdmjj\") pod \"test-model-47e9af-kserve-5557db6747-hkddj\" (UID: \"981a91d9-8a9c-4d3e-a2fb-1e69d12624f8\") " pod="ai-tenant-e2e-inf-b-47e9af/test-model-47e9af-kserve-5557db6747-hkddj" Oct 31 04:59:47.793226 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:59:47.792947 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/981a91d9-8a9c-4d3e-a2fb-1e69d12624f8-dshm\") pod \"test-model-47e9af-kserve-5557db6747-hkddj\" (UID: \"981a91d9-8a9c-4d3e-a2fb-1e69d12624f8\") " pod="ai-tenant-e2e-inf-b-47e9af/test-model-47e9af-kserve-5557db6747-hkddj" Oct 31 04:59:47.793226 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:59:47.793014 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/981a91d9-8a9c-4d3e-a2fb-1e69d12624f8-tls-certs\") pod \"test-model-47e9af-kserve-5557db6747-hkddj\" (UID: \"981a91d9-8a9c-4d3e-a2fb-1e69d12624f8\") " pod="ai-tenant-e2e-inf-b-47e9af/test-model-47e9af-kserve-5557db6747-hkddj" Oct 31 04:59:47.793226 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:59:47.793074 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/981a91d9-8a9c-4d3e-a2fb-1e69d12624f8-tmp-dir\") pod \"test-model-47e9af-kserve-5557db6747-hkddj\" (UID: \"981a91d9-8a9c-4d3e-a2fb-1e69d12624f8\") " pod="ai-tenant-e2e-inf-b-47e9af/test-model-47e9af-kserve-5557db6747-hkddj" Oct 31 04:59:47.795082 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:59:47.795040 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/981a91d9-8a9c-4d3e-a2fb-1e69d12624f8-model-cache\") pod \"test-model-47e9af-kserve-5557db6747-hkddj\" (UID: \"981a91d9-8a9c-4d3e-a2fb-1e69d12624f8\") " pod="ai-tenant-e2e-inf-b-47e9af/test-model-47e9af-kserve-5557db6747-hkddj" Oct 31 04:59:47.795192 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:59:47.795131 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/981a91d9-8a9c-4d3e-a2fb-1e69d12624f8-home\") pod \"test-model-47e9af-kserve-5557db6747-hkddj\" (UID: \"981a91d9-8a9c-4d3e-a2fb-1e69d12624f8\") " pod="ai-tenant-e2e-inf-b-47e9af/test-model-47e9af-kserve-5557db6747-hkddj" Oct 31 04:59:47.896813 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:59:47.896784 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/981a91d9-8a9c-4d3e-a2fb-1e69d12624f8-dshm\") pod \"test-model-47e9af-kserve-5557db6747-hkddj\" (UID: \"981a91d9-8a9c-4d3e-a2fb-1e69d12624f8\") " pod="ai-tenant-e2e-inf-b-47e9af/test-model-47e9af-kserve-5557db6747-hkddj" Oct 31 04:59:47.896959 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:59:47.896835 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/981a91d9-8a9c-4d3e-a2fb-1e69d12624f8-tls-certs\") pod \"test-model-47e9af-kserve-5557db6747-hkddj\" (UID: \"981a91d9-8a9c-4d3e-a2fb-1e69d12624f8\") " pod="ai-tenant-e2e-inf-b-47e9af/test-model-47e9af-kserve-5557db6747-hkddj" Oct 31 04:59:47.896959 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:59:47.896867 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/981a91d9-8a9c-4d3e-a2fb-1e69d12624f8-tmp-dir\") pod \"test-model-47e9af-kserve-5557db6747-hkddj\" (UID: \"981a91d9-8a9c-4d3e-a2fb-1e69d12624f8\") " pod="ai-tenant-e2e-inf-b-47e9af/test-model-47e9af-kserve-5557db6747-hkddj" Oct 31 04:59:47.897089 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:59:47.896972 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/981a91d9-8a9c-4d3e-a2fb-1e69d12624f8-model-cache\") pod \"test-model-47e9af-kserve-5557db6747-hkddj\" (UID: \"981a91d9-8a9c-4d3e-a2fb-1e69d12624f8\") " pod="ai-tenant-e2e-inf-b-47e9af/test-model-47e9af-kserve-5557db6747-hkddj" Oct 31 04:59:47.897089 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:59:47.897031 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/981a91d9-8a9c-4d3e-a2fb-1e69d12624f8-home\") pod \"test-model-47e9af-kserve-5557db6747-hkddj\" (UID: \"981a91d9-8a9c-4d3e-a2fb-1e69d12624f8\") " pod="ai-tenant-e2e-inf-b-47e9af/test-model-47e9af-kserve-5557db6747-hkddj" Oct 31 04:59:47.897089 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:59:47.897080 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdmjj\" (UniqueName: \"kubernetes.io/projected/981a91d9-8a9c-4d3e-a2fb-1e69d12624f8-kube-api-access-cdmjj\") pod \"test-model-47e9af-kserve-5557db6747-hkddj\" (UID: \"981a91d9-8a9c-4d3e-a2fb-1e69d12624f8\") " pod="ai-tenant-e2e-inf-b-47e9af/test-model-47e9af-kserve-5557db6747-hkddj" Oct 31 04:59:47.897389 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:59:47.897365 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/981a91d9-8a9c-4d3e-a2fb-1e69d12624f8-tmp-dir\") pod \"test-model-47e9af-kserve-5557db6747-hkddj\" (UID: \"981a91d9-8a9c-4d3e-a2fb-1e69d12624f8\") " pod="ai-tenant-e2e-inf-b-47e9af/test-model-47e9af-kserve-5557db6747-hkddj" Oct 31 04:59:47.897478 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:59:47.897424 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/981a91d9-8a9c-4d3e-a2fb-1e69d12624f8-model-cache\") pod \"test-model-47e9af-kserve-5557db6747-hkddj\" (UID: \"981a91d9-8a9c-4d3e-a2fb-1e69d12624f8\") " pod="ai-tenant-e2e-inf-b-47e9af/test-model-47e9af-kserve-5557db6747-hkddj" Oct 31 04:59:47.897478 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:59:47.897445 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/981a91d9-8a9c-4d3e-a2fb-1e69d12624f8-home\") pod \"test-model-47e9af-kserve-5557db6747-hkddj\" (UID: \"981a91d9-8a9c-4d3e-a2fb-1e69d12624f8\") " pod="ai-tenant-e2e-inf-b-47e9af/test-model-47e9af-kserve-5557db6747-hkddj" Oct 31 04:59:47.899060 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:59:47.899042 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/981a91d9-8a9c-4d3e-a2fb-1e69d12624f8-dshm\") pod \"test-model-47e9af-kserve-5557db6747-hkddj\" (UID: \"981a91d9-8a9c-4d3e-a2fb-1e69d12624f8\") " pod="ai-tenant-e2e-inf-b-47e9af/test-model-47e9af-kserve-5557db6747-hkddj" Oct 31 04:59:47.899240 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:59:47.899225 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/981a91d9-8a9c-4d3e-a2fb-1e69d12624f8-tls-certs\") pod \"test-model-47e9af-kserve-5557db6747-hkddj\" (UID: \"981a91d9-8a9c-4d3e-a2fb-1e69d12624f8\") " pod="ai-tenant-e2e-inf-b-47e9af/test-model-47e9af-kserve-5557db6747-hkddj" Oct 31 04:59:47.920460 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:59:47.920437 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdmjj\" (UniqueName: \"kubernetes.io/projected/981a91d9-8a9c-4d3e-a2fb-1e69d12624f8-kube-api-access-cdmjj\") pod \"test-model-47e9af-kserve-5557db6747-hkddj\" (UID: \"981a91d9-8a9c-4d3e-a2fb-1e69d12624f8\") " pod="ai-tenant-e2e-inf-b-47e9af/test-model-47e9af-kserve-5557db6747-hkddj" Oct 31 04:59:47.927715 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:59:47.927692 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 04:59:48.110421 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:59:48.083930 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-47e9af/test-model-47e9af-kserve-5557db6747-hkddj" Oct 31 04:59:48.492109 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:59:48.492073 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod981a91d9_8a9c_4d3e_a2fb_1e69d12624f8.slice/crio-fdc25705f477129d9363afba30e262bf1f510ceb116ff7334fc18e34a574275b WatchSource:0}: Error finding container fdc25705f477129d9363afba30e262bf1f510ceb116ff7334fc18e34a574275b: Status 404 returned error can't find the container with id fdc25705f477129d9363afba30e262bf1f510ceb116ff7334fc18e34a574275b Oct 31 04:59:48.494040 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:59:48.494016 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-47e9af/test-model-47e9af-kserve-5557db6747-hkddj"] Oct 31 04:59:49.023781 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:59:49.023743 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-47e9af/test-model-47e9af-kserve-5557db6747-hkddj" event={"ID":"981a91d9-8a9c-4d3e-a2fb-1e69d12624f8","Type":"ContainerStarted","Data":"c52464818f34c56acd634f3f426fee2521fd6af54da97f2d8ac8bad35c166a77"} Oct 31 04:59:49.023781 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:59:49.023790 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-47e9af/test-model-47e9af-kserve-5557db6747-hkddj" event={"ID":"981a91d9-8a9c-4d3e-a2fb-1e69d12624f8","Type":"ContainerStarted","Data":"fdc25705f477129d9363afba30e262bf1f510ceb116ff7334fc18e34a574275b"} Oct 31 04:59:49.070231 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:59:49.070183 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-47e9af/test-model-47e9af-kserve-5557db6747-hkddj" podStartSLOduration=2.0701697550000002 podStartE2EDuration="2.070169755s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:59:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:59:49.068940328 +0000 UTC m=+3336.783274945" watchObservedRunningTime="2026-10-31 04:59:49.070169755 +0000 UTC m=+3336.784504371" Oct 31 04:59:49.084002 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:59:49.083977 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-47e9af/test-model-47e9af-kserve-5557db6747-hkddj" Oct 31 04:59:50.833357 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:59:50.833083 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-04d111/test-model-04d111-kserve-5d676b5997-f7fqj"] Oct 31 04:59:50.845393 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:59:50.841776 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-04d111/test-model-04d111-kserve-5d676b5997-f7fqj" Oct 31 04:59:50.880933 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:59:50.874264 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-04d111\"/\"kube-root-ca.crt\"" Oct 31 04:59:50.880933 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:59:50.875038 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-04d111\"/\"default-dockercfg-jbzws\"" Oct 31 04:59:50.880933 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:59:50.875117 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-04d111\"/\"test-model-04d111-kserve-self-signed-certs\"" Oct 31 04:59:50.883551 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:59:50.883511 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-04d111\"/\"openshift-service-ca.crt\"" Oct 31 04:59:50.890219 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:59:50.890189 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-04d111/test-model-04d111-kserve-5d676b5997-f7fqj"] Oct 31 04:59:50.900762 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:59:50.900721 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c48d91ef-0b60-4cad-8f97-7405a26337f0-tmp-dir\") pod \"test-model-04d111-kserve-5d676b5997-f7fqj\" (UID: \"c48d91ef-0b60-4cad-8f97-7405a26337f0\") " pod="ai-tenant-e2e-inf-a-04d111/test-model-04d111-kserve-5d676b5997-f7fqj" Oct 31 04:59:50.900959 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:59:50.900939 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c48d91ef-0b60-4cad-8f97-7405a26337f0-tls-certs\") pod \"test-model-04d111-kserve-5d676b5997-f7fqj\" (UID: \"c48d91ef-0b60-4cad-8f97-7405a26337f0\") " pod="ai-tenant-e2e-inf-a-04d111/test-model-04d111-kserve-5d676b5997-f7fqj" Oct 31 04:59:50.901162 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:59:50.901142 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c48d91ef-0b60-4cad-8f97-7405a26337f0-model-cache\") pod \"test-model-04d111-kserve-5d676b5997-f7fqj\" (UID: \"c48d91ef-0b60-4cad-8f97-7405a26337f0\") " pod="ai-tenant-e2e-inf-a-04d111/test-model-04d111-kserve-5d676b5997-f7fqj" Oct 31 04:59:50.901234 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:59:50.901185 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c48d91ef-0b60-4cad-8f97-7405a26337f0-dshm\") pod \"test-model-04d111-kserve-5d676b5997-f7fqj\" (UID: \"c48d91ef-0b60-4cad-8f97-7405a26337f0\") " pod="ai-tenant-e2e-inf-a-04d111/test-model-04d111-kserve-5d676b5997-f7fqj" Oct 31 04:59:50.901294 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:59:50.901257 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsv9l\" (UniqueName: \"kubernetes.io/projected/c48d91ef-0b60-4cad-8f97-7405a26337f0-kube-api-access-hsv9l\") pod \"test-model-04d111-kserve-5d676b5997-f7fqj\" (UID: \"c48d91ef-0b60-4cad-8f97-7405a26337f0\") " pod="ai-tenant-e2e-inf-a-04d111/test-model-04d111-kserve-5d676b5997-f7fqj" Oct 31 04:59:50.901384 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:59:50.901308 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c48d91ef-0b60-4cad-8f97-7405a26337f0-home\") pod \"test-model-04d111-kserve-5d676b5997-f7fqj\" (UID: \"c48d91ef-0b60-4cad-8f97-7405a26337f0\") " pod="ai-tenant-e2e-inf-a-04d111/test-model-04d111-kserve-5d676b5997-f7fqj" Oct 31 04:59:51.003176 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:59:51.003146 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c48d91ef-0b60-4cad-8f97-7405a26337f0-tls-certs\") pod \"test-model-04d111-kserve-5d676b5997-f7fqj\" (UID: \"c48d91ef-0b60-4cad-8f97-7405a26337f0\") " pod="ai-tenant-e2e-inf-a-04d111/test-model-04d111-kserve-5d676b5997-f7fqj" Oct 31 04:59:51.003377 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:59:51.003346 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c48d91ef-0b60-4cad-8f97-7405a26337f0-model-cache\") pod \"test-model-04d111-kserve-5d676b5997-f7fqj\" (UID: \"c48d91ef-0b60-4cad-8f97-7405a26337f0\") " pod="ai-tenant-e2e-inf-a-04d111/test-model-04d111-kserve-5d676b5997-f7fqj" Oct 31 04:59:51.003461 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:59:51.003409 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c48d91ef-0b60-4cad-8f97-7405a26337f0-dshm\") pod \"test-model-04d111-kserve-5d676b5997-f7fqj\" (UID: \"c48d91ef-0b60-4cad-8f97-7405a26337f0\") " pod="ai-tenant-e2e-inf-a-04d111/test-model-04d111-kserve-5d676b5997-f7fqj" Oct 31 04:59:51.003525 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:59:51.003475 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsv9l\" (UniqueName: \"kubernetes.io/projected/c48d91ef-0b60-4cad-8f97-7405a26337f0-kube-api-access-hsv9l\") pod \"test-model-04d111-kserve-5d676b5997-f7fqj\" (UID: \"c48d91ef-0b60-4cad-8f97-7405a26337f0\") " pod="ai-tenant-e2e-inf-a-04d111/test-model-04d111-kserve-5d676b5997-f7fqj" Oct 31 04:59:51.003589 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:59:51.003532 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c48d91ef-0b60-4cad-8f97-7405a26337f0-home\") pod \"test-model-04d111-kserve-5d676b5997-f7fqj\" (UID: \"c48d91ef-0b60-4cad-8f97-7405a26337f0\") " pod="ai-tenant-e2e-inf-a-04d111/test-model-04d111-kserve-5d676b5997-f7fqj" Oct 31 04:59:51.003653 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:59:51.003612 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c48d91ef-0b60-4cad-8f97-7405a26337f0-tmp-dir\") pod \"test-model-04d111-kserve-5d676b5997-f7fqj\" (UID: \"c48d91ef-0b60-4cad-8f97-7405a26337f0\") " pod="ai-tenant-e2e-inf-a-04d111/test-model-04d111-kserve-5d676b5997-f7fqj" Oct 31 04:59:51.003811 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:59:51.003775 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c48d91ef-0b60-4cad-8f97-7405a26337f0-model-cache\") pod \"test-model-04d111-kserve-5d676b5997-f7fqj\" (UID: \"c48d91ef-0b60-4cad-8f97-7405a26337f0\") " pod="ai-tenant-e2e-inf-a-04d111/test-model-04d111-kserve-5d676b5997-f7fqj" Oct 31 04:59:51.003991 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:59:51.003969 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c48d91ef-0b60-4cad-8f97-7405a26337f0-home\") pod \"test-model-04d111-kserve-5d676b5997-f7fqj\" (UID: \"c48d91ef-0b60-4cad-8f97-7405a26337f0\") " pod="ai-tenant-e2e-inf-a-04d111/test-model-04d111-kserve-5d676b5997-f7fqj" Oct 31 04:59:51.004084 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:59:51.004045 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c48d91ef-0b60-4cad-8f97-7405a26337f0-tmp-dir\") pod \"test-model-04d111-kserve-5d676b5997-f7fqj\" (UID: \"c48d91ef-0b60-4cad-8f97-7405a26337f0\") " pod="ai-tenant-e2e-inf-a-04d111/test-model-04d111-kserve-5d676b5997-f7fqj" Oct 31 04:59:51.005662 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:59:51.005644 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c48d91ef-0b60-4cad-8f97-7405a26337f0-dshm\") pod \"test-model-04d111-kserve-5d676b5997-f7fqj\" (UID: \"c48d91ef-0b60-4cad-8f97-7405a26337f0\") " pod="ai-tenant-e2e-inf-a-04d111/test-model-04d111-kserve-5d676b5997-f7fqj" Oct 31 04:59:51.006047 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:59:51.006028 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c48d91ef-0b60-4cad-8f97-7405a26337f0-tls-certs\") pod \"test-model-04d111-kserve-5d676b5997-f7fqj\" (UID: \"c48d91ef-0b60-4cad-8f97-7405a26337f0\") " pod="ai-tenant-e2e-inf-a-04d111/test-model-04d111-kserve-5d676b5997-f7fqj" Oct 31 04:59:51.026890 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:59:51.026863 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsv9l\" (UniqueName: \"kubernetes.io/projected/c48d91ef-0b60-4cad-8f97-7405a26337f0-kube-api-access-hsv9l\") pod \"test-model-04d111-kserve-5d676b5997-f7fqj\" (UID: \"c48d91ef-0b60-4cad-8f97-7405a26337f0\") " pod="ai-tenant-e2e-inf-a-04d111/test-model-04d111-kserve-5d676b5997-f7fqj" Oct 31 04:59:51.203131 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:59:51.200499 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-04d111/test-model-04d111-kserve-5d676b5997-f7fqj" Oct 31 04:59:51.614844 ip-10-0-134-155 kubenswrapper[4376]: W1031 04:59:51.614815 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc48d91ef_0b60_4cad_8f97_7405a26337f0.slice/crio-e633b5f912630429e5292cd9babb5238dc0830470dbb049a048ce4e76bb88f45 WatchSource:0}: Error finding container e633b5f912630429e5292cd9babb5238dc0830470dbb049a048ce4e76bb88f45: Status 404 returned error can't find the container with id e633b5f912630429e5292cd9babb5238dc0830470dbb049a048ce4e76bb88f45 Oct 31 04:59:51.617487 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:59:51.617456 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-04d111/test-model-04d111-kserve-5d676b5997-f7fqj"] Oct 31 04:59:52.102194 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:59:52.102155 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-04d111/test-model-04d111-kserve-5d676b5997-f7fqj" event={"ID":"c48d91ef-0b60-4cad-8f97-7405a26337f0","Type":"ContainerStarted","Data":"a699f7dfc0299c485ac4f6d7874ad40698576cf2ccd15921104919d194e5bbc6"} Oct 31 04:59:52.102693 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:59:52.102206 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-04d111/test-model-04d111-kserve-5d676b5997-f7fqj" event={"ID":"c48d91ef-0b60-4cad-8f97-7405a26337f0","Type":"ContainerStarted","Data":"e633b5f912630429e5292cd9babb5238dc0830470dbb049a048ce4e76bb88f45"} Oct 31 04:59:52.204137 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:59:52.204088 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-04d111/test-model-04d111-kserve-5d676b5997-f7fqj" Oct 31 04:59:58.084383 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:59:58.084343 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-47e9af/test-model-47e9af-kserve-5557db6747-hkddj" Oct 31 04:59:58.097601 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:59:58.097567 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-47e9af/test-model-47e9af-kserve-5557db6747-hkddj" Oct 31 04:59:58.140548 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:59:58.140490 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-04d111/test-model-04d111-kserve-5d676b5997-f7fqj" podStartSLOduration=8.140477541 podStartE2EDuration="8.140477541s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:59:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:59:52.167590829 +0000 UTC m=+3339.881925447" watchObservedRunningTime="2026-10-31 04:59:58.140477541 +0000 UTC m=+3345.854812162" Oct 31 04:59:58.209532 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:59:58.209501 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-47e9af/test-model-47e9af-kserve-5557db6747-hkddj" Oct 31 04:59:59.908564 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:59:59.908536 4376 patch_prober.go:29] interesting pod/e2e-inf-b-12c0bc-openshift-default-84bddf978c-ghhlp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.221:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 04:59:59.909720 ip-10-0-134-155 kubenswrapper[4376]: I1031 04:59:59.908609 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-12c0bc-openshift-default-84bddf978c-ghhlp" podUID="7bde4a1c-a591-4864-8276-2c3a74b99e70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.221:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 05:00:00.216496 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:00.216411 4376 generic.go:356] "Generic (PLEG): container finished" podID="7bde4a1c-a591-4864-8276-2c3a74b99e70" containerID="24de4e330c1dfdcedac40896966a3533b0dcb513c23465a2cebc3c5507138eb8" exitCode=0 Oct 31 05:00:00.216496 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:00.216446 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-12c0bc-openshift-default-84bddf978c-ghhlp" event={"ID":"7bde4a1c-a591-4864-8276-2c3a74b99e70","Type":"ContainerDied","Data":"24de4e330c1dfdcedac40896966a3533b0dcb513c23465a2cebc3c5507138eb8"} Oct 31 05:00:00.216960 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:00.216939 4376 scope.go:122] "RemoveContainer" containerID="24de4e330c1dfdcedac40896966a3533b0dcb513c23465a2cebc3c5507138eb8" Oct 31 05:00:00.249908 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:00.249864 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 05:00:00.250087 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:00.249963 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 05:00:00.250087 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:00.250013 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 05:00:00.343234 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:00.339636 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-12c0bc-29890380-wbz6z"] Oct 31 05:00:00.345671 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:00.345629 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-12c0bc-29890380-wbz6z" Oct 31 05:00:00.357462 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:00.357428 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-12c0bc-29890380-wbz6z"] Oct 31 05:00:00.369226 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:00.369195 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ded435-29890380-bwm8n"] Oct 31 05:00:00.374035 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:00.374009 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzb8r\"" Oct 31 05:00:00.374299 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:00.374283 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ded435-29890380-bwm8n" Oct 31 05:00:00.376699 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:00.376680 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9633f0-29890380-jc5lw"] Oct 31 05:00:00.383022 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:00.382988 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9633f0-29890380-jc5lw" Oct 31 05:00:00.384624 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:00.384608 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-439989-29890380-rf2nf"] Oct 31 05:00:00.406760 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:00.406737 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-439989-29890380-rf2nf" Oct 31 05:00:00.415549 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:00.415529 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ded435-29890380-bwm8n"] Oct 31 05:00:00.435790 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:00.421462 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9633f0-29890380-jc5lw"] Oct 31 05:00:00.435790 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:00.429266 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-766728-29890380-9xphp"] Oct 31 05:00:00.451057 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:00.446000 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmjft\" (UniqueName: \"kubernetes.io/projected/a63f4216-a71c-45b7-98b9-3b08500503fd-kube-api-access-dmjft\") pod \"maas-api-key-cleanup-e2e-inf-b-12c0bc-29890380-wbz6z\" (UID: \"a63f4216-a71c-45b7-98b9-3b08500503fd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-12c0bc-29890380-wbz6z" Oct 31 05:00:00.451057 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:00.446172 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a63f4216-a71c-45b7-98b9-3b08500503fd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-12c0bc-29890380-wbz6z\" (UID: \"a63f4216-a71c-45b7-98b9-3b08500503fd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-12c0bc-29890380-wbz6z" Oct 31 05:00:00.451595 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:00.451126 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-766728-29890380-9xphp" Oct 31 05:00:00.495253 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:00.495220 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d6ac82-29890380-xtwh8"] Oct 31 05:00:00.503408 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:00.503381 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-439989-29890380-rf2nf"] Oct 31 05:00:00.503550 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:00.503473 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-77a153-29890380-sdd6b"] Oct 31 05:00:00.506681 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:00.505829 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d6ac82-29890380-xtwh8" Oct 31 05:00:00.542279 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:00.542249 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-829ac7-29890380-pt7nf"] Oct 31 05:00:00.543630 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:00.543608 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-77a153-29890380-sdd6b" Oct 31 05:00:00.547501 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:00.547483 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-766728-29890380-9xphp"] Oct 31 05:00:00.548903 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:00.548884 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c1fe26-29890380-h89x7"] Oct 31 05:00:00.549841 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:00.548819 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-829ac7-29890380-pt7nf" Oct 31 05:00:00.550078 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:00.548708 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24pk8\" (UniqueName: \"kubernetes.io/projected/8dfadf3e-94d3-4e77-bd97-5bb554b03d2f-kube-api-access-24pk8\") pod \"maas-api-key-cleanup-e2e-worker-w1-439989-29890380-rf2nf\" (UID: \"8dfadf3e-94d3-4e77-bd97-5bb554b03d2f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-439989-29890380-rf2nf" Oct 31 05:00:00.550260 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:00.550242 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmjft\" (UniqueName: \"kubernetes.io/projected/a63f4216-a71c-45b7-98b9-3b08500503fd-kube-api-access-dmjft\") pod \"maas-api-key-cleanup-e2e-inf-b-12c0bc-29890380-wbz6z\" (UID: \"a63f4216-a71c-45b7-98b9-3b08500503fd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-12c0bc-29890380-wbz6z" Oct 31 05:00:00.550512 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:00.550491 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcbr5\" (UniqueName: \"kubernetes.io/projected/9850fba4-11d3-4785-adc0-5ddd89293842-kube-api-access-xcbr5\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-766728-29890380-9xphp\" (UID: \"9850fba4-11d3-4785-adc0-5ddd89293842\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-766728-29890380-9xphp" Oct 31 05:00:00.550710 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:00.550692 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a63f4216-a71c-45b7-98b9-3b08500503fd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-12c0bc-29890380-wbz6z\" (UID: \"a63f4216-a71c-45b7-98b9-3b08500503fd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-12c0bc-29890380-wbz6z" Oct 31 05:00:00.550925 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:00.550908 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9850fba4-11d3-4785-adc0-5ddd89293842-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-766728-29890380-9xphp\" (UID: \"9850fba4-11d3-4785-adc0-5ddd89293842\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-766728-29890380-9xphp" Oct 31 05:00:00.551077 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:00.551062 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f6a15d6-d47f-47de-817f-fc70feb236e8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-ded435-29890380-bwm8n\" (UID: \"5f6a15d6-d47f-47de-817f-fc70feb236e8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ded435-29890380-bwm8n" Oct 31 05:00:00.551191 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:00.551176 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ecbbd44a-efa7-4e45-8514-1a57eccc71e5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-9633f0-29890380-jc5lw\" (UID: \"ecbbd44a-efa7-4e45-8514-1a57eccc71e5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9633f0-29890380-jc5lw" Oct 31 05:00:00.551294 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:00.551281 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t97bc\" (UniqueName: \"kubernetes.io/projected/5f6a15d6-d47f-47de-817f-fc70feb236e8-kube-api-access-t97bc\") pod \"maas-api-key-cleanup-e2e-extmod-b-ded435-29890380-bwm8n\" (UID: \"5f6a15d6-d47f-47de-817f-fc70feb236e8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ded435-29890380-bwm8n" Oct 31 05:00:00.554966 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:00.554935 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2l466\" (UniqueName: \"kubernetes.io/projected/ecbbd44a-efa7-4e45-8514-1a57eccc71e5-kube-api-access-2l466\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-9633f0-29890380-jc5lw\" (UID: \"ecbbd44a-efa7-4e45-8514-1a57eccc71e5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9633f0-29890380-jc5lw" Oct 31 05:00:00.555083 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:00.555040 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8dfadf3e-94d3-4e77-bd97-5bb554b03d2f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-439989-29890380-rf2nf\" (UID: \"8dfadf3e-94d3-4e77-bd97-5bb554b03d2f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-439989-29890380-rf2nf" Oct 31 05:00:00.555423 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:00.555396 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a63f4216-a71c-45b7-98b9-3b08500503fd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-12c0bc-29890380-wbz6z\" (UID: \"a63f4216-a71c-45b7-98b9-3b08500503fd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-12c0bc-29890380-wbz6z" Oct 31 05:00:00.559366 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:00.559323 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d6ac82-29890380-xtwh8"] Oct 31 05:00:00.559467 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:00.559382 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-77a153-29890380-sdd6b"] Oct 31 05:00:00.559467 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:00.559462 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c1fe26-29890380-h89x7" Oct 31 05:00:00.564559 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:00.562244 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c9707f-29890380-k76x2"] Oct 31 05:00:00.566643 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:00.566624 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c9707f-29890380-k76x2" Oct 31 05:00:00.568473 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:00.568454 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-04d111-29890380-62zcn"] Oct 31 05:00:00.572141 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:00.572121 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-04d111-29890380-62zcn" Oct 31 05:00:00.574323 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:00.574306 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmjft\" (UniqueName: \"kubernetes.io/projected/a63f4216-a71c-45b7-98b9-3b08500503fd-kube-api-access-dmjft\") pod \"maas-api-key-cleanup-e2e-inf-b-12c0bc-29890380-wbz6z\" (UID: \"a63f4216-a71c-45b7-98b9-3b08500503fd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-12c0bc-29890380-wbz6z" Oct 31 05:00:00.609354 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:00.598429 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-171edd-29890380-2kgrc"] Oct 31 05:00:00.617750 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:00.615811 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-171edd-29890380-2kgrc" Oct 31 05:00:00.620491 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:00.618310 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c1fe26-29890380-h89x7"] Oct 31 05:00:00.639160 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:00.626623 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-04d111-29890380-62zcn"] Oct 31 05:00:00.639160 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:00.635313 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c9707f-29890380-k76x2"] Oct 31 05:00:00.659382 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:00.657166 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6f4r6\" (UniqueName: \"kubernetes.io/projected/978b4c58-1c25-4e04-bfbf-5cd4e0a7001d-kube-api-access-6f4r6\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-d6ac82-29890380-xtwh8\" (UID: \"978b4c58-1c25-4e04-bfbf-5cd4e0a7001d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d6ac82-29890380-xtwh8" Oct 31 05:00:00.659382 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:00.657243 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jc96x\" (UniqueName: \"kubernetes.io/projected/7b953e1c-6983-456f-b2c8-77e9314aa758-kube-api-access-jc96x\") pod \"maas-api-key-cleanup-e2e-inf-b-829ac7-29890380-pt7nf\" (UID: \"7b953e1c-6983-456f-b2c8-77e9314aa758\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-829ac7-29890380-pt7nf" Oct 31 05:00:00.659382 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:00.657296 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhg2z\" (UniqueName: \"kubernetes.io/projected/09f6c274-8499-4b06-91af-6af516c3e56a-kube-api-access-mhg2z\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-77a153-29890380-sdd6b\" (UID: \"09f6c274-8499-4b06-91af-6af516c3e56a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-77a153-29890380-sdd6b" Oct 31 05:00:00.659382 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:00.657437 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/978b4c58-1c25-4e04-bfbf-5cd4e0a7001d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-d6ac82-29890380-xtwh8\" (UID: \"978b4c58-1c25-4e04-bfbf-5cd4e0a7001d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d6ac82-29890380-xtwh8" Oct 31 05:00:00.659382 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:00.657534 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9850fba4-11d3-4785-adc0-5ddd89293842-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-766728-29890380-9xphp\" (UID: \"9850fba4-11d3-4785-adc0-5ddd89293842\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-766728-29890380-9xphp" Oct 31 05:00:00.659382 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:00.657580 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79b8c49d-28ac-49fa-b101-5f553fdac035-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-c1fe26-29890380-h89x7\" (UID: \"79b8c49d-28ac-49fa-b101-5f553fdac035\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c1fe26-29890380-h89x7" Oct 31 05:00:00.659382 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:00.657610 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f6a15d6-d47f-47de-817f-fc70feb236e8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-ded435-29890380-bwm8n\" (UID: \"5f6a15d6-d47f-47de-817f-fc70feb236e8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ded435-29890380-bwm8n" Oct 31 05:00:00.659382 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:00.657651 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ecbbd44a-efa7-4e45-8514-1a57eccc71e5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-9633f0-29890380-jc5lw\" (UID: \"ecbbd44a-efa7-4e45-8514-1a57eccc71e5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9633f0-29890380-jc5lw" Oct 31 05:00:00.659382 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:00.657684 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t97bc\" (UniqueName: \"kubernetes.io/projected/5f6a15d6-d47f-47de-817f-fc70feb236e8-kube-api-access-t97bc\") pod \"maas-api-key-cleanup-e2e-extmod-b-ded435-29890380-bwm8n\" (UID: \"5f6a15d6-d47f-47de-817f-fc70feb236e8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ded435-29890380-bwm8n" Oct 31 05:00:00.659382 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:00.657754 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2l466\" (UniqueName: \"kubernetes.io/projected/ecbbd44a-efa7-4e45-8514-1a57eccc71e5-kube-api-access-2l466\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-9633f0-29890380-jc5lw\" (UID: \"ecbbd44a-efa7-4e45-8514-1a57eccc71e5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9633f0-29890380-jc5lw" Oct 31 05:00:00.659382 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:00.657817 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwm5b\" (UniqueName: \"kubernetes.io/projected/79b8c49d-28ac-49fa-b101-5f553fdac035-kube-api-access-jwm5b\") pod \"maas-api-key-cleanup-e2e-inf-b-c1fe26-29890380-h89x7\" (UID: \"79b8c49d-28ac-49fa-b101-5f553fdac035\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c1fe26-29890380-h89x7" Oct 31 05:00:00.659382 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:00.657860 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8dfadf3e-94d3-4e77-bd97-5bb554b03d2f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-439989-29890380-rf2nf\" (UID: \"8dfadf3e-94d3-4e77-bd97-5bb554b03d2f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-439989-29890380-rf2nf" Oct 31 05:00:00.659382 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:00.657886 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x92tm\" (UniqueName: \"kubernetes.io/projected/7212f966-42ad-4ed2-8e45-fde6a7908c2c-kube-api-access-x92tm\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-c9707f-29890380-k76x2\" (UID: \"7212f966-42ad-4ed2-8e45-fde6a7908c2c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c9707f-29890380-k76x2" Oct 31 05:00:00.659382 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:00.657931 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7212f966-42ad-4ed2-8e45-fde6a7908c2c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-c9707f-29890380-k76x2\" (UID: \"7212f966-42ad-4ed2-8e45-fde6a7908c2c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c9707f-29890380-k76x2" Oct 31 05:00:00.659382 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:00.657974 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24pk8\" (UniqueName: \"kubernetes.io/projected/8dfadf3e-94d3-4e77-bd97-5bb554b03d2f-kube-api-access-24pk8\") pod \"maas-api-key-cleanup-e2e-worker-w1-439989-29890380-rf2nf\" (UID: \"8dfadf3e-94d3-4e77-bd97-5bb554b03d2f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-439989-29890380-rf2nf" Oct 31 05:00:00.659382 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:00.658039 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b953e1c-6983-456f-b2c8-77e9314aa758-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-829ac7-29890380-pt7nf\" (UID: \"7b953e1c-6983-456f-b2c8-77e9314aa758\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-829ac7-29890380-pt7nf" Oct 31 05:00:00.659382 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:00.658122 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09f6c274-8499-4b06-91af-6af516c3e56a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-77a153-29890380-sdd6b\" (UID: \"09f6c274-8499-4b06-91af-6af516c3e56a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-77a153-29890380-sdd6b" Oct 31 05:00:00.659382 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:00.658150 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcbr5\" (UniqueName: \"kubernetes.io/projected/9850fba4-11d3-4785-adc0-5ddd89293842-kube-api-access-xcbr5\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-766728-29890380-9xphp\" (UID: \"9850fba4-11d3-4785-adc0-5ddd89293842\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-766728-29890380-9xphp" Oct 31 05:00:00.659382 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:00.659367 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9850fba4-11d3-4785-adc0-5ddd89293842-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-766728-29890380-9xphp\" (UID: \"9850fba4-11d3-4785-adc0-5ddd89293842\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-766728-29890380-9xphp" Oct 31 05:00:00.660587 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:00.660111 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f6a15d6-d47f-47de-817f-fc70feb236e8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-ded435-29890380-bwm8n\" (UID: \"5f6a15d6-d47f-47de-817f-fc70feb236e8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ded435-29890380-bwm8n" Oct 31 05:00:00.660809 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:00.660775 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ecbbd44a-efa7-4e45-8514-1a57eccc71e5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-9633f0-29890380-jc5lw\" (UID: \"ecbbd44a-efa7-4e45-8514-1a57eccc71e5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9633f0-29890380-jc5lw" Oct 31 05:00:00.671418 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:00.664530 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8dfadf3e-94d3-4e77-bd97-5bb554b03d2f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-439989-29890380-rf2nf\" (UID: \"8dfadf3e-94d3-4e77-bd97-5bb554b03d2f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-439989-29890380-rf2nf" Oct 31 05:00:00.689103 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:00.689064 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-12c0bc-29890380-wbz6z" Oct 31 05:00:00.690726 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:00.690208 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-829ac7-29890380-pt7nf"] Oct 31 05:00:00.695674 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:00.695584 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcbr5\" (UniqueName: \"kubernetes.io/projected/9850fba4-11d3-4785-adc0-5ddd89293842-kube-api-access-xcbr5\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-766728-29890380-9xphp\" (UID: \"9850fba4-11d3-4785-adc0-5ddd89293842\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-766728-29890380-9xphp" Oct 31 05:00:00.696472 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:00.696204 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t97bc\" (UniqueName: \"kubernetes.io/projected/5f6a15d6-d47f-47de-817f-fc70feb236e8-kube-api-access-t97bc\") pod \"maas-api-key-cleanup-e2e-extmod-b-ded435-29890380-bwm8n\" (UID: \"5f6a15d6-d47f-47de-817f-fc70feb236e8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ded435-29890380-bwm8n" Oct 31 05:00:00.696472 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:00.696416 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24pk8\" (UniqueName: \"kubernetes.io/projected/8dfadf3e-94d3-4e77-bd97-5bb554b03d2f-kube-api-access-24pk8\") pod \"maas-api-key-cleanup-e2e-worker-w1-439989-29890380-rf2nf\" (UID: \"8dfadf3e-94d3-4e77-bd97-5bb554b03d2f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-439989-29890380-rf2nf" Oct 31 05:00:00.697703 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:00.697245 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2l466\" (UniqueName: \"kubernetes.io/projected/ecbbd44a-efa7-4e45-8514-1a57eccc71e5-kube-api-access-2l466\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-9633f0-29890380-jc5lw\" (UID: \"ecbbd44a-efa7-4e45-8514-1a57eccc71e5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9633f0-29890380-jc5lw" Oct 31 05:00:00.700177 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:00.700155 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-171edd-29890380-2kgrc"] Oct 31 05:00:00.706049 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:00.706024 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ded435-29890380-bwm8n" Oct 31 05:00:00.761365 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:00.760641 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwm5b\" (UniqueName: \"kubernetes.io/projected/79b8c49d-28ac-49fa-b101-5f553fdac035-kube-api-access-jwm5b\") pod \"maas-api-key-cleanup-e2e-inf-b-c1fe26-29890380-h89x7\" (UID: \"79b8c49d-28ac-49fa-b101-5f553fdac035\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c1fe26-29890380-h89x7" Oct 31 05:00:00.761365 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:00.760702 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x92tm\" (UniqueName: \"kubernetes.io/projected/7212f966-42ad-4ed2-8e45-fde6a7908c2c-kube-api-access-x92tm\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-c9707f-29890380-k76x2\" (UID: \"7212f966-42ad-4ed2-8e45-fde6a7908c2c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c9707f-29890380-k76x2" Oct 31 05:00:00.761365 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:00.760741 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ae817be-98c3-4760-b1ed-b545d5b24e0f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-171edd-29890380-2kgrc\" (UID: \"2ae817be-98c3-4760-b1ed-b545d5b24e0f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-171edd-29890380-2kgrc" Oct 31 05:00:00.761365 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:00.760787 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7212f966-42ad-4ed2-8e45-fde6a7908c2c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-c9707f-29890380-k76x2\" (UID: \"7212f966-42ad-4ed2-8e45-fde6a7908c2c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c9707f-29890380-k76x2" Oct 31 05:00:00.761365 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:00.760822 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9853a85-21be-4351-a76c-78d4da0aa7dc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-04d111-29890380-62zcn\" (UID: \"a9853a85-21be-4351-a76c-78d4da0aa7dc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-04d111-29890380-62zcn" Oct 31 05:00:00.761365 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:00.760890 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxbxx\" (UniqueName: \"kubernetes.io/projected/a9853a85-21be-4351-a76c-78d4da0aa7dc-kube-api-access-hxbxx\") pod \"maas-api-key-cleanup-e2e-inf-a-04d111-29890380-62zcn\" (UID: \"a9853a85-21be-4351-a76c-78d4da0aa7dc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-04d111-29890380-62zcn" Oct 31 05:00:00.761365 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:00.760929 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b953e1c-6983-456f-b2c8-77e9314aa758-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-829ac7-29890380-pt7nf\" (UID: \"7b953e1c-6983-456f-b2c8-77e9314aa758\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-829ac7-29890380-pt7nf" Oct 31 05:00:00.761365 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:00.761008 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09f6c274-8499-4b06-91af-6af516c3e56a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-77a153-29890380-sdd6b\" (UID: \"09f6c274-8499-4b06-91af-6af516c3e56a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-77a153-29890380-sdd6b" Oct 31 05:00:00.761365 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:00.761085 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6f4r6\" (UniqueName: \"kubernetes.io/projected/978b4c58-1c25-4e04-bfbf-5cd4e0a7001d-kube-api-access-6f4r6\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-d6ac82-29890380-xtwh8\" (UID: \"978b4c58-1c25-4e04-bfbf-5cd4e0a7001d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d6ac82-29890380-xtwh8" Oct 31 05:00:00.761365 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:00.761140 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jc96x\" (UniqueName: \"kubernetes.io/projected/7b953e1c-6983-456f-b2c8-77e9314aa758-kube-api-access-jc96x\") pod \"maas-api-key-cleanup-e2e-inf-b-829ac7-29890380-pt7nf\" (UID: \"7b953e1c-6983-456f-b2c8-77e9314aa758\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-829ac7-29890380-pt7nf" Oct 31 05:00:00.761365 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:00.761198 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhg2z\" (UniqueName: \"kubernetes.io/projected/09f6c274-8499-4b06-91af-6af516c3e56a-kube-api-access-mhg2z\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-77a153-29890380-sdd6b\" (UID: \"09f6c274-8499-4b06-91af-6af516c3e56a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-77a153-29890380-sdd6b" Oct 31 05:00:00.761365 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:00.761241 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/978b4c58-1c25-4e04-bfbf-5cd4e0a7001d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-d6ac82-29890380-xtwh8\" (UID: \"978b4c58-1c25-4e04-bfbf-5cd4e0a7001d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d6ac82-29890380-xtwh8" Oct 31 05:00:00.762124 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:00.761986 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09f6c274-8499-4b06-91af-6af516c3e56a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-77a153-29890380-sdd6b\" (UID: \"09f6c274-8499-4b06-91af-6af516c3e56a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-77a153-29890380-sdd6b" Oct 31 05:00:00.762341 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:00.762308 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/978b4c58-1c25-4e04-bfbf-5cd4e0a7001d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-d6ac82-29890380-xtwh8\" (UID: \"978b4c58-1c25-4e04-bfbf-5cd4e0a7001d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d6ac82-29890380-xtwh8" Oct 31 05:00:00.763032 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:00.763003 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b953e1c-6983-456f-b2c8-77e9314aa758-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-829ac7-29890380-pt7nf\" (UID: \"7b953e1c-6983-456f-b2c8-77e9314aa758\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-829ac7-29890380-pt7nf" Oct 31 05:00:00.763438 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:00.763414 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79b8c49d-28ac-49fa-b101-5f553fdac035-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-c1fe26-29890380-h89x7\" (UID: \"79b8c49d-28ac-49fa-b101-5f553fdac035\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c1fe26-29890380-h89x7" Oct 31 05:00:00.763536 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:00.763504 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmgwj\" (UniqueName: \"kubernetes.io/projected/2ae817be-98c3-4760-b1ed-b545d5b24e0f-kube-api-access-lmgwj\") pod \"maas-api-key-cleanup-e2e-inf-a-171edd-29890380-2kgrc\" (UID: \"2ae817be-98c3-4760-b1ed-b545d5b24e0f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-171edd-29890380-2kgrc" Oct 31 05:00:00.764548 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:00.764523 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79b8c49d-28ac-49fa-b101-5f553fdac035-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-c1fe26-29890380-h89x7\" (UID: \"79b8c49d-28ac-49fa-b101-5f553fdac035\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c1fe26-29890380-h89x7" Oct 31 05:00:00.765586 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:00.765548 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7212f966-42ad-4ed2-8e45-fde6a7908c2c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-c9707f-29890380-k76x2\" (UID: \"7212f966-42ad-4ed2-8e45-fde6a7908c2c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c9707f-29890380-k76x2" Oct 31 05:00:00.765732 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:00.765607 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9633f0-29890380-jc5lw" Oct 31 05:00:00.785279 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:00.785247 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhg2z\" (UniqueName: \"kubernetes.io/projected/09f6c274-8499-4b06-91af-6af516c3e56a-kube-api-access-mhg2z\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-77a153-29890380-sdd6b\" (UID: \"09f6c274-8499-4b06-91af-6af516c3e56a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-77a153-29890380-sdd6b" Oct 31 05:00:00.787636 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:00.787391 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-439989-29890380-rf2nf" Oct 31 05:00:00.788533 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:00.788455 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6f4r6\" (UniqueName: \"kubernetes.io/projected/978b4c58-1c25-4e04-bfbf-5cd4e0a7001d-kube-api-access-6f4r6\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-d6ac82-29890380-xtwh8\" (UID: \"978b4c58-1c25-4e04-bfbf-5cd4e0a7001d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d6ac82-29890380-xtwh8" Oct 31 05:00:00.794278 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:00.791967 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwm5b\" (UniqueName: \"kubernetes.io/projected/79b8c49d-28ac-49fa-b101-5f553fdac035-kube-api-access-jwm5b\") pod \"maas-api-key-cleanup-e2e-inf-b-c1fe26-29890380-h89x7\" (UID: \"79b8c49d-28ac-49fa-b101-5f553fdac035\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c1fe26-29890380-h89x7" Oct 31 05:00:00.794278 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:00.793195 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x92tm\" (UniqueName: \"kubernetes.io/projected/7212f966-42ad-4ed2-8e45-fde6a7908c2c-kube-api-access-x92tm\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-c9707f-29890380-k76x2\" (UID: \"7212f966-42ad-4ed2-8e45-fde6a7908c2c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c9707f-29890380-k76x2" Oct 31 05:00:00.800713 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:00.800248 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-766728-29890380-9xphp" Oct 31 05:00:00.803039 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:00.802986 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jc96x\" (UniqueName: \"kubernetes.io/projected/7b953e1c-6983-456f-b2c8-77e9314aa758-kube-api-access-jc96x\") pod \"maas-api-key-cleanup-e2e-inf-b-829ac7-29890380-pt7nf\" (UID: \"7b953e1c-6983-456f-b2c8-77e9314aa758\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-829ac7-29890380-pt7nf" Oct 31 05:00:00.829076 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:00.828438 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d6ac82-29890380-xtwh8" Oct 31 05:00:00.865439 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:00.865405 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmgwj\" (UniqueName: \"kubernetes.io/projected/2ae817be-98c3-4760-b1ed-b545d5b24e0f-kube-api-access-lmgwj\") pod \"maas-api-key-cleanup-e2e-inf-a-171edd-29890380-2kgrc\" (UID: \"2ae817be-98c3-4760-b1ed-b545d5b24e0f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-171edd-29890380-2kgrc" Oct 31 05:00:00.865592 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:00.865547 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ae817be-98c3-4760-b1ed-b545d5b24e0f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-171edd-29890380-2kgrc\" (UID: \"2ae817be-98c3-4760-b1ed-b545d5b24e0f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-171edd-29890380-2kgrc" Oct 31 05:00:00.865691 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:00.865673 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9853a85-21be-4351-a76c-78d4da0aa7dc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-04d111-29890380-62zcn\" (UID: \"a9853a85-21be-4351-a76c-78d4da0aa7dc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-04d111-29890380-62zcn" Oct 31 05:00:00.865776 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:00.865761 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxbxx\" (UniqueName: \"kubernetes.io/projected/a9853a85-21be-4351-a76c-78d4da0aa7dc-kube-api-access-hxbxx\") pod \"maas-api-key-cleanup-e2e-inf-a-04d111-29890380-62zcn\" (UID: \"a9853a85-21be-4351-a76c-78d4da0aa7dc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-04d111-29890380-62zcn" Oct 31 05:00:00.868009 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:00.867957 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ae817be-98c3-4760-b1ed-b545d5b24e0f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-171edd-29890380-2kgrc\" (UID: \"2ae817be-98c3-4760-b1ed-b545d5b24e0f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-171edd-29890380-2kgrc" Oct 31 05:00:00.869311 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:00.869228 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9853a85-21be-4351-a76c-78d4da0aa7dc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-04d111-29890380-62zcn\" (UID: \"a9853a85-21be-4351-a76c-78d4da0aa7dc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-04d111-29890380-62zcn" Oct 31 05:00:00.871430 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:00.871402 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-77a153-29890380-sdd6b" Oct 31 05:00:00.892974 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:00.891618 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxbxx\" (UniqueName: \"kubernetes.io/projected/a9853a85-21be-4351-a76c-78d4da0aa7dc-kube-api-access-hxbxx\") pod \"maas-api-key-cleanup-e2e-inf-a-04d111-29890380-62zcn\" (UID: \"a9853a85-21be-4351-a76c-78d4da0aa7dc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-04d111-29890380-62zcn" Oct 31 05:00:00.907161 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:00.907132 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmgwj\" (UniqueName: \"kubernetes.io/projected/2ae817be-98c3-4760-b1ed-b545d5b24e0f-kube-api-access-lmgwj\") pod \"maas-api-key-cleanup-e2e-inf-a-171edd-29890380-2kgrc\" (UID: \"2ae817be-98c3-4760-b1ed-b545d5b24e0f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-171edd-29890380-2kgrc" Oct 31 05:00:00.959150 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:00.959115 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-829ac7-29890380-pt7nf" Oct 31 05:00:01.053905 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:01.053870 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c1fe26-29890380-h89x7" Oct 31 05:00:01.074724 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:01.074531 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c9707f-29890380-k76x2" Oct 31 05:00:01.104113 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:01.101410 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-04d111-29890380-62zcn" Oct 31 05:00:01.152686 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:01.152227 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-171edd-29890380-2kgrc" Oct 31 05:00:01.210136 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:01.210046 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-04d111/test-model-04d111-kserve-5d676b5997-f7fqj" Oct 31 05:00:01.235393 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:01.234702 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-12c0bc-openshift-default-84bddf978c-ghhlp" event={"ID":"7bde4a1c-a591-4864-8276-2c3a74b99e70","Type":"ContainerStarted","Data":"26280a9a05b01397f9ac031363d2028b594ef6e4fd2a7cd1456cbf43dbb55b18"} Oct 31 05:00:01.251349 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:01.238093 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-04d111/test-model-04d111-kserve-5d676b5997-f7fqj" Oct 31 05:00:01.257626 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:01.257601 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-12c0bc-29890380-wbz6z"] Oct 31 05:00:01.261195 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:01.261136 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-04d111/test-model-04d111-kserve-5d676b5997-f7fqj" Oct 31 05:00:01.469891 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:01.469841 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ded435-29890380-bwm8n"] Oct 31 05:00:01.589609 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:01.589590 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9633f0-29890380-jc5lw"] Oct 31 05:00:01.601314 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:01.601286 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-439989-29890380-rf2nf"] Oct 31 05:00:01.602826 ip-10-0-134-155 kubenswrapper[4376]: W1031 05:00:01.602790 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8dfadf3e_94d3_4e77_bd97_5bb554b03d2f.slice/crio-c167f93d9612641d6dc9934fe1f9dc370244247255bb7b9251557dd3e773a396 WatchSource:0}: Error finding container c167f93d9612641d6dc9934fe1f9dc370244247255bb7b9251557dd3e773a396: Status 404 returned error can't find the container with id c167f93d9612641d6dc9934fe1f9dc370244247255bb7b9251557dd3e773a396 Oct 31 05:00:01.683256 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:01.683177 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d6ac82-29890380-xtwh8"] Oct 31 05:00:01.693955 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:01.693858 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-77a153-29890380-sdd6b"] Oct 31 05:00:01.700829 ip-10-0-134-155 kubenswrapper[4376]: W1031 05:00:01.699432 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9850fba4_11d3_4785_adc0_5ddd89293842.slice/crio-ded170c2b38c7241da47302b565dbf738496cb6720aab7dc89bc4ec7bf7f5329 WatchSource:0}: Error finding container ded170c2b38c7241da47302b565dbf738496cb6720aab7dc89bc4ec7bf7f5329: Status 404 returned error can't find the container with id ded170c2b38c7241da47302b565dbf738496cb6720aab7dc89bc4ec7bf7f5329 Oct 31 05:00:01.700829 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:01.700689 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-12c0bc-openshift-default-84bddf978c-ghhlp" Oct 31 05:00:01.701072 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:01.700991 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-766728-29890380-9xphp"] Oct 31 05:00:01.702422 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:01.702379 4376 patch_prober.go:29] interesting pod/e2e-inf-b-12c0bc-openshift-default-84bddf978c-ghhlp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" start-of-body= Oct 31 05:00:01.702533 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:01.702425 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-12c0bc-openshift-default-84bddf978c-ghhlp" podUID="7bde4a1c-a591-4864-8276-2c3a74b99e70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" Oct 31 05:00:01.779775 ip-10-0-134-155 kubenswrapper[4376]: W1031 05:00:01.779748 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ae817be_98c3_4760_b1ed_b545d5b24e0f.slice/crio-92655d5a98f23bedabe4a5a9b36f942850945d9ad291e9845fe8085e0ba1aaf2 WatchSource:0}: Error finding container 92655d5a98f23bedabe4a5a9b36f942850945d9ad291e9845fe8085e0ba1aaf2: Status 404 returned error can't find the container with id 92655d5a98f23bedabe4a5a9b36f942850945d9ad291e9845fe8085e0ba1aaf2 Oct 31 05:00:01.780377 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:01.780351 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-171edd-29890380-2kgrc"] Oct 31 05:00:01.809773 ip-10-0-134-155 kubenswrapper[4376]: W1031 05:00:01.809728 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b953e1c_6983_456f_b2c8_77e9314aa758.slice/crio-437bacf525ad3a1a955c7599915ddab8d812739f5b9560197fe7df432d113641 WatchSource:0}: Error finding container 437bacf525ad3a1a955c7599915ddab8d812739f5b9560197fe7df432d113641: Status 404 returned error can't find the container with id 437bacf525ad3a1a955c7599915ddab8d812739f5b9560197fe7df432d113641 Oct 31 05:00:01.810768 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:01.810743 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-829ac7-29890380-pt7nf"] Oct 31 05:00:01.826135 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:01.826114 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c9707f-29890380-k76x2"] Oct 31 05:00:01.876882 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:01.876856 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c1fe26-29890380-h89x7"] Oct 31 05:00:01.893120 ip-10-0-134-155 kubenswrapper[4376]: W1031 05:00:01.891112 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79b8c49d_28ac_49fa_b101_5f553fdac035.slice/crio-71d0900ca404e25e4fae7a6fba144d464879d17234f7551e751b5201bba0f151 WatchSource:0}: Error finding container 71d0900ca404e25e4fae7a6fba144d464879d17234f7551e751b5201bba0f151: Status 404 returned error can't find the container with id 71d0900ca404e25e4fae7a6fba144d464879d17234f7551e751b5201bba0f151 Oct 31 05:00:01.918193 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:01.918141 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-04d111-29890380-62zcn"] Oct 31 05:00:01.918417 ip-10-0-134-155 kubenswrapper[4376]: W1031 05:00:01.918392 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9853a85_21be_4351_a76c_78d4da0aa7dc.slice/crio-2b98b774a0a4d82c7af9e6d6675de8998399edb04ff95c3803c983b101b8e387 WatchSource:0}: Error finding container 2b98b774a0a4d82c7af9e6d6675de8998399edb04ff95c3803c983b101b8e387: Status 404 returned error can't find the container with id 2b98b774a0a4d82c7af9e6d6675de8998399edb04ff95c3803c983b101b8e387 Oct 31 05:00:02.276754 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:02.276722 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-12c0bc-29890380-wbz6z" event={"ID":"a63f4216-a71c-45b7-98b9-3b08500503fd","Type":"ContainerStarted","Data":"fec84ecc7792840cdbde9268d7936f283da1c585e4e022bc8ea09a4c5e365787"} Oct 31 05:00:02.277365 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:02.276787 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-12c0bc-29890380-wbz6z" event={"ID":"a63f4216-a71c-45b7-98b9-3b08500503fd","Type":"ContainerStarted","Data":"07f8cfa56c5119bd07577377d19e69885171eb1ae2c3fa6811176cada45a4e53"} Oct 31 05:00:02.283197 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:02.281963 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c9707f-29890380-k76x2" event={"ID":"7212f966-42ad-4ed2-8e45-fde6a7908c2c","Type":"ContainerStarted","Data":"28e9dd2c9c71cf02e148ddddd32d82a6847065ad8a9104f5766f968a615b3e1f"} Oct 31 05:00:02.283502 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:02.283478 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c1fe26-29890380-h89x7" event={"ID":"79b8c49d-28ac-49fa-b101-5f553fdac035","Type":"ContainerStarted","Data":"71d0900ca404e25e4fae7a6fba144d464879d17234f7551e751b5201bba0f151"} Oct 31 05:00:02.285291 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:02.285265 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-766728-29890380-9xphp" event={"ID":"9850fba4-11d3-4785-adc0-5ddd89293842","Type":"ContainerStarted","Data":"ded170c2b38c7241da47302b565dbf738496cb6720aab7dc89bc4ec7bf7f5329"} Oct 31 05:00:02.287262 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:02.287243 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9633f0-29890380-jc5lw" event={"ID":"ecbbd44a-efa7-4e45-8514-1a57eccc71e5","Type":"ContainerStarted","Data":"05473bb79b6773f5ad44da92a8b7f7aa4d9b5f019917d60493de819a01c73c78"} Oct 31 05:00:02.288728 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:02.288700 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-04d111-29890380-62zcn" event={"ID":"a9853a85-21be-4351-a76c-78d4da0aa7dc","Type":"ContainerStarted","Data":"2b98b774a0a4d82c7af9e6d6675de8998399edb04ff95c3803c983b101b8e387"} Oct 31 05:00:02.290073 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:02.290051 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-829ac7-29890380-pt7nf" event={"ID":"7b953e1c-6983-456f-b2c8-77e9314aa758","Type":"ContainerStarted","Data":"437bacf525ad3a1a955c7599915ddab8d812739f5b9560197fe7df432d113641"} Oct 31 05:00:02.291376 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:02.291352 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-77a153-29890380-sdd6b" event={"ID":"09f6c274-8499-4b06-91af-6af516c3e56a","Type":"ContainerStarted","Data":"d8e9f787ab252b262c82b223267f6c9c215088a13e7c927b5e86748e29c057b0"} Oct 31 05:00:02.292842 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:02.292817 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ded435-29890380-bwm8n" event={"ID":"5f6a15d6-d47f-47de-817f-fc70feb236e8","Type":"ContainerStarted","Data":"cbf53e770eb73a968206c047a4fe66abfaf6eeafa7461bde7c3202a933ea03fb"} Oct 31 05:00:02.298360 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:02.296148 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d6ac82-29890380-xtwh8" event={"ID":"978b4c58-1c25-4e04-bfbf-5cd4e0a7001d","Type":"ContainerStarted","Data":"43a74639293e623020d57c652c21ba1f7076b06dd12d76a4dac4a1ec32df3263"} Oct 31 05:00:02.298360 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:02.297830 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-439989-29890380-rf2nf" event={"ID":"8dfadf3e-94d3-4e77-bd97-5bb554b03d2f","Type":"ContainerStarted","Data":"c167f93d9612641d6dc9934fe1f9dc370244247255bb7b9251557dd3e773a396"} Oct 31 05:00:02.300314 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:02.300290 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-171edd-29890380-2kgrc" event={"ID":"2ae817be-98c3-4760-b1ed-b545d5b24e0f","Type":"ContainerStarted","Data":"92655d5a98f23bedabe4a5a9b36f942850945d9ad291e9845fe8085e0ba1aaf2"} Oct 31 05:00:02.320993 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:02.320952 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-12c0bc-29890380-wbz6z" podStartSLOduration=2.3209401610000002 podStartE2EDuration="2.320940161s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 05:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 05:00:02.31760833 +0000 UTC m=+3350.031942947" watchObservedRunningTime="2026-10-31 05:00:02.320940161 +0000 UTC m=+3350.035274782" Oct 31 05:00:02.725666 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:02.720578 4376 patch_prober.go:29] interesting pod/e2e-inf-b-12c0bc-openshift-default-84bddf978c-ghhlp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" start-of-body= Oct 31 05:00:02.725666 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:02.720625 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-12c0bc-openshift-default-84bddf978c-ghhlp" podUID="7bde4a1c-a591-4864-8276-2c3a74b99e70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" Oct 31 05:00:03.313866 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:03.313729 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c1fe26-29890380-h89x7" event={"ID":"79b8c49d-28ac-49fa-b101-5f553fdac035","Type":"ContainerStarted","Data":"4248202aedbc4d28b179949e4983081e2f9f916297de6b0d739b9d142ef97293"} Oct 31 05:00:03.315443 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:03.315420 4376 generic.go:356] "Generic (PLEG): container finished" podID="9850fba4-11d3-4785-adc0-5ddd89293842" containerID="153491c73707eaad9dd41d86610d7bbf4510c83a01003ea187b0977434ea748a" exitCode=60 Oct 31 05:00:03.315533 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:03.315479 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-766728-29890380-9xphp" event={"ID":"9850fba4-11d3-4785-adc0-5ddd89293842","Type":"ContainerDied","Data":"153491c73707eaad9dd41d86610d7bbf4510c83a01003ea187b0977434ea748a"} Oct 31 05:00:03.315812 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:03.315797 4376 scope.go:122] "RemoveContainer" containerID="153491c73707eaad9dd41d86610d7bbf4510c83a01003ea187b0977434ea748a" Oct 31 05:00:03.319203 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:03.319178 4376 generic.go:356] "Generic (PLEG): container finished" podID="ecbbd44a-efa7-4e45-8514-1a57eccc71e5" containerID="7a673d19e73dd82962f0ad83d7d360609210973b5a1339eceb9a7db50c12b40b" exitCode=60 Oct 31 05:00:03.319309 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:03.319280 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9633f0-29890380-jc5lw" event={"ID":"ecbbd44a-efa7-4e45-8514-1a57eccc71e5","Type":"ContainerDied","Data":"7a673d19e73dd82962f0ad83d7d360609210973b5a1339eceb9a7db50c12b40b"} Oct 31 05:00:03.319683 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:03.319551 4376 scope.go:122] "RemoveContainer" containerID="7a673d19e73dd82962f0ad83d7d360609210973b5a1339eceb9a7db50c12b40b" Oct 31 05:00:03.321726 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:03.321707 4376 generic.go:356] "Generic (PLEG): container finished" podID="a9853a85-21be-4351-a76c-78d4da0aa7dc" containerID="2ae3b8ad5076ba18377d54ad490a6c78b8566dc790523b2d419962409b0e958a" exitCode=60 Oct 31 05:00:03.321822 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:03.321783 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-04d111-29890380-62zcn" event={"ID":"a9853a85-21be-4351-a76c-78d4da0aa7dc","Type":"ContainerDied","Data":"2ae3b8ad5076ba18377d54ad490a6c78b8566dc790523b2d419962409b0e958a"} Oct 31 05:00:03.322003 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:03.321986 4376 scope.go:122] "RemoveContainer" containerID="2ae3b8ad5076ba18377d54ad490a6c78b8566dc790523b2d419962409b0e958a" Oct 31 05:00:03.323530 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:03.323508 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-829ac7-29890380-pt7nf" event={"ID":"7b953e1c-6983-456f-b2c8-77e9314aa758","Type":"ContainerStarted","Data":"daf4da1298f8454e71fedd8a3b0b10319fc0b3e4250c2c39cadbb4a179dd493f"} Oct 31 05:00:03.325146 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:03.325127 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-77a153-29890380-sdd6b" event={"ID":"09f6c274-8499-4b06-91af-6af516c3e56a","Type":"ContainerStarted","Data":"7d9d54b5c54c0e42e925f9d5bdfdd5e379e454643722b73b8a12f7fd4dec1ddb"} Oct 31 05:00:03.326756 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:03.326735 4376 generic.go:356] "Generic (PLEG): container finished" podID="5f6a15d6-d47f-47de-817f-fc70feb236e8" containerID="9f2ec7678c989043357c4c48defabab3112f445e5bbdc02330d11d87f3f9bff2" exitCode=60 Oct 31 05:00:03.326840 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:03.326800 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ded435-29890380-bwm8n" event={"ID":"5f6a15d6-d47f-47de-817f-fc70feb236e8","Type":"ContainerDied","Data":"9f2ec7678c989043357c4c48defabab3112f445e5bbdc02330d11d87f3f9bff2"} Oct 31 05:00:03.327259 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:03.326967 4376 scope.go:122] "RemoveContainer" containerID="9f2ec7678c989043357c4c48defabab3112f445e5bbdc02330d11d87f3f9bff2" Oct 31 05:00:03.328678 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:03.328660 4376 generic.go:356] "Generic (PLEG): container finished" podID="978b4c58-1c25-4e04-bfbf-5cd4e0a7001d" containerID="4e78cae8bdb304d97a0f7f4a085f26667c402115141a44776be279f271dc2e41" exitCode=60 Oct 31 05:00:03.328787 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:03.328727 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d6ac82-29890380-xtwh8" event={"ID":"978b4c58-1c25-4e04-bfbf-5cd4e0a7001d","Type":"ContainerDied","Data":"4e78cae8bdb304d97a0f7f4a085f26667c402115141a44776be279f271dc2e41"} Oct 31 05:00:03.328949 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:03.328932 4376 scope.go:122] "RemoveContainer" containerID="4e78cae8bdb304d97a0f7f4a085f26667c402115141a44776be279f271dc2e41" Oct 31 05:00:03.334059 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:03.334040 4376 generic.go:356] "Generic (PLEG): container finished" podID="8dfadf3e-94d3-4e77-bd97-5bb554b03d2f" containerID="fb1a95f99de3101b566bc0b567ca9c525cf1d0c1fe95181694f80c891386abbd" exitCode=60 Oct 31 05:00:03.334148 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:03.334133 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-439989-29890380-rf2nf" event={"ID":"8dfadf3e-94d3-4e77-bd97-5bb554b03d2f","Type":"ContainerDied","Data":"fb1a95f99de3101b566bc0b567ca9c525cf1d0c1fe95181694f80c891386abbd"} Oct 31 05:00:03.334341 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:03.334310 4376 scope.go:122] "RemoveContainer" containerID="fb1a95f99de3101b566bc0b567ca9c525cf1d0c1fe95181694f80c891386abbd" Oct 31 05:00:03.335912 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:03.335892 4376 generic.go:356] "Generic (PLEG): container finished" podID="2ae817be-98c3-4760-b1ed-b545d5b24e0f" containerID="852c252116984ecbf43ede42af636c4dc70a0bac6517531fa7de72fe19a2446e" exitCode=60 Oct 31 05:00:03.335984 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:03.335963 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-171edd-29890380-2kgrc" event={"ID":"2ae817be-98c3-4760-b1ed-b545d5b24e0f","Type":"ContainerDied","Data":"852c252116984ecbf43ede42af636c4dc70a0bac6517531fa7de72fe19a2446e"} Oct 31 05:00:03.336159 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:03.336147 4376 scope.go:122] "RemoveContainer" containerID="852c252116984ecbf43ede42af636c4dc70a0bac6517531fa7de72fe19a2446e" Oct 31 05:00:03.368374 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:03.367797 4376 generic.go:356] "Generic (PLEG): container finished" podID="a63f4216-a71c-45b7-98b9-3b08500503fd" containerID="fec84ecc7792840cdbde9268d7936f283da1c585e4e022bc8ea09a4c5e365787" exitCode=60 Oct 31 05:00:03.368374 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:03.367912 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-12c0bc-29890380-wbz6z" event={"ID":"a63f4216-a71c-45b7-98b9-3b08500503fd","Type":"ContainerDied","Data":"fec84ecc7792840cdbde9268d7936f283da1c585e4e022bc8ea09a4c5e365787"} Oct 31 05:00:03.368727 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:03.368672 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c1fe26-29890380-h89x7" podStartSLOduration=3.368653875 podStartE2EDuration="3.368653875s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 05:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 05:00:03.365080513 +0000 UTC m=+3351.079415132" watchObservedRunningTime="2026-10-31 05:00:03.368653875 +0000 UTC m=+3351.082988495" Oct 31 05:00:03.371873 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:03.371856 4376 scope.go:122] "RemoveContainer" containerID="fec84ecc7792840cdbde9268d7936f283da1c585e4e022bc8ea09a4c5e365787" Oct 31 05:00:03.375146 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:03.375129 4376 generic.go:356] "Generic (PLEG): container finished" podID="7212f966-42ad-4ed2-8e45-fde6a7908c2c" containerID="228a2c860510a387940647d08197c45fa498f5a0be61336359f6de04d2ae594a" exitCode=60 Oct 31 05:00:03.375625 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:03.375354 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c9707f-29890380-k76x2" event={"ID":"7212f966-42ad-4ed2-8e45-fde6a7908c2c","Type":"ContainerDied","Data":"228a2c860510a387940647d08197c45fa498f5a0be61336359f6de04d2ae594a"} Oct 31 05:00:03.376131 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:03.375560 4376 scope.go:122] "RemoveContainer" containerID="228a2c860510a387940647d08197c45fa498f5a0be61336359f6de04d2ae594a" Oct 31 05:00:03.510608 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:03.510562 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-829ac7-29890380-pt7nf" podStartSLOduration=3.510549105 podStartE2EDuration="3.510549105s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 05:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 05:00:03.466603383 +0000 UTC m=+3351.180938001" watchObservedRunningTime="2026-10-31 05:00:03.510549105 +0000 UTC m=+3351.224883721" Oct 31 05:00:03.689931 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:03.689879 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-77a153-29890380-sdd6b" podStartSLOduration=3.689862907 podStartE2EDuration="3.689862907s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 05:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 05:00:03.685384252 +0000 UTC m=+3351.399718887" watchObservedRunningTime="2026-10-31 05:00:03.689862907 +0000 UTC m=+3351.404197547" Oct 31 05:00:03.700380 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:03.700355 4376 patch_prober.go:29] interesting pod/e2e-inf-b-12c0bc-openshift-default-84bddf978c-ghhlp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" start-of-body= Oct 31 05:00:03.700508 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:03.700398 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-12c0bc-openshift-default-84bddf978c-ghhlp" podUID="7bde4a1c-a591-4864-8276-2c3a74b99e70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" Oct 31 05:00:04.390384 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:04.390356 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9633f0-29890380-jc5lw" event={"ID":"ecbbd44a-efa7-4e45-8514-1a57eccc71e5","Type":"ContainerStarted","Data":"fee90b045745163874d3ce8d1d390793bfae79163cd8ac97c1b1db79e3ae3ac1"} Oct 31 05:00:04.392540 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:04.392515 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-04d111-29890380-62zcn" event={"ID":"a9853a85-21be-4351-a76c-78d4da0aa7dc","Type":"ContainerStarted","Data":"35c8d823dd3b6310e2df3beb04c8a5a85fa0afa80eae041fe933af6dc6e5fe35"} Oct 31 05:00:04.394039 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:04.394017 4376 generic.go:356] "Generic (PLEG): container finished" podID="7b953e1c-6983-456f-b2c8-77e9314aa758" containerID="daf4da1298f8454e71fedd8a3b0b10319fc0b3e4250c2c39cadbb4a179dd493f" exitCode=60 Oct 31 05:00:04.394129 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:04.394083 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-829ac7-29890380-pt7nf" event={"ID":"7b953e1c-6983-456f-b2c8-77e9314aa758","Type":"ContainerDied","Data":"daf4da1298f8454e71fedd8a3b0b10319fc0b3e4250c2c39cadbb4a179dd493f"} Oct 31 05:00:04.394298 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:04.394278 4376 scope.go:122] "RemoveContainer" containerID="daf4da1298f8454e71fedd8a3b0b10319fc0b3e4250c2c39cadbb4a179dd493f" Oct 31 05:00:04.395888 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:04.395849 4376 generic.go:356] "Generic (PLEG): container finished" podID="09f6c274-8499-4b06-91af-6af516c3e56a" containerID="7d9d54b5c54c0e42e925f9d5bdfdd5e379e454643722b73b8a12f7fd4dec1ddb" exitCode=60 Oct 31 05:00:04.395960 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:04.395922 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-77a153-29890380-sdd6b" event={"ID":"09f6c274-8499-4b06-91af-6af516c3e56a","Type":"ContainerDied","Data":"7d9d54b5c54c0e42e925f9d5bdfdd5e379e454643722b73b8a12f7fd4dec1ddb"} Oct 31 05:00:04.396133 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:04.396104 4376 scope.go:122] "RemoveContainer" containerID="7d9d54b5c54c0e42e925f9d5bdfdd5e379e454643722b73b8a12f7fd4dec1ddb" Oct 31 05:00:04.398629 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:04.398323 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ded435-29890380-bwm8n" event={"ID":"5f6a15d6-d47f-47de-817f-fc70feb236e8","Type":"ContainerStarted","Data":"2ab0a38673b93473da115aadc99d88a00e591ef94e8010319b1484825c6e29c0"} Oct 31 05:00:04.403291 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:04.402892 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-439989-29890380-rf2nf" event={"ID":"8dfadf3e-94d3-4e77-bd97-5bb554b03d2f","Type":"ContainerStarted","Data":"a6f0a4b7d76434f4153b9c5ee16256183bb375e4db3b606097ed48e506e8c624"} Oct 31 05:00:04.405790 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:04.405202 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-12c0bc-29890380-wbz6z" event={"ID":"a63f4216-a71c-45b7-98b9-3b08500503fd","Type":"ContainerStarted","Data":"b86f54905df3196f08b381db0e1865d1c2de39d817d6d0fa12dc4c1be2b81083"} Oct 31 05:00:04.408321 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:04.408047 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c9707f-29890380-k76x2" event={"ID":"7212f966-42ad-4ed2-8e45-fde6a7908c2c","Type":"ContainerStarted","Data":"21ece3713eb78e7d7a0838d717578c72cc68a2b95ef29b39e5867eb213b5bccf"} Oct 31 05:00:04.411362 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:04.410572 4376 generic.go:356] "Generic (PLEG): container finished" podID="79b8c49d-28ac-49fa-b101-5f553fdac035" containerID="4248202aedbc4d28b179949e4983081e2f9f916297de6b0d739b9d142ef97293" exitCode=60 Oct 31 05:00:04.411362 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:04.410598 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c1fe26-29890380-h89x7" event={"ID":"79b8c49d-28ac-49fa-b101-5f553fdac035","Type":"ContainerDied","Data":"4248202aedbc4d28b179949e4983081e2f9f916297de6b0d739b9d142ef97293"} Oct 31 05:00:04.411362 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:04.410815 4376 scope.go:122] "RemoveContainer" containerID="4248202aedbc4d28b179949e4983081e2f9f916297de6b0d739b9d142ef97293" Oct 31 05:00:04.431319 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:04.431283 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-766728-29890380-9xphp" podStartSLOduration=4.431270426 podStartE2EDuration="4.431270426s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 05:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 05:00:04.42916212 +0000 UTC m=+3352.143496737" watchObservedRunningTime="2026-10-31 05:00:04.431270426 +0000 UTC m=+3352.145605042" Oct 31 05:00:04.496977 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:04.496916 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d6ac82-29890380-xtwh8" podStartSLOduration=4.49689913 podStartE2EDuration="4.49689913s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 05:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 05:00:04.486283039 +0000 UTC m=+3352.200617657" watchObservedRunningTime="2026-10-31 05:00:04.49689913 +0000 UTC m=+3352.211233748" Oct 31 05:00:04.618502 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:04.618416 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-04d111-29890380-62zcn" podStartSLOduration=4.618403571 podStartE2EDuration="4.618403571s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 05:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 05:00:04.573450638 +0000 UTC m=+3352.287785257" watchObservedRunningTime="2026-10-31 05:00:04.618403571 +0000 UTC m=+3352.332738187" Oct 31 05:00:04.619273 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:04.619241 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-439989-29890380-rf2nf" podStartSLOduration=4.619232352 podStartE2EDuration="4.619232352s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 05:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 05:00:04.615285738 +0000 UTC m=+3352.329620373" watchObservedRunningTime="2026-10-31 05:00:04.619232352 +0000 UTC m=+3352.333566968" Oct 31 05:00:04.733353 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:04.729838 4376 patch_prober.go:29] interesting pod/e2e-inf-b-12c0bc-openshift-default-84bddf978c-ghhlp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" start-of-body= Oct 31 05:00:04.733353 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:04.729899 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-12c0bc-openshift-default-84bddf978c-ghhlp" podUID="7bde4a1c-a591-4864-8276-2c3a74b99e70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" Oct 31 05:00:04.739352 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:04.736145 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ded435-29890380-bwm8n" podStartSLOduration=4.7361257850000005 podStartE2EDuration="4.736125785s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 05:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 05:00:04.713253409 +0000 UTC m=+3352.427588041" watchObservedRunningTime="2026-10-31 05:00:04.736125785 +0000 UTC m=+3352.450460445" Oct 31 05:00:04.877875 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:04.877791 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c9707f-29890380-k76x2" podStartSLOduration=4.877775815 podStartE2EDuration="4.877775815s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 05:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 05:00:04.87448309 +0000 UTC m=+3352.588817708" watchObservedRunningTime="2026-10-31 05:00:04.877775815 +0000 UTC m=+3352.592110431" Oct 31 05:00:04.919978 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:04.919923 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9633f0-29890380-jc5lw" podStartSLOduration=4.919908865 podStartE2EDuration="4.919908865s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 05:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 05:00:04.916516459 +0000 UTC m=+3352.630851076" watchObservedRunningTime="2026-10-31 05:00:04.919908865 +0000 UTC m=+3352.634243528" Oct 31 05:00:05.420829 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:05.420746 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c1fe26-29890380-h89x7" event={"ID":"79b8c49d-28ac-49fa-b101-5f553fdac035","Type":"ContainerStarted","Data":"880aee831dd880e881221efcb7dd61e2322e3fc5e39d9d55f940ddc73d566b9e"} Oct 31 05:00:05.422527 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:05.422503 4376 generic.go:356] "Generic (PLEG): container finished" podID="9850fba4-11d3-4785-adc0-5ddd89293842" containerID="387e7f9074661504d45b68a4748a0ee6ec00865d344dea2ae063710166430d78" exitCode=60 Oct 31 05:00:05.422659 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:05.422556 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-766728-29890380-9xphp" event={"ID":"9850fba4-11d3-4785-adc0-5ddd89293842","Type":"ContainerDied","Data":"387e7f9074661504d45b68a4748a0ee6ec00865d344dea2ae063710166430d78"} Oct 31 05:00:05.422659 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:05.422613 4376 scope.go:122] "RemoveContainer" containerID="153491c73707eaad9dd41d86610d7bbf4510c83a01003ea187b0977434ea748a" Oct 31 05:00:05.423013 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:05.422991 4376 scope.go:122] "RemoveContainer" containerID="387e7f9074661504d45b68a4748a0ee6ec00865d344dea2ae063710166430d78" Oct 31 05:00:05.423368 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:00:05.423345 4376 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-766728-29890380-9xphp_odh-ai-gateway-infra(9850fba4-11d3-4785-adc0-5ddd89293842)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-766728-29890380-9xphp" podUID="9850fba4-11d3-4785-adc0-5ddd89293842" Oct 31 05:00:05.424594 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:05.424575 4376 generic.go:356] "Generic (PLEG): container finished" podID="ecbbd44a-efa7-4e45-8514-1a57eccc71e5" containerID="fee90b045745163874d3ce8d1d390793bfae79163cd8ac97c1b1db79e3ae3ac1" exitCode=60 Oct 31 05:00:05.424675 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:05.424653 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9633f0-29890380-jc5lw" event={"ID":"ecbbd44a-efa7-4e45-8514-1a57eccc71e5","Type":"ContainerDied","Data":"fee90b045745163874d3ce8d1d390793bfae79163cd8ac97c1b1db79e3ae3ac1"} Oct 31 05:00:05.424857 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:05.424842 4376 scope.go:122] "RemoveContainer" containerID="fee90b045745163874d3ce8d1d390793bfae79163cd8ac97c1b1db79e3ae3ac1" Oct 31 05:00:05.425099 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:00:05.425079 4376 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-9633f0-29890380-jc5lw_odh-ai-gateway-infra(ecbbd44a-efa7-4e45-8514-1a57eccc71e5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9633f0-29890380-jc5lw" podUID="ecbbd44a-efa7-4e45-8514-1a57eccc71e5" Oct 31 05:00:05.426655 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:05.426631 4376 generic.go:356] "Generic (PLEG): container finished" podID="a9853a85-21be-4351-a76c-78d4da0aa7dc" containerID="35c8d823dd3b6310e2df3beb04c8a5a85fa0afa80eae041fe933af6dc6e5fe35" exitCode=60 Oct 31 05:00:05.426761 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:05.426741 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-04d111-29890380-62zcn" event={"ID":"a9853a85-21be-4351-a76c-78d4da0aa7dc","Type":"ContainerDied","Data":"35c8d823dd3b6310e2df3beb04c8a5a85fa0afa80eae041fe933af6dc6e5fe35"} Oct 31 05:00:05.427016 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:05.426999 4376 scope.go:122] "RemoveContainer" containerID="35c8d823dd3b6310e2df3beb04c8a5a85fa0afa80eae041fe933af6dc6e5fe35" Oct 31 05:00:05.427310 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:00:05.427285 4376 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-04d111-29890380-62zcn_odh-ai-gateway-infra(a9853a85-21be-4351-a76c-78d4da0aa7dc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-04d111-29890380-62zcn" podUID="a9853a85-21be-4351-a76c-78d4da0aa7dc" Oct 31 05:00:05.428661 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:05.428640 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-829ac7-29890380-pt7nf" event={"ID":"7b953e1c-6983-456f-b2c8-77e9314aa758","Type":"ContainerStarted","Data":"4f771f169661e2b56bc8bd9464e1d7981b6ecfe95c0a168d3a891331786a5a22"} Oct 31 05:00:05.430609 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:05.430590 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-77a153-29890380-sdd6b" event={"ID":"09f6c274-8499-4b06-91af-6af516c3e56a","Type":"ContainerStarted","Data":"d0b81b164bb58e9623a058006211bacddc07fbd537cc3b4d5fa73bfc50f6313c"} Oct 31 05:00:05.432369 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:05.432349 4376 generic.go:356] "Generic (PLEG): container finished" podID="5f6a15d6-d47f-47de-817f-fc70feb236e8" containerID="2ab0a38673b93473da115aadc99d88a00e591ef94e8010319b1484825c6e29c0" exitCode=60 Oct 31 05:00:05.432455 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:05.432374 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ded435-29890380-bwm8n" event={"ID":"5f6a15d6-d47f-47de-817f-fc70feb236e8","Type":"ContainerDied","Data":"2ab0a38673b93473da115aadc99d88a00e591ef94e8010319b1484825c6e29c0"} Oct 31 05:00:05.432651 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:05.432638 4376 scope.go:122] "RemoveContainer" containerID="2ab0a38673b93473da115aadc99d88a00e591ef94e8010319b1484825c6e29c0" Oct 31 05:00:05.432901 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:00:05.432884 4376 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-ded435-29890380-bwm8n_odh-ai-gateway-infra(5f6a15d6-d47f-47de-817f-fc70feb236e8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ded435-29890380-bwm8n" podUID="5f6a15d6-d47f-47de-817f-fc70feb236e8" Oct 31 05:00:05.434294 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:05.434278 4376 generic.go:356] "Generic (PLEG): container finished" podID="978b4c58-1c25-4e04-bfbf-5cd4e0a7001d" containerID="8ae831e41865317107e036d7aa6e1e552a3769eadb8f00869c49636337b377cf" exitCode=60 Oct 31 05:00:05.434405 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:05.434349 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d6ac82-29890380-xtwh8" event={"ID":"978b4c58-1c25-4e04-bfbf-5cd4e0a7001d","Type":"ContainerDied","Data":"8ae831e41865317107e036d7aa6e1e552a3769eadb8f00869c49636337b377cf"} Oct 31 05:00:05.434699 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:05.434684 4376 scope.go:122] "RemoveContainer" containerID="8ae831e41865317107e036d7aa6e1e552a3769eadb8f00869c49636337b377cf" Oct 31 05:00:05.434906 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:00:05.434889 4376 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-d6ac82-29890380-xtwh8_odh-ai-gateway-infra(978b4c58-1c25-4e04-bfbf-5cd4e0a7001d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d6ac82-29890380-xtwh8" podUID="978b4c58-1c25-4e04-bfbf-5cd4e0a7001d" Oct 31 05:00:05.436773 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:05.436753 4376 generic.go:356] "Generic (PLEG): container finished" podID="8dfadf3e-94d3-4e77-bd97-5bb554b03d2f" containerID="a6f0a4b7d76434f4153b9c5ee16256183bb375e4db3b606097ed48e506e8c624" exitCode=60 Oct 31 05:00:05.436873 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:05.436830 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-439989-29890380-rf2nf" event={"ID":"8dfadf3e-94d3-4e77-bd97-5bb554b03d2f","Type":"ContainerDied","Data":"a6f0a4b7d76434f4153b9c5ee16256183bb375e4db3b606097ed48e506e8c624"} Oct 31 05:00:05.437031 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:05.437012 4376 scope.go:122] "RemoveContainer" containerID="a6f0a4b7d76434f4153b9c5ee16256183bb375e4db3b606097ed48e506e8c624" Oct 31 05:00:05.437262 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:00:05.437242 4376 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-439989-29890380-rf2nf_odh-ai-gateway-infra(8dfadf3e-94d3-4e77-bd97-5bb554b03d2f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-439989-29890380-rf2nf" podUID="8dfadf3e-94d3-4e77-bd97-5bb554b03d2f" Oct 31 05:00:05.457352 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:05.454948 4376 generic.go:356] "Generic (PLEG): container finished" podID="2ae817be-98c3-4760-b1ed-b545d5b24e0f" containerID="d8aea9034a23a2093e6c4ed2b140f5323e732d1fa5eb3e718ff76edd45801a2a" exitCode=60 Oct 31 05:00:05.457352 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:05.455040 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-171edd-29890380-2kgrc" event={"ID":"2ae817be-98c3-4760-b1ed-b545d5b24e0f","Type":"ContainerDied","Data":"d8aea9034a23a2093e6c4ed2b140f5323e732d1fa5eb3e718ff76edd45801a2a"} Oct 31 05:00:05.457352 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:05.455491 4376 scope.go:122] "RemoveContainer" containerID="d8aea9034a23a2093e6c4ed2b140f5323e732d1fa5eb3e718ff76edd45801a2a" Oct 31 05:00:05.457352 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:00:05.455849 4376 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-171edd-29890380-2kgrc_odh-ai-gateway-infra(2ae817be-98c3-4760-b1ed-b545d5b24e0f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-171edd-29890380-2kgrc" podUID="2ae817be-98c3-4760-b1ed-b545d5b24e0f" Oct 31 05:00:05.479702 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:05.478383 4376 generic.go:356] "Generic (PLEG): container finished" podID="a63f4216-a71c-45b7-98b9-3b08500503fd" containerID="b86f54905df3196f08b381db0e1865d1c2de39d817d6d0fa12dc4c1be2b81083" exitCode=60 Oct 31 05:00:05.479702 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:05.478388 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-12c0bc-29890380-wbz6z" event={"ID":"a63f4216-a71c-45b7-98b9-3b08500503fd","Type":"ContainerDied","Data":"b86f54905df3196f08b381db0e1865d1c2de39d817d6d0fa12dc4c1be2b81083"} Oct 31 05:00:05.479702 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:05.478792 4376 scope.go:122] "RemoveContainer" containerID="b86f54905df3196f08b381db0e1865d1c2de39d817d6d0fa12dc4c1be2b81083" Oct 31 05:00:05.479702 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:00:05.479118 4376 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-12c0bc-29890380-wbz6z_odh-ai-gateway-infra(a63f4216-a71c-45b7-98b9-3b08500503fd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-12c0bc-29890380-wbz6z" podUID="a63f4216-a71c-45b7-98b9-3b08500503fd" Oct 31 05:00:05.481617 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:05.481591 4376 generic.go:356] "Generic (PLEG): container finished" podID="7212f966-42ad-4ed2-8e45-fde6a7908c2c" containerID="21ece3713eb78e7d7a0838d717578c72cc68a2b95ef29b39e5867eb213b5bccf" exitCode=60 Oct 31 05:00:05.481772 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:05.481649 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c9707f-29890380-k76x2" event={"ID":"7212f966-42ad-4ed2-8e45-fde6a7908c2c","Type":"ContainerDied","Data":"21ece3713eb78e7d7a0838d717578c72cc68a2b95ef29b39e5867eb213b5bccf"} Oct 31 05:00:05.482010 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:05.481993 4376 scope.go:122] "RemoveContainer" containerID="21ece3713eb78e7d7a0838d717578c72cc68a2b95ef29b39e5867eb213b5bccf" Oct 31 05:00:05.482315 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:00:05.482288 4376 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-c9707f-29890380-k76x2_odh-ai-gateway-infra(7212f966-42ad-4ed2-8e45-fde6a7908c2c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c9707f-29890380-k76x2" podUID="7212f966-42ad-4ed2-8e45-fde6a7908c2c" Oct 31 05:00:05.504987 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:05.504965 4376 scope.go:122] "RemoveContainer" containerID="7a673d19e73dd82962f0ad83d7d360609210973b5a1339eceb9a7db50c12b40b" Oct 31 05:00:05.518303 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:05.518283 4376 scope.go:122] "RemoveContainer" containerID="2ae3b8ad5076ba18377d54ad490a6c78b8566dc790523b2d419962409b0e958a" Oct 31 05:00:05.531237 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:05.531220 4376 scope.go:122] "RemoveContainer" containerID="9f2ec7678c989043357c4c48defabab3112f445e5bbdc02330d11d87f3f9bff2" Oct 31 05:00:05.574964 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:05.574944 4376 scope.go:122] "RemoveContainer" containerID="4e78cae8bdb304d97a0f7f4a085f26667c402115141a44776be279f271dc2e41" Oct 31 05:00:05.588417 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:05.588393 4376 scope.go:122] "RemoveContainer" containerID="fb1a95f99de3101b566bc0b567ca9c525cf1d0c1fe95181694f80c891386abbd" Oct 31 05:00:05.601309 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:05.601270 4376 scope.go:122] "RemoveContainer" containerID="852c252116984ecbf43ede42af636c4dc70a0bac6517531fa7de72fe19a2446e" Oct 31 05:00:05.613169 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:05.613153 4376 scope.go:122] "RemoveContainer" containerID="fec84ecc7792840cdbde9268d7936f283da1c585e4e022bc8ea09a4c5e365787" Oct 31 05:00:05.624693 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:05.624669 4376 scope.go:122] "RemoveContainer" containerID="228a2c860510a387940647d08197c45fa498f5a0be61336359f6de04d2ae594a" Oct 31 05:00:05.700410 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:05.700311 4376 patch_prober.go:29] interesting pod/e2e-inf-b-12c0bc-openshift-default-84bddf978c-ghhlp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" start-of-body= Oct 31 05:00:05.700410 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:05.700369 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-12c0bc-openshift-default-84bddf978c-ghhlp" podUID="7bde4a1c-a591-4864-8276-2c3a74b99e70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" Oct 31 05:00:06.492338 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:06.492295 4376 generic.go:356] "Generic (PLEG): container finished" podID="09f6c274-8499-4b06-91af-6af516c3e56a" containerID="d0b81b164bb58e9623a058006211bacddc07fbd537cc3b4d5fa73bfc50f6313c" exitCode=60 Oct 31 05:00:06.492755 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:06.492375 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-77a153-29890380-sdd6b" event={"ID":"09f6c274-8499-4b06-91af-6af516c3e56a","Type":"ContainerDied","Data":"d0b81b164bb58e9623a058006211bacddc07fbd537cc3b4d5fa73bfc50f6313c"} Oct 31 05:00:06.492755 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:06.492427 4376 scope.go:122] "RemoveContainer" containerID="7d9d54b5c54c0e42e925f9d5bdfdd5e379e454643722b73b8a12f7fd4dec1ddb" Oct 31 05:00:06.492755 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:06.492609 4376 scope.go:122] "RemoveContainer" containerID="d0b81b164bb58e9623a058006211bacddc07fbd537cc3b4d5fa73bfc50f6313c" Oct 31 05:00:06.492949 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:00:06.492884 4376 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-77a153-29890380-sdd6b_odh-ai-gateway-infra(09f6c274-8499-4b06-91af-6af516c3e56a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-77a153-29890380-sdd6b" podUID="09f6c274-8499-4b06-91af-6af516c3e56a" Oct 31 05:00:06.494490 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:06.494473 4376 scope.go:122] "RemoveContainer" containerID="2ab0a38673b93473da115aadc99d88a00e591ef94e8010319b1484825c6e29c0" Oct 31 05:00:06.494712 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:00:06.494689 4376 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-ded435-29890380-bwm8n_odh-ai-gateway-infra(5f6a15d6-d47f-47de-817f-fc70feb236e8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ded435-29890380-bwm8n" podUID="5f6a15d6-d47f-47de-817f-fc70feb236e8" Oct 31 05:00:06.497574 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:06.497557 4376 scope.go:122] "RemoveContainer" containerID="a6f0a4b7d76434f4153b9c5ee16256183bb375e4db3b606097ed48e506e8c624" Oct 31 05:00:06.497784 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:00:06.497761 4376 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-439989-29890380-rf2nf_odh-ai-gateway-infra(8dfadf3e-94d3-4e77-bd97-5bb554b03d2f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-439989-29890380-rf2nf" podUID="8dfadf3e-94d3-4e77-bd97-5bb554b03d2f" Oct 31 05:00:06.499252 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:06.499232 4376 scope.go:122] "RemoveContainer" containerID="d8aea9034a23a2093e6c4ed2b140f5323e732d1fa5eb3e718ff76edd45801a2a" Oct 31 05:00:06.499528 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:00:06.499509 4376 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-171edd-29890380-2kgrc_odh-ai-gateway-infra(2ae817be-98c3-4760-b1ed-b545d5b24e0f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-171edd-29890380-2kgrc" podUID="2ae817be-98c3-4760-b1ed-b545d5b24e0f" Oct 31 05:00:06.500706 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:06.500689 4376 scope.go:122] "RemoveContainer" containerID="b86f54905df3196f08b381db0e1865d1c2de39d817d6d0fa12dc4c1be2b81083" Oct 31 05:00:06.500951 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:00:06.500926 4376 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-12c0bc-29890380-wbz6z_odh-ai-gateway-infra(a63f4216-a71c-45b7-98b9-3b08500503fd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-12c0bc-29890380-wbz6z" podUID="a63f4216-a71c-45b7-98b9-3b08500503fd" Oct 31 05:00:06.502301 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:06.502277 4376 scope.go:122] "RemoveContainer" containerID="21ece3713eb78e7d7a0838d717578c72cc68a2b95ef29b39e5867eb213b5bccf" Oct 31 05:00:06.502584 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:00:06.502562 4376 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-c9707f-29890380-k76x2_odh-ai-gateway-infra(7212f966-42ad-4ed2-8e45-fde6a7908c2c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c9707f-29890380-k76x2" podUID="7212f966-42ad-4ed2-8e45-fde6a7908c2c" Oct 31 05:00:06.504526 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:06.504501 4376 generic.go:356] "Generic (PLEG): container finished" podID="79b8c49d-28ac-49fa-b101-5f553fdac035" containerID="880aee831dd880e881221efcb7dd61e2322e3fc5e39d9d55f940ddc73d566b9e" exitCode=60 Oct 31 05:00:06.504621 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:06.504581 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c1fe26-29890380-h89x7" event={"ID":"79b8c49d-28ac-49fa-b101-5f553fdac035","Type":"ContainerDied","Data":"880aee831dd880e881221efcb7dd61e2322e3fc5e39d9d55f940ddc73d566b9e"} Oct 31 05:00:06.504766 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:06.504751 4376 scope.go:122] "RemoveContainer" containerID="880aee831dd880e881221efcb7dd61e2322e3fc5e39d9d55f940ddc73d566b9e" Oct 31 05:00:06.504993 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:00:06.504967 4376 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-c1fe26-29890380-h89x7_odh-ai-gateway-infra(79b8c49d-28ac-49fa-b101-5f553fdac035)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c1fe26-29890380-h89x7" podUID="79b8c49d-28ac-49fa-b101-5f553fdac035" Oct 31 05:00:06.507473 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:06.507456 4376 scope.go:122] "RemoveContainer" containerID="4248202aedbc4d28b179949e4983081e2f9f916297de6b0d739b9d142ef97293" Oct 31 05:00:06.507929 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:06.507914 4376 scope.go:122] "RemoveContainer" containerID="fee90b045745163874d3ce8d1d390793bfae79163cd8ac97c1b1db79e3ae3ac1" Oct 31 05:00:06.508141 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:00:06.508118 4376 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-9633f0-29890380-jc5lw_odh-ai-gateway-infra(ecbbd44a-efa7-4e45-8514-1a57eccc71e5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9633f0-29890380-jc5lw" podUID="ecbbd44a-efa7-4e45-8514-1a57eccc71e5" Oct 31 05:00:06.509556 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:06.509541 4376 scope.go:122] "RemoveContainer" containerID="35c8d823dd3b6310e2df3beb04c8a5a85fa0afa80eae041fe933af6dc6e5fe35" Oct 31 05:00:06.509776 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:00:06.509754 4376 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-04d111-29890380-62zcn_odh-ai-gateway-infra(a9853a85-21be-4351-a76c-78d4da0aa7dc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-04d111-29890380-62zcn" podUID="a9853a85-21be-4351-a76c-78d4da0aa7dc" Oct 31 05:00:06.511082 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:06.511062 4376 generic.go:356] "Generic (PLEG): container finished" podID="7b953e1c-6983-456f-b2c8-77e9314aa758" containerID="4f771f169661e2b56bc8bd9464e1d7981b6ecfe95c0a168d3a891331786a5a22" exitCode=60 Oct 31 05:00:06.511181 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:06.511119 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-829ac7-29890380-pt7nf" event={"ID":"7b953e1c-6983-456f-b2c8-77e9314aa758","Type":"ContainerDied","Data":"4f771f169661e2b56bc8bd9464e1d7981b6ecfe95c0a168d3a891331786a5a22"} Oct 31 05:00:06.511372 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:06.511304 4376 scope.go:122] "RemoveContainer" containerID="4f771f169661e2b56bc8bd9464e1d7981b6ecfe95c0a168d3a891331786a5a22" Oct 31 05:00:06.511518 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:00:06.511501 4376 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-829ac7-29890380-pt7nf_odh-ai-gateway-infra(7b953e1c-6983-456f-b2c8-77e9314aa758)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-829ac7-29890380-pt7nf" podUID="7b953e1c-6983-456f-b2c8-77e9314aa758" Oct 31 05:00:06.520485 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:06.520467 4376 scope.go:122] "RemoveContainer" containerID="daf4da1298f8454e71fedd8a3b0b10319fc0b3e4250c2c39cadbb4a179dd493f" Oct 31 05:00:06.711120 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:06.711081 4376 patch_prober.go:29] interesting pod/e2e-inf-b-12c0bc-openshift-default-84bddf978c-ghhlp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" start-of-body= Oct 31 05:00:06.711465 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:06.711432 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-12c0bc-openshift-default-84bddf978c-ghhlp" podUID="7bde4a1c-a591-4864-8276-2c3a74b99e70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" Oct 31 05:00:07.522270 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:07.522242 4376 scope.go:122] "RemoveContainer" containerID="880aee831dd880e881221efcb7dd61e2322e3fc5e39d9d55f940ddc73d566b9e" Oct 31 05:00:07.522742 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:00:07.522573 4376 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-c1fe26-29890380-h89x7_odh-ai-gateway-infra(79b8c49d-28ac-49fa-b101-5f553fdac035)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c1fe26-29890380-h89x7" podUID="79b8c49d-28ac-49fa-b101-5f553fdac035" Oct 31 05:00:07.524164 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:07.524138 4376 scope.go:122] "RemoveContainer" containerID="4f771f169661e2b56bc8bd9464e1d7981b6ecfe95c0a168d3a891331786a5a22" Oct 31 05:00:07.524464 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:00:07.524428 4376 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-829ac7-29890380-pt7nf_odh-ai-gateway-infra(7b953e1c-6983-456f-b2c8-77e9314aa758)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-829ac7-29890380-pt7nf" podUID="7b953e1c-6983-456f-b2c8-77e9314aa758" Oct 31 05:00:07.525850 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:07.525834 4376 scope.go:122] "RemoveContainer" containerID="d0b81b164bb58e9623a058006211bacddc07fbd537cc3b4d5fa73bfc50f6313c" Oct 31 05:00:07.526058 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:00:07.526040 4376 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-77a153-29890380-sdd6b_odh-ai-gateway-infra(09f6c274-8499-4b06-91af-6af516c3e56a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-77a153-29890380-sdd6b" podUID="09f6c274-8499-4b06-91af-6af516c3e56a" Oct 31 05:00:07.705739 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:07.705704 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-12c0bc-openshift-default-84bddf978c-ghhlp" Oct 31 05:00:07.706214 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:07.706192 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-12c0bc-openshift-default-84bddf978c-ghhlp" Oct 31 05:00:07.707033 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:07.707013 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-12c0bc-openshift-default-84bddf978c-ghhlp" Oct 31 05:00:11.000773 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:00:11.000735 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 05:00:15.436824 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:15.436796 4376 scope.go:122] "RemoveContainer" containerID="21ece3713eb78e7d7a0838d717578c72cc68a2b95ef29b39e5867eb213b5bccf" Oct 31 05:00:15.437248 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:15.437163 4376 scope.go:122] "RemoveContainer" containerID="8ae831e41865317107e036d7aa6e1e552a3769eadb8f00869c49636337b377cf" Oct 31 05:00:15.437315 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:15.437263 4376 scope.go:122] "RemoveContainer" containerID="a6f0a4b7d76434f4153b9c5ee16256183bb375e4db3b606097ed48e506e8c624" Oct 31 05:00:16.436533 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:16.436510 4376 scope.go:122] "RemoveContainer" containerID="b86f54905df3196f08b381db0e1865d1c2de39d817d6d0fa12dc4c1be2b81083" Oct 31 05:00:16.436682 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:16.436648 4376 scope.go:122] "RemoveContainer" containerID="2ab0a38673b93473da115aadc99d88a00e591ef94e8010319b1484825c6e29c0" Oct 31 05:00:16.606433 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:16.606364 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c9707f-29890380-k76x2" event={"ID":"7212f966-42ad-4ed2-8e45-fde6a7908c2c","Type":"ContainerStarted","Data":"b308d66abe65688c7c62e0ba10a035a211936f2afbcce7f4ad28ff4676b906df"} Oct 31 05:00:16.608573 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:16.608549 4376 generic.go:356] "Generic (PLEG): container finished" podID="978b4c58-1c25-4e04-bfbf-5cd4e0a7001d" containerID="2f246ab3350b66e2fc1e0994acf426bb005238d5360b6a7522d96470c07a6565" exitCode=60 Oct 31 05:00:16.608743 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:16.608616 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d6ac82-29890380-xtwh8" event={"ID":"978b4c58-1c25-4e04-bfbf-5cd4e0a7001d","Type":"ContainerDied","Data":"2f246ab3350b66e2fc1e0994acf426bb005238d5360b6a7522d96470c07a6565"} Oct 31 05:00:16.608743 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:16.608670 4376 scope.go:122] "RemoveContainer" containerID="8ae831e41865317107e036d7aa6e1e552a3769eadb8f00869c49636337b377cf" Oct 31 05:00:16.609011 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:16.608993 4376 scope.go:122] "RemoveContainer" containerID="2f246ab3350b66e2fc1e0994acf426bb005238d5360b6a7522d96470c07a6565" Oct 31 05:00:16.609344 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:00:16.609307 4376 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-d6ac82-29890380-xtwh8_odh-ai-gateway-infra(978b4c58-1c25-4e04-bfbf-5cd4e0a7001d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d6ac82-29890380-xtwh8" podUID="978b4c58-1c25-4e04-bfbf-5cd4e0a7001d" Oct 31 05:00:16.610895 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:16.610876 4376 generic.go:356] "Generic (PLEG): container finished" podID="8dfadf3e-94d3-4e77-bd97-5bb554b03d2f" containerID="49ee5ea2a94ae2fb0db01680a2157703d548c7ad0e27c2605b5781e87266e509" exitCode=60 Oct 31 05:00:16.611002 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:16.610939 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-439989-29890380-rf2nf" event={"ID":"8dfadf3e-94d3-4e77-bd97-5bb554b03d2f","Type":"ContainerDied","Data":"49ee5ea2a94ae2fb0db01680a2157703d548c7ad0e27c2605b5781e87266e509"} Oct 31 05:00:16.611300 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:16.611284 4376 scope.go:122] "RemoveContainer" containerID="49ee5ea2a94ae2fb0db01680a2157703d548c7ad0e27c2605b5781e87266e509" Oct 31 05:00:16.611660 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:00:16.611641 4376 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-439989-29890380-rf2nf_odh-ai-gateway-infra(8dfadf3e-94d3-4e77-bd97-5bb554b03d2f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-439989-29890380-rf2nf" podUID="8dfadf3e-94d3-4e77-bd97-5bb554b03d2f" Oct 31 05:00:16.624727 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:16.624709 4376 scope.go:122] "RemoveContainer" containerID="a6f0a4b7d76434f4153b9c5ee16256183bb375e4db3b606097ed48e506e8c624" Oct 31 05:00:17.436753 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:17.436721 4376 scope.go:122] "RemoveContainer" containerID="d8aea9034a23a2093e6c4ed2b140f5323e732d1fa5eb3e718ff76edd45801a2a" Oct 31 05:00:17.436937 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:17.436844 4376 scope.go:122] "RemoveContainer" containerID="d0b81b164bb58e9623a058006211bacddc07fbd537cc3b4d5fa73bfc50f6313c" Oct 31 05:00:17.437023 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:17.436973 4376 scope.go:122] "RemoveContainer" containerID="880aee831dd880e881221efcb7dd61e2322e3fc5e39d9d55f940ddc73d566b9e" Oct 31 05:00:17.621592 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:17.621568 4376 generic.go:356] "Generic (PLEG): container finished" podID="5f6a15d6-d47f-47de-817f-fc70feb236e8" containerID="f4528ab80e49b1c805373a6e07c267cc9ce84dc39f12596eba1edf90576c5eb4" exitCode=60 Oct 31 05:00:17.621962 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:17.621641 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ded435-29890380-bwm8n" event={"ID":"5f6a15d6-d47f-47de-817f-fc70feb236e8","Type":"ContainerDied","Data":"f4528ab80e49b1c805373a6e07c267cc9ce84dc39f12596eba1edf90576c5eb4"} Oct 31 05:00:17.621962 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:17.621701 4376 scope.go:122] "RemoveContainer" containerID="2ab0a38673b93473da115aadc99d88a00e591ef94e8010319b1484825c6e29c0" Oct 31 05:00:17.622105 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:17.622090 4376 scope.go:122] "RemoveContainer" containerID="f4528ab80e49b1c805373a6e07c267cc9ce84dc39f12596eba1edf90576c5eb4" Oct 31 05:00:17.622481 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:00:17.622456 4376 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-ded435-29890380-bwm8n_odh-ai-gateway-infra(5f6a15d6-d47f-47de-817f-fc70feb236e8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ded435-29890380-bwm8n" podUID="5f6a15d6-d47f-47de-817f-fc70feb236e8" Oct 31 05:00:17.626922 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:17.626902 4376 generic.go:356] "Generic (PLEG): container finished" podID="a63f4216-a71c-45b7-98b9-3b08500503fd" containerID="f2291e6d7c8007a4fc6685be414eeb08ec625b5c2f4dc2e19458cfb4f63a1638" exitCode=60 Oct 31 05:00:17.627019 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:17.626976 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-12c0bc-29890380-wbz6z" event={"ID":"a63f4216-a71c-45b7-98b9-3b08500503fd","Type":"ContainerDied","Data":"f2291e6d7c8007a4fc6685be414eeb08ec625b5c2f4dc2e19458cfb4f63a1638"} Oct 31 05:00:17.627358 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:17.627322 4376 scope.go:122] "RemoveContainer" containerID="f2291e6d7c8007a4fc6685be414eeb08ec625b5c2f4dc2e19458cfb4f63a1638" Oct 31 05:00:17.627691 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:00:17.627669 4376 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-12c0bc-29890380-wbz6z_odh-ai-gateway-infra(a63f4216-a71c-45b7-98b9-3b08500503fd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-12c0bc-29890380-wbz6z" podUID="a63f4216-a71c-45b7-98b9-3b08500503fd" Oct 31 05:00:17.628939 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:17.628921 4376 generic.go:356] "Generic (PLEG): container finished" podID="7212f966-42ad-4ed2-8e45-fde6a7908c2c" containerID="b308d66abe65688c7c62e0ba10a035a211936f2afbcce7f4ad28ff4676b906df" exitCode=60 Oct 31 05:00:17.629012 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:17.628969 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c9707f-29890380-k76x2" event={"ID":"7212f966-42ad-4ed2-8e45-fde6a7908c2c","Type":"ContainerDied","Data":"b308d66abe65688c7c62e0ba10a035a211936f2afbcce7f4ad28ff4676b906df"} Oct 31 05:00:17.629256 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:17.629242 4376 scope.go:122] "RemoveContainer" containerID="b308d66abe65688c7c62e0ba10a035a211936f2afbcce7f4ad28ff4676b906df" Oct 31 05:00:17.629505 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:00:17.629486 4376 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-c9707f-29890380-k76x2_odh-ai-gateway-infra(7212f966-42ad-4ed2-8e45-fde6a7908c2c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c9707f-29890380-k76x2" podUID="7212f966-42ad-4ed2-8e45-fde6a7908c2c" Oct 31 05:00:17.637753 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:17.637498 4376 scope.go:122] "RemoveContainer" containerID="b86f54905df3196f08b381db0e1865d1c2de39d817d6d0fa12dc4c1be2b81083" Oct 31 05:00:17.654964 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:17.654936 4376 scope.go:122] "RemoveContainer" containerID="21ece3713eb78e7d7a0838d717578c72cc68a2b95ef29b39e5867eb213b5bccf" Oct 31 05:00:17.667062 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:17.667040 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c9707f-29890380-k76x2"] Oct 31 05:00:17.706885 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:17.706824 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d6ac82-29890380-xtwh8"] Oct 31 05:00:17.745833 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:17.745805 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-439989-29890380-rf2nf"] Oct 31 05:00:18.018972 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:18.018949 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d6ac82-29890380-xtwh8" Oct 31 05:00:18.075979 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:18.075956 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-439989-29890380-rf2nf" Oct 31 05:00:18.130582 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:18.130554 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/978b4c58-1c25-4e04-bfbf-5cd4e0a7001d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/978b4c58-1c25-4e04-bfbf-5cd4e0a7001d-maas-api-service-ca\") pod \"978b4c58-1c25-4e04-bfbf-5cd4e0a7001d\" (UID: \"978b4c58-1c25-4e04-bfbf-5cd4e0a7001d\") " Oct 31 05:00:18.130807 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:18.130674 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8dfadf3e-94d3-4e77-bd97-5bb554b03d2f-kube-api-access-24pk8\" (UniqueName: \"kubernetes.io/projected/8dfadf3e-94d3-4e77-bd97-5bb554b03d2f-kube-api-access-24pk8\") pod \"8dfadf3e-94d3-4e77-bd97-5bb554b03d2f\" (UID: \"8dfadf3e-94d3-4e77-bd97-5bb554b03d2f\") " Oct 31 05:00:18.130955 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:18.130929 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/978b4c58-1c25-4e04-bfbf-5cd4e0a7001d-maas-api-service-ca" pod "978b4c58-1c25-4e04-bfbf-5cd4e0a7001d" (UID: "978b4c58-1c25-4e04-bfbf-5cd4e0a7001d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:00:18.131022 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:18.131000 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8dfadf3e-94d3-4e77-bd97-5bb554b03d2f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8dfadf3e-94d3-4e77-bd97-5bb554b03d2f-maas-api-service-ca\") pod \"8dfadf3e-94d3-4e77-bd97-5bb554b03d2f\" (UID: \"8dfadf3e-94d3-4e77-bd97-5bb554b03d2f\") " Oct 31 05:00:18.131141 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:18.131127 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/978b4c58-1c25-4e04-bfbf-5cd4e0a7001d-kube-api-access-6f4r6\" (UniqueName: \"kubernetes.io/projected/978b4c58-1c25-4e04-bfbf-5cd4e0a7001d-kube-api-access-6f4r6\") pod \"978b4c58-1c25-4e04-bfbf-5cd4e0a7001d\" (UID: \"978b4c58-1c25-4e04-bfbf-5cd4e0a7001d\") " Oct 31 05:00:18.131346 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:18.131314 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8dfadf3e-94d3-4e77-bd97-5bb554b03d2f-maas-api-service-ca" pod "8dfadf3e-94d3-4e77-bd97-5bb554b03d2f" (UID: "8dfadf3e-94d3-4e77-bd97-5bb554b03d2f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:00:18.131827 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:18.131805 4376 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/978b4c58-1c25-4e04-bfbf-5cd4e0a7001d-maas-api-service-ca\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:00:18.131908 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:18.131831 4376 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8dfadf3e-94d3-4e77-bd97-5bb554b03d2f-maas-api-service-ca\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:00:18.133499 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:18.133467 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8dfadf3e-94d3-4e77-bd97-5bb554b03d2f-kube-api-access-24pk8" pod "8dfadf3e-94d3-4e77-bd97-5bb554b03d2f" (UID: "8dfadf3e-94d3-4e77-bd97-5bb554b03d2f"). InnerVolumeSpecName "kube-api-access-24pk8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:00:18.133593 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:18.133571 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/978b4c58-1c25-4e04-bfbf-5cd4e0a7001d-kube-api-access-6f4r6" pod "978b4c58-1c25-4e04-bfbf-5cd4e0a7001d" (UID: "978b4c58-1c25-4e04-bfbf-5cd4e0a7001d"). InnerVolumeSpecName "kube-api-access-6f4r6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:00:18.232892 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:18.232834 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6f4r6\" (UniqueName: \"kubernetes.io/projected/978b4c58-1c25-4e04-bfbf-5cd4e0a7001d-kube-api-access-6f4r6\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:00:18.232892 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:18.232861 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24pk8\" (UniqueName: \"kubernetes.io/projected/8dfadf3e-94d3-4e77-bd97-5bb554b03d2f-kube-api-access-24pk8\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:00:18.435915 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:18.435877 4376 scope.go:122] "RemoveContainer" containerID="4f771f169661e2b56bc8bd9464e1d7981b6ecfe95c0a168d3a891331786a5a22" Oct 31 05:00:18.436377 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:18.436313 4376 scope.go:122] "RemoveContainer" containerID="35c8d823dd3b6310e2df3beb04c8a5a85fa0afa80eae041fe933af6dc6e5fe35" Oct 31 05:00:18.436535 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:18.436460 4376 scope.go:122] "RemoveContainer" containerID="fee90b045745163874d3ce8d1d390793bfae79163cd8ac97c1b1db79e3ae3ac1" Oct 31 05:00:18.641121 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:18.641089 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-77a153-29890380-sdd6b" event={"ID":"09f6c274-8499-4b06-91af-6af516c3e56a","Type":"ContainerStarted","Data":"b9c2de8b318796f223d97db2d880e454cd33603fa797b58846325ce9c8e54aae"} Oct 31 05:00:18.647927 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:18.647907 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d6ac82-29890380-xtwh8" Oct 31 05:00:18.648047 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:18.647946 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d6ac82-29890380-xtwh8" event={"ID":"978b4c58-1c25-4e04-bfbf-5cd4e0a7001d","Type":"ContainerDied","Data":"43a74639293e623020d57c652c21ba1f7076b06dd12d76a4dac4a1ec32df3263"} Oct 31 05:00:18.648103 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:18.648042 4376 scope.go:122] "RemoveContainer" containerID="2f246ab3350b66e2fc1e0994acf426bb005238d5360b6a7522d96470c07a6565" Oct 31 05:00:18.649432 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:18.649414 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-439989-29890380-rf2nf" Oct 31 05:00:18.649550 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:18.649433 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-439989-29890380-rf2nf" event={"ID":"8dfadf3e-94d3-4e77-bd97-5bb554b03d2f","Type":"ContainerDied","Data":"c167f93d9612641d6dc9934fe1f9dc370244247255bb7b9251557dd3e773a396"} Oct 31 05:00:18.652600 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:18.652579 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-171edd-29890380-2kgrc" event={"ID":"2ae817be-98c3-4760-b1ed-b545d5b24e0f","Type":"ContainerStarted","Data":"63240c6d244fe984e6c23db016d4f2a8ee185e12da072b9350281bd25a429932"} Oct 31 05:00:18.657935 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:18.657917 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c1fe26-29890380-h89x7" event={"ID":"79b8c49d-28ac-49fa-b101-5f553fdac035","Type":"ContainerStarted","Data":"99ca664394ccf80df44d63d974b0524a98d5a65d945ace6c072411caa55f08af"} Oct 31 05:00:18.658214 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:18.658200 4376 scope.go:122] "RemoveContainer" containerID="99ca664394ccf80df44d63d974b0524a98d5a65d945ace6c072411caa55f08af" Oct 31 05:00:18.658513 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:00:18.658490 4376 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-c1fe26-29890380-h89x7_odh-ai-gateway-infra(79b8c49d-28ac-49fa-b101-5f553fdac035)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c1fe26-29890380-h89x7" podUID="79b8c49d-28ac-49fa-b101-5f553fdac035" Oct 31 05:00:18.678414 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:18.678389 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ded435-29890380-bwm8n"] Oct 31 05:00:18.693657 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:18.693638 4376 scope.go:122] "RemoveContainer" containerID="49ee5ea2a94ae2fb0db01680a2157703d548c7ad0e27c2605b5781e87266e509" Oct 31 05:00:18.776104 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:18.774343 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-12c0bc-29890380-wbz6z"] Oct 31 05:00:18.858483 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:18.858201 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-171edd-29890380-2kgrc" podStartSLOduration=18.858187071 podStartE2EDuration="18.858187071s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 05:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 05:00:18.852732451 +0000 UTC m=+3366.567067071" watchObservedRunningTime="2026-10-31 05:00:18.858187071 +0000 UTC m=+3366.572521692" Oct 31 05:00:18.898156 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:18.897246 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c9707f-29890380-k76x2" Oct 31 05:00:18.954532 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:18.954499 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7212f966-42ad-4ed2-8e45-fde6a7908c2c-kube-api-access-x92tm\" (UniqueName: \"kubernetes.io/projected/7212f966-42ad-4ed2-8e45-fde6a7908c2c-kube-api-access-x92tm\") pod \"7212f966-42ad-4ed2-8e45-fde6a7908c2c\" (UID: \"7212f966-42ad-4ed2-8e45-fde6a7908c2c\") " Oct 31 05:00:18.954790 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:18.954762 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7212f966-42ad-4ed2-8e45-fde6a7908c2c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7212f966-42ad-4ed2-8e45-fde6a7908c2c-maas-api-service-ca\") pod \"7212f966-42ad-4ed2-8e45-fde6a7908c2c\" (UID: \"7212f966-42ad-4ed2-8e45-fde6a7908c2c\") " Oct 31 05:00:18.957281 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:18.956433 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7212f966-42ad-4ed2-8e45-fde6a7908c2c-maas-api-service-ca" pod "7212f966-42ad-4ed2-8e45-fde6a7908c2c" (UID: "7212f966-42ad-4ed2-8e45-fde6a7908c2c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:00:18.957867 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:18.957808 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-439989-29890380-rf2nf"] Oct 31 05:00:18.960447 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:18.960422 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7212f966-42ad-4ed2-8e45-fde6a7908c2c-kube-api-access-x92tm" pod "7212f966-42ad-4ed2-8e45-fde6a7908c2c" (UID: "7212f966-42ad-4ed2-8e45-fde6a7908c2c"). InnerVolumeSpecName "kube-api-access-x92tm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:00:18.973098 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:18.972723 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-439989-29890380-rf2nf"] Oct 31 05:00:19.077242 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:19.077181 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x92tm\" (UniqueName: \"kubernetes.io/projected/7212f966-42ad-4ed2-8e45-fde6a7908c2c-kube-api-access-x92tm\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:00:19.077242 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:19.077214 4376 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7212f966-42ad-4ed2-8e45-fde6a7908c2c-maas-api-service-ca\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:00:19.094988 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:19.094905 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ded435-29890380-bwm8n" Oct 31 05:00:19.124605 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:19.124574 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d6ac82-29890380-xtwh8"] Oct 31 05:00:19.131793 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:19.131767 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d6ac82-29890380-xtwh8"] Oct 31 05:00:19.139498 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:19.139477 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-12c0bc-29890380-wbz6z" Oct 31 05:00:19.178470 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:19.178436 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f6a15d6-d47f-47de-817f-fc70feb236e8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f6a15d6-d47f-47de-817f-fc70feb236e8-maas-api-service-ca\") pod \"5f6a15d6-d47f-47de-817f-fc70feb236e8\" (UID: \"5f6a15d6-d47f-47de-817f-fc70feb236e8\") " Oct 31 05:00:19.178729 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:19.178706 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f6a15d6-d47f-47de-817f-fc70feb236e8-kube-api-access-t97bc\" (UniqueName: \"kubernetes.io/projected/5f6a15d6-d47f-47de-817f-fc70feb236e8-kube-api-access-t97bc\") pod \"5f6a15d6-d47f-47de-817f-fc70feb236e8\" (UID: \"5f6a15d6-d47f-47de-817f-fc70feb236e8\") " Oct 31 05:00:19.178834 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:19.178814 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f6a15d6-d47f-47de-817f-fc70feb236e8-maas-api-service-ca" pod "5f6a15d6-d47f-47de-817f-fc70feb236e8" (UID: "5f6a15d6-d47f-47de-817f-fc70feb236e8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:00:19.179545 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:19.179523 4376 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f6a15d6-d47f-47de-817f-fc70feb236e8-maas-api-service-ca\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:00:19.237980 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:19.237946 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f6a15d6-d47f-47de-817f-fc70feb236e8-kube-api-access-t97bc" pod "5f6a15d6-d47f-47de-817f-fc70feb236e8" (UID: "5f6a15d6-d47f-47de-817f-fc70feb236e8"). InnerVolumeSpecName "kube-api-access-t97bc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:00:19.280912 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:19.280842 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a63f4216-a71c-45b7-98b9-3b08500503fd-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a63f4216-a71c-45b7-98b9-3b08500503fd-maas-api-service-ca\") pod \"a63f4216-a71c-45b7-98b9-3b08500503fd\" (UID: \"a63f4216-a71c-45b7-98b9-3b08500503fd\") " Oct 31 05:00:19.281039 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:19.280968 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a63f4216-a71c-45b7-98b9-3b08500503fd-kube-api-access-dmjft\" (UniqueName: \"kubernetes.io/projected/a63f4216-a71c-45b7-98b9-3b08500503fd-kube-api-access-dmjft\") pod \"a63f4216-a71c-45b7-98b9-3b08500503fd\" (UID: \"a63f4216-a71c-45b7-98b9-3b08500503fd\") " Oct 31 05:00:19.281196 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:19.281176 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a63f4216-a71c-45b7-98b9-3b08500503fd-maas-api-service-ca" pod "a63f4216-a71c-45b7-98b9-3b08500503fd" (UID: "a63f4216-a71c-45b7-98b9-3b08500503fd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:00:19.281796 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:19.281764 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t97bc\" (UniqueName: \"kubernetes.io/projected/5f6a15d6-d47f-47de-817f-fc70feb236e8-kube-api-access-t97bc\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:00:19.281796 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:19.281796 4376 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a63f4216-a71c-45b7-98b9-3b08500503fd-maas-api-service-ca\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:00:19.282968 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:19.282950 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a63f4216-a71c-45b7-98b9-3b08500503fd-kube-api-access-dmjft" pod "a63f4216-a71c-45b7-98b9-3b08500503fd" (UID: "a63f4216-a71c-45b7-98b9-3b08500503fd"). InnerVolumeSpecName "kube-api-access-dmjft". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:00:19.383411 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:19.383384 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dmjft\" (UniqueName: \"kubernetes.io/projected/a63f4216-a71c-45b7-98b9-3b08500503fd-kube-api-access-dmjft\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:00:19.436383 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:19.436345 4376 scope.go:122] "RemoveContainer" containerID="387e7f9074661504d45b68a4748a0ee6ec00865d344dea2ae063710166430d78" Oct 31 05:00:19.682607 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:19.682574 4376 generic.go:356] "Generic (PLEG): container finished" podID="2ae817be-98c3-4760-b1ed-b545d5b24e0f" containerID="63240c6d244fe984e6c23db016d4f2a8ee185e12da072b9350281bd25a429932" exitCode=60 Oct 31 05:00:19.683098 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:19.682667 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-171edd-29890380-2kgrc" event={"ID":"2ae817be-98c3-4760-b1ed-b545d5b24e0f","Type":"ContainerDied","Data":"63240c6d244fe984e6c23db016d4f2a8ee185e12da072b9350281bd25a429932"} Oct 31 05:00:19.683098 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:19.682767 4376 scope.go:122] "RemoveContainer" containerID="d8aea9034a23a2093e6c4ed2b140f5323e732d1fa5eb3e718ff76edd45801a2a" Oct 31 05:00:19.683550 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:19.683526 4376 scope.go:122] "RemoveContainer" containerID="63240c6d244fe984e6c23db016d4f2a8ee185e12da072b9350281bd25a429932" Oct 31 05:00:19.683901 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:00:19.683881 4376 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-171edd-29890380-2kgrc_odh-ai-gateway-infra(2ae817be-98c3-4760-b1ed-b545d5b24e0f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-171edd-29890380-2kgrc" podUID="2ae817be-98c3-4760-b1ed-b545d5b24e0f" Oct 31 05:00:19.685086 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:19.685068 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-12c0bc-29890380-wbz6z" Oct 31 05:00:19.685182 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:19.685071 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-12c0bc-29890380-wbz6z" event={"ID":"a63f4216-a71c-45b7-98b9-3b08500503fd","Type":"ContainerDied","Data":"07f8cfa56c5119bd07577377d19e69885171eb1ae2c3fa6811176cada45a4e53"} Oct 31 05:00:19.686519 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:19.686499 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c9707f-29890380-k76x2" Oct 31 05:00:19.686628 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:19.686522 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c9707f-29890380-k76x2" event={"ID":"7212f966-42ad-4ed2-8e45-fde6a7908c2c","Type":"ContainerDied","Data":"28e9dd2c9c71cf02e148ddddd32d82a6847065ad8a9104f5766f968a615b3e1f"} Oct 31 05:00:19.689456 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:19.689432 4376 generic.go:356] "Generic (PLEG): container finished" podID="79b8c49d-28ac-49fa-b101-5f553fdac035" containerID="99ca664394ccf80df44d63d974b0524a98d5a65d945ace6c072411caa55f08af" exitCode=60 Oct 31 05:00:19.689552 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:19.689504 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c1fe26-29890380-h89x7" event={"ID":"79b8c49d-28ac-49fa-b101-5f553fdac035","Type":"ContainerDied","Data":"99ca664394ccf80df44d63d974b0524a98d5a65d945ace6c072411caa55f08af"} Oct 31 05:00:19.689876 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:19.689855 4376 scope.go:122] "RemoveContainer" containerID="99ca664394ccf80df44d63d974b0524a98d5a65d945ace6c072411caa55f08af" Oct 31 05:00:19.690177 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:00:19.690154 4376 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-c1fe26-29890380-h89x7_odh-ai-gateway-infra(79b8c49d-28ac-49fa-b101-5f553fdac035)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c1fe26-29890380-h89x7" podUID="79b8c49d-28ac-49fa-b101-5f553fdac035" Oct 31 05:00:19.695004 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:19.694980 4376 generic.go:356] "Generic (PLEG): container finished" podID="ecbbd44a-efa7-4e45-8514-1a57eccc71e5" containerID="e4b8ed9b88d53f2ca7ef97b35b15a7b6cc8b075f7ec3509fe1d0e905341e08d5" exitCode=60 Oct 31 05:00:19.695094 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:19.695059 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9633f0-29890380-jc5lw" event={"ID":"ecbbd44a-efa7-4e45-8514-1a57eccc71e5","Type":"ContainerDied","Data":"e4b8ed9b88d53f2ca7ef97b35b15a7b6cc8b075f7ec3509fe1d0e905341e08d5"} Oct 31 05:00:19.695412 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:19.695393 4376 scope.go:122] "RemoveContainer" containerID="e4b8ed9b88d53f2ca7ef97b35b15a7b6cc8b075f7ec3509fe1d0e905341e08d5" Oct 31 05:00:19.695748 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:00:19.695666 4376 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-9633f0-29890380-jc5lw_odh-ai-gateway-infra(ecbbd44a-efa7-4e45-8514-1a57eccc71e5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9633f0-29890380-jc5lw" podUID="ecbbd44a-efa7-4e45-8514-1a57eccc71e5" Oct 31 05:00:19.697663 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:19.697642 4376 generic.go:356] "Generic (PLEG): container finished" podID="a9853a85-21be-4351-a76c-78d4da0aa7dc" containerID="b840ffb04eb4f9af37ca5624cbb78cbf2e25313d35b7ef3fa4f4f9c25b00d3a1" exitCode=60 Oct 31 05:00:19.697753 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:19.697715 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-04d111-29890380-62zcn" event={"ID":"a9853a85-21be-4351-a76c-78d4da0aa7dc","Type":"ContainerDied","Data":"b840ffb04eb4f9af37ca5624cbb78cbf2e25313d35b7ef3fa4f4f9c25b00d3a1"} Oct 31 05:00:19.698059 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:19.698043 4376 scope.go:122] "RemoveContainer" containerID="b840ffb04eb4f9af37ca5624cbb78cbf2e25313d35b7ef3fa4f4f9c25b00d3a1" Oct 31 05:00:19.698415 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:00:19.698391 4376 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-04d111-29890380-62zcn_odh-ai-gateway-infra(a9853a85-21be-4351-a76c-78d4da0aa7dc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-04d111-29890380-62zcn" podUID="a9853a85-21be-4351-a76c-78d4da0aa7dc" Oct 31 05:00:19.700151 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:19.700099 4376 generic.go:356] "Generic (PLEG): container finished" podID="7b953e1c-6983-456f-b2c8-77e9314aa758" containerID="3e282356e5f24d514fffd12af8d3983ef08e1b0cf209b1165e0d377ee66486d1" exitCode=60 Oct 31 05:00:19.700267 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:19.700171 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-829ac7-29890380-pt7nf" event={"ID":"7b953e1c-6983-456f-b2c8-77e9314aa758","Type":"ContainerDied","Data":"3e282356e5f24d514fffd12af8d3983ef08e1b0cf209b1165e0d377ee66486d1"} Oct 31 05:00:19.700267 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:19.700233 4376 scope.go:122] "RemoveContainer" containerID="f2291e6d7c8007a4fc6685be414eeb08ec625b5c2f4dc2e19458cfb4f63a1638" Oct 31 05:00:19.700537 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:19.700518 4376 scope.go:122] "RemoveContainer" containerID="3e282356e5f24d514fffd12af8d3983ef08e1b0cf209b1165e0d377ee66486d1" Oct 31 05:00:19.700750 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:19.700524 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-77a153-29890380-sdd6b"] Oct 31 05:00:19.700837 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:00:19.700772 4376 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-829ac7-29890380-pt7nf_odh-ai-gateway-infra(7b953e1c-6983-456f-b2c8-77e9314aa758)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-829ac7-29890380-pt7nf" podUID="7b953e1c-6983-456f-b2c8-77e9314aa758" Oct 31 05:00:19.703407 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:19.703386 4376 generic.go:356] "Generic (PLEG): container finished" podID="09f6c274-8499-4b06-91af-6af516c3e56a" containerID="b9c2de8b318796f223d97db2d880e454cd33603fa797b58846325ce9c8e54aae" exitCode=60 Oct 31 05:00:19.703503 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:19.703484 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-77a153-29890380-sdd6b" event={"ID":"09f6c274-8499-4b06-91af-6af516c3e56a","Type":"ContainerDied","Data":"b9c2de8b318796f223d97db2d880e454cd33603fa797b58846325ce9c8e54aae"} Oct 31 05:00:19.704970 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:19.704951 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ded435-29890380-bwm8n" event={"ID":"5f6a15d6-d47f-47de-817f-fc70feb236e8","Type":"ContainerDied","Data":"cbf53e770eb73a968206c047a4fe66abfaf6eeafa7461bde7c3202a933ea03fb"} Oct 31 05:00:19.705069 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:19.704996 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ded435-29890380-bwm8n" Oct 31 05:00:19.713114 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:19.713097 4376 scope.go:122] "RemoveContainer" containerID="b308d66abe65688c7c62e0ba10a035a211936f2afbcce7f4ad28ff4676b906df" Oct 31 05:00:19.724002 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:19.723980 4376 scope.go:122] "RemoveContainer" containerID="880aee831dd880e881221efcb7dd61e2322e3fc5e39d9d55f940ddc73d566b9e" Oct 31 05:00:19.741499 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:19.741476 4376 scope.go:122] "RemoveContainer" containerID="fee90b045745163874d3ce8d1d390793bfae79163cd8ac97c1b1db79e3ae3ac1" Oct 31 05:00:19.742944 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:19.742888 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c1fe26-29890380-h89x7"] Oct 31 05:00:19.869486 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:19.849366 4376 scope.go:122] "RemoveContainer" containerID="35c8d823dd3b6310e2df3beb04c8a5a85fa0afa80eae041fe933af6dc6e5fe35" Oct 31 05:00:19.869486 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:19.866738 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ded435-29890380-bwm8n"] Oct 31 05:00:19.876940 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:19.876911 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ded435-29890380-bwm8n"] Oct 31 05:00:19.890961 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:19.890940 4376 scope.go:122] "RemoveContainer" containerID="4f771f169661e2b56bc8bd9464e1d7981b6ecfe95c0a168d3a891331786a5a22" Oct 31 05:00:19.898322 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:19.898302 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-12c0bc-29890380-wbz6z"] Oct 31 05:00:19.907548 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:19.907369 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-12c0bc-29890380-wbz6z"] Oct 31 05:00:19.918288 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:19.918262 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-171edd-29890380-2kgrc"] Oct 31 05:00:19.920234 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:19.919633 4376 scope.go:122] "RemoveContainer" containerID="d0b81b164bb58e9623a058006211bacddc07fbd537cc3b4d5fa73bfc50f6313c" Oct 31 05:00:19.954245 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:19.949935 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c9707f-29890380-k76x2"] Oct 31 05:00:19.959954 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:19.959932 4376 scope.go:122] "RemoveContainer" containerID="f4528ab80e49b1c805373a6e07c267cc9ce84dc39f12596eba1edf90576c5eb4" Oct 31 05:00:19.968982 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:19.964414 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c9707f-29890380-k76x2"] Oct 31 05:00:19.971899 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:19.971880 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-77a153-29890380-sdd6b" Oct 31 05:00:20.132411 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:20.132382 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09f6c274-8499-4b06-91af-6af516c3e56a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09f6c274-8499-4b06-91af-6af516c3e56a-maas-api-service-ca\") pod \"09f6c274-8499-4b06-91af-6af516c3e56a\" (UID: \"09f6c274-8499-4b06-91af-6af516c3e56a\") " Oct 31 05:00:20.132577 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:20.132530 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09f6c274-8499-4b06-91af-6af516c3e56a-kube-api-access-mhg2z\" (UniqueName: \"kubernetes.io/projected/09f6c274-8499-4b06-91af-6af516c3e56a-kube-api-access-mhg2z\") pod \"09f6c274-8499-4b06-91af-6af516c3e56a\" (UID: \"09f6c274-8499-4b06-91af-6af516c3e56a\") " Oct 31 05:00:20.132734 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:20.132712 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09f6c274-8499-4b06-91af-6af516c3e56a-maas-api-service-ca" pod "09f6c274-8499-4b06-91af-6af516c3e56a" (UID: "09f6c274-8499-4b06-91af-6af516c3e56a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:00:20.133053 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:20.133035 4376 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09f6c274-8499-4b06-91af-6af516c3e56a-maas-api-service-ca\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:00:20.134679 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:20.134658 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09f6c274-8499-4b06-91af-6af516c3e56a-kube-api-access-mhg2z" pod "09f6c274-8499-4b06-91af-6af516c3e56a" (UID: "09f6c274-8499-4b06-91af-6af516c3e56a"). InnerVolumeSpecName "kube-api-access-mhg2z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:00:20.234484 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:20.234454 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhg2z\" (UniqueName: \"kubernetes.io/projected/09f6c274-8499-4b06-91af-6af516c3e56a-kube-api-access-mhg2z\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:00:20.447054 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:20.446981 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f6a15d6-d47f-47de-817f-fc70feb236e8" path="/var/lib/kubelet/pods/5f6a15d6-d47f-47de-817f-fc70feb236e8/volumes" Oct 31 05:00:20.447449 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:20.447433 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7212f966-42ad-4ed2-8e45-fde6a7908c2c" path="/var/lib/kubelet/pods/7212f966-42ad-4ed2-8e45-fde6a7908c2c/volumes" Oct 31 05:00:20.447860 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:20.447846 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8dfadf3e-94d3-4e77-bd97-5bb554b03d2f" path="/var/lib/kubelet/pods/8dfadf3e-94d3-4e77-bd97-5bb554b03d2f/volumes" Oct 31 05:00:20.448579 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:20.448564 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="978b4c58-1c25-4e04-bfbf-5cd4e0a7001d" path="/var/lib/kubelet/pods/978b4c58-1c25-4e04-bfbf-5cd4e0a7001d/volumes" Oct 31 05:00:20.448971 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:20.448957 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a63f4216-a71c-45b7-98b9-3b08500503fd" path="/var/lib/kubelet/pods/a63f4216-a71c-45b7-98b9-3b08500503fd/volumes" Oct 31 05:00:20.714968 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:20.714898 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-77a153-29890380-sdd6b" Oct 31 05:00:20.715433 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:20.714902 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-77a153-29890380-sdd6b" event={"ID":"09f6c274-8499-4b06-91af-6af516c3e56a","Type":"ContainerDied","Data":"d8e9f787ab252b262c82b223267f6c9c215088a13e7c927b5e86748e29c057b0"} Oct 31 05:00:20.715433 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:20.715036 4376 scope.go:122] "RemoveContainer" containerID="b9c2de8b318796f223d97db2d880e454cd33603fa797b58846325ce9c8e54aae" Oct 31 05:00:20.724804 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:20.724758 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-766728-29890380-9xphp" event={"ID":"9850fba4-11d3-4785-adc0-5ddd89293842","Type":"ContainerStarted","Data":"bcdfb085235b310bbf44c39ac9fbb77046280f24301a055b6afe3b54cc25b3e0"} Oct 31 05:00:20.890884 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:20.890855 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-77a153-29890380-sdd6b"] Oct 31 05:00:20.895720 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:20.895691 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-77a153-29890380-sdd6b"] Oct 31 05:00:20.925179 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:20.925146 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-171edd-29890380-2kgrc" Oct 31 05:00:20.934544 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:20.934521 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c1fe26-29890380-h89x7" Oct 31 05:00:21.023986 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:21.023947 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9633f0-29890380-jc5lw"] Oct 31 05:00:21.048286 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:21.048256 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ae817be-98c3-4760-b1ed-b545d5b24e0f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ae817be-98c3-4760-b1ed-b545d5b24e0f-maas-api-service-ca\") pod \"2ae817be-98c3-4760-b1ed-b545d5b24e0f\" (UID: \"2ae817be-98c3-4760-b1ed-b545d5b24e0f\") " Oct 31 05:00:21.048486 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:21.048361 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79b8c49d-28ac-49fa-b101-5f553fdac035-kube-api-access-jwm5b\" (UniqueName: \"kubernetes.io/projected/79b8c49d-28ac-49fa-b101-5f553fdac035-kube-api-access-jwm5b\") pod \"79b8c49d-28ac-49fa-b101-5f553fdac035\" (UID: \"79b8c49d-28ac-49fa-b101-5f553fdac035\") " Oct 31 05:00:21.048486 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:21.048403 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79b8c49d-28ac-49fa-b101-5f553fdac035-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79b8c49d-28ac-49fa-b101-5f553fdac035-maas-api-service-ca\") pod \"79b8c49d-28ac-49fa-b101-5f553fdac035\" (UID: \"79b8c49d-28ac-49fa-b101-5f553fdac035\") " Oct 31 05:00:21.048698 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:21.048672 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ae817be-98c3-4760-b1ed-b545d5b24e0f-maas-api-service-ca" pod "2ae817be-98c3-4760-b1ed-b545d5b24e0f" (UID: "2ae817be-98c3-4760-b1ed-b545d5b24e0f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:00:21.048760 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:21.048686 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ae817be-98c3-4760-b1ed-b545d5b24e0f-kube-api-access-lmgwj\" (UniqueName: \"kubernetes.io/projected/2ae817be-98c3-4760-b1ed-b545d5b24e0f-kube-api-access-lmgwj\") pod \"2ae817be-98c3-4760-b1ed-b545d5b24e0f\" (UID: \"2ae817be-98c3-4760-b1ed-b545d5b24e0f\") " Oct 31 05:00:21.049378 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:21.049319 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79b8c49d-28ac-49fa-b101-5f553fdac035-maas-api-service-ca" pod "79b8c49d-28ac-49fa-b101-5f553fdac035" (UID: "79b8c49d-28ac-49fa-b101-5f553fdac035"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:00:21.049650 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:21.049629 4376 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ae817be-98c3-4760-b1ed-b545d5b24e0f-maas-api-service-ca\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:00:21.049732 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:21.049655 4376 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79b8c49d-28ac-49fa-b101-5f553fdac035-maas-api-service-ca\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:00:21.052705 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:21.052158 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ae817be-98c3-4760-b1ed-b545d5b24e0f-kube-api-access-lmgwj" pod "2ae817be-98c3-4760-b1ed-b545d5b24e0f" (UID: "2ae817be-98c3-4760-b1ed-b545d5b24e0f"). InnerVolumeSpecName "kube-api-access-lmgwj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:00:21.053136 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:21.053112 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79b8c49d-28ac-49fa-b101-5f553fdac035-kube-api-access-jwm5b" pod "79b8c49d-28ac-49fa-b101-5f553fdac035" (UID: "79b8c49d-28ac-49fa-b101-5f553fdac035"). InnerVolumeSpecName "kube-api-access-jwm5b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:00:21.096273 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:21.096245 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-04d111-29890380-62zcn"] Oct 31 05:00:21.142502 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:21.141870 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-829ac7-29890380-pt7nf"] Oct 31 05:00:21.157828 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:21.152384 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lmgwj\" (UniqueName: \"kubernetes.io/projected/2ae817be-98c3-4760-b1ed-b545d5b24e0f-kube-api-access-lmgwj\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:00:21.157828 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:21.152420 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwm5b\" (UniqueName: \"kubernetes.io/projected/79b8c49d-28ac-49fa-b101-5f553fdac035-kube-api-access-jwm5b\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:00:21.248775 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:21.248703 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9633f0-29890380-jc5lw" Oct 31 05:00:21.353766 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:21.353733 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-04d111-29890380-62zcn" Oct 31 05:00:21.358813 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:21.358782 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecbbd44a-efa7-4e45-8514-1a57eccc71e5-kube-api-access-2l466\" (UniqueName: \"kubernetes.io/projected/ecbbd44a-efa7-4e45-8514-1a57eccc71e5-kube-api-access-2l466\") pod \"ecbbd44a-efa7-4e45-8514-1a57eccc71e5\" (UID: \"ecbbd44a-efa7-4e45-8514-1a57eccc71e5\") " Oct 31 05:00:21.358966 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:21.358884 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ecbbd44a-efa7-4e45-8514-1a57eccc71e5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ecbbd44a-efa7-4e45-8514-1a57eccc71e5-maas-api-service-ca\") pod \"ecbbd44a-efa7-4e45-8514-1a57eccc71e5\" (UID: \"ecbbd44a-efa7-4e45-8514-1a57eccc71e5\") " Oct 31 05:00:21.359323 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:21.359295 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecbbd44a-efa7-4e45-8514-1a57eccc71e5-maas-api-service-ca" pod "ecbbd44a-efa7-4e45-8514-1a57eccc71e5" (UID: "ecbbd44a-efa7-4e45-8514-1a57eccc71e5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:00:21.359797 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:21.359772 4376 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ecbbd44a-efa7-4e45-8514-1a57eccc71e5-maas-api-service-ca\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:00:21.361700 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:21.361664 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecbbd44a-efa7-4e45-8514-1a57eccc71e5-kube-api-access-2l466" pod "ecbbd44a-efa7-4e45-8514-1a57eccc71e5" (UID: "ecbbd44a-efa7-4e45-8514-1a57eccc71e5"). InnerVolumeSpecName "kube-api-access-2l466". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:00:21.381080 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:21.381057 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-829ac7-29890380-pt7nf" Oct 31 05:00:21.460790 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:21.460751 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9853a85-21be-4351-a76c-78d4da0aa7dc-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9853a85-21be-4351-a76c-78d4da0aa7dc-maas-api-service-ca\") pod \"a9853a85-21be-4351-a76c-78d4da0aa7dc\" (UID: \"a9853a85-21be-4351-a76c-78d4da0aa7dc\") " Oct 31 05:00:21.460982 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:21.460935 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9853a85-21be-4351-a76c-78d4da0aa7dc-kube-api-access-hxbxx\" (UniqueName: \"kubernetes.io/projected/a9853a85-21be-4351-a76c-78d4da0aa7dc-kube-api-access-hxbxx\") pod \"a9853a85-21be-4351-a76c-78d4da0aa7dc\" (UID: \"a9853a85-21be-4351-a76c-78d4da0aa7dc\") " Oct 31 05:00:21.461206 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:21.461177 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9853a85-21be-4351-a76c-78d4da0aa7dc-maas-api-service-ca" pod "a9853a85-21be-4351-a76c-78d4da0aa7dc" (UID: "a9853a85-21be-4351-a76c-78d4da0aa7dc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:00:21.461745 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:21.461719 4376 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9853a85-21be-4351-a76c-78d4da0aa7dc-maas-api-service-ca\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:00:21.461922 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:21.461749 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2l466\" (UniqueName: \"kubernetes.io/projected/ecbbd44a-efa7-4e45-8514-1a57eccc71e5-kube-api-access-2l466\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:00:21.463459 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:21.463437 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9853a85-21be-4351-a76c-78d4da0aa7dc-kube-api-access-hxbxx" pod "a9853a85-21be-4351-a76c-78d4da0aa7dc" (UID: "a9853a85-21be-4351-a76c-78d4da0aa7dc"). InnerVolumeSpecName "kube-api-access-hxbxx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:00:21.562812 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:21.562747 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b953e1c-6983-456f-b2c8-77e9314aa758-kube-api-access-jc96x\" (UniqueName: \"kubernetes.io/projected/7b953e1c-6983-456f-b2c8-77e9314aa758-kube-api-access-jc96x\") pod \"7b953e1c-6983-456f-b2c8-77e9314aa758\" (UID: \"7b953e1c-6983-456f-b2c8-77e9314aa758\") " Oct 31 05:00:21.562812 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:21.562809 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b953e1c-6983-456f-b2c8-77e9314aa758-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b953e1c-6983-456f-b2c8-77e9314aa758-maas-api-service-ca\") pod \"7b953e1c-6983-456f-b2c8-77e9314aa758\" (UID: \"7b953e1c-6983-456f-b2c8-77e9314aa758\") " Oct 31 05:00:21.563169 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:21.563143 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b953e1c-6983-456f-b2c8-77e9314aa758-maas-api-service-ca" pod "7b953e1c-6983-456f-b2c8-77e9314aa758" (UID: "7b953e1c-6983-456f-b2c8-77e9314aa758"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:00:21.563518 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:21.563498 4376 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b953e1c-6983-456f-b2c8-77e9314aa758-maas-api-service-ca\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:00:21.563585 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:21.563522 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxbxx\" (UniqueName: \"kubernetes.io/projected/a9853a85-21be-4351-a76c-78d4da0aa7dc-kube-api-access-hxbxx\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:00:21.564845 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:21.564827 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b953e1c-6983-456f-b2c8-77e9314aa758-kube-api-access-jc96x" pod "7b953e1c-6983-456f-b2c8-77e9314aa758" (UID: "7b953e1c-6983-456f-b2c8-77e9314aa758"). InnerVolumeSpecName "kube-api-access-jc96x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:00:21.664789 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:21.664752 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jc96x\" (UniqueName: \"kubernetes.io/projected/7b953e1c-6983-456f-b2c8-77e9314aa758-kube-api-access-jc96x\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:00:21.741167 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:21.741140 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-171edd-29890380-2kgrc" Oct 31 05:00:21.741559 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:21.741159 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-171edd-29890380-2kgrc" event={"ID":"2ae817be-98c3-4760-b1ed-b545d5b24e0f","Type":"ContainerDied","Data":"92655d5a98f23bedabe4a5a9b36f942850945d9ad291e9845fe8085e0ba1aaf2"} Oct 31 05:00:21.741559 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:21.741231 4376 scope.go:122] "RemoveContainer" containerID="63240c6d244fe984e6c23db016d4f2a8ee185e12da072b9350281bd25a429932" Oct 31 05:00:21.742806 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:21.742785 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c1fe26-29890380-h89x7" Oct 31 05:00:21.742915 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:21.742804 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c1fe26-29890380-h89x7" event={"ID":"79b8c49d-28ac-49fa-b101-5f553fdac035","Type":"ContainerDied","Data":"71d0900ca404e25e4fae7a6fba144d464879d17234f7551e751b5201bba0f151"} Oct 31 05:00:21.745159 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:21.745139 4376 generic.go:356] "Generic (PLEG): container finished" podID="9850fba4-11d3-4785-adc0-5ddd89293842" containerID="bcdfb085235b310bbf44c39ac9fbb77046280f24301a055b6afe3b54cc25b3e0" exitCode=60 Oct 31 05:00:21.745258 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:21.745213 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-766728-29890380-9xphp" event={"ID":"9850fba4-11d3-4785-adc0-5ddd89293842","Type":"ContainerDied","Data":"bcdfb085235b310bbf44c39ac9fbb77046280f24301a055b6afe3b54cc25b3e0"} Oct 31 05:00:21.746227 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:21.745703 4376 scope.go:122] "RemoveContainer" containerID="bcdfb085235b310bbf44c39ac9fbb77046280f24301a055b6afe3b54cc25b3e0" Oct 31 05:00:21.746909 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:21.746892 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9633f0-29890380-jc5lw" Oct 31 05:00:21.747001 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:21.746972 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9633f0-29890380-jc5lw" event={"ID":"ecbbd44a-efa7-4e45-8514-1a57eccc71e5","Type":"ContainerDied","Data":"05473bb79b6773f5ad44da92a8b7f7aa4d9b5f019917d60493de819a01c73c78"} Oct 31 05:00:21.749101 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:00:21.749074 4376 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-766728-29890380-9xphp_odh-ai-gateway-infra(9850fba4-11d3-4785-adc0-5ddd89293842)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-766728-29890380-9xphp" podUID="9850fba4-11d3-4785-adc0-5ddd89293842" Oct 31 05:00:21.755378 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:21.755363 4376 scope.go:122] "RemoveContainer" containerID="99ca664394ccf80df44d63d974b0524a98d5a65d945ace6c072411caa55f08af" Oct 31 05:00:21.755688 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:21.755642 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-04d111-29890380-62zcn" Oct 31 05:00:21.755688 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:21.755666 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-04d111-29890380-62zcn" event={"ID":"a9853a85-21be-4351-a76c-78d4da0aa7dc","Type":"ContainerDied","Data":"2b98b774a0a4d82c7af9e6d6675de8998399edb04ff95c3803c983b101b8e387"} Oct 31 05:00:21.757576 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:21.757466 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-829ac7-29890380-pt7nf" Oct 31 05:00:21.757576 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:21.757486 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-829ac7-29890380-pt7nf" event={"ID":"7b953e1c-6983-456f-b2c8-77e9314aa758","Type":"ContainerDied","Data":"437bacf525ad3a1a955c7599915ddab8d812739f5b9560197fe7df432d113641"} Oct 31 05:00:21.785111 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:21.783884 4376 scope.go:122] "RemoveContainer" containerID="387e7f9074661504d45b68a4748a0ee6ec00865d344dea2ae063710166430d78" Oct 31 05:00:21.804285 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:21.804237 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-766728-29890380-9xphp"] Oct 31 05:00:21.836539 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:21.836452 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c1fe26-29890380-h89x7"] Oct 31 05:00:21.848157 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:21.848134 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c1fe26-29890380-h89x7"] Oct 31 05:00:21.863363 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:21.857192 4376 scope.go:122] "RemoveContainer" containerID="e4b8ed9b88d53f2ca7ef97b35b15a7b6cc8b075f7ec3509fe1d0e905341e08d5" Oct 31 05:00:21.886519 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:21.886489 4376 scope.go:122] "RemoveContainer" containerID="b840ffb04eb4f9af37ca5624cbb78cbf2e25313d35b7ef3fa4f4f9c25b00d3a1" Oct 31 05:00:21.893711 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:21.893683 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-171edd-29890380-2kgrc"] Oct 31 05:00:21.899961 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:21.899938 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-171edd-29890380-2kgrc"] Oct 31 05:00:21.902993 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:21.902968 4376 scope.go:122] "RemoveContainer" containerID="3e282356e5f24d514fffd12af8d3983ef08e1b0cf209b1165e0d377ee66486d1" Oct 31 05:00:21.974716 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:21.974684 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9633f0-29890380-jc5lw"] Oct 31 05:00:21.981770 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:21.981656 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9633f0-29890380-jc5lw"] Oct 31 05:00:22.010593 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:22.010563 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-04d111-29890380-62zcn"] Oct 31 05:00:22.014499 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:22.014476 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-04d111-29890380-62zcn"] Oct 31 05:00:22.049249 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:22.049227 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-829ac7-29890380-pt7nf"] Oct 31 05:00:22.055702 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:22.055673 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-829ac7-29890380-pt7nf"] Oct 31 05:00:22.446022 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:22.445991 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09f6c274-8499-4b06-91af-6af516c3e56a" path="/var/lib/kubelet/pods/09f6c274-8499-4b06-91af-6af516c3e56a/volumes" Oct 31 05:00:22.446483 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:22.446467 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ae817be-98c3-4760-b1ed-b545d5b24e0f" path="/var/lib/kubelet/pods/2ae817be-98c3-4760-b1ed-b545d5b24e0f/volumes" Oct 31 05:00:22.446891 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:22.446879 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79b8c49d-28ac-49fa-b101-5f553fdac035" path="/var/lib/kubelet/pods/79b8c49d-28ac-49fa-b101-5f553fdac035/volumes" Oct 31 05:00:22.447608 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:22.447594 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b953e1c-6983-456f-b2c8-77e9314aa758" path="/var/lib/kubelet/pods/7b953e1c-6983-456f-b2c8-77e9314aa758/volumes" Oct 31 05:00:22.447984 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:22.447972 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9853a85-21be-4351-a76c-78d4da0aa7dc" path="/var/lib/kubelet/pods/a9853a85-21be-4351-a76c-78d4da0aa7dc/volumes" Oct 31 05:00:22.448372 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:22.448357 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ecbbd44a-efa7-4e45-8514-1a57eccc71e5" path="/var/lib/kubelet/pods/ecbbd44a-efa7-4e45-8514-1a57eccc71e5/volumes" Oct 31 05:00:22.954931 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:22.954904 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-766728-29890380-9xphp" Oct 31 05:00:23.005819 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:23.005793 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9850fba4-11d3-4785-adc0-5ddd89293842-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9850fba4-11d3-4785-adc0-5ddd89293842-maas-api-service-ca\") pod \"9850fba4-11d3-4785-adc0-5ddd89293842\" (UID: \"9850fba4-11d3-4785-adc0-5ddd89293842\") " Oct 31 05:00:23.005982 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:23.005962 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9850fba4-11d3-4785-adc0-5ddd89293842-kube-api-access-xcbr5\" (UniqueName: \"kubernetes.io/projected/9850fba4-11d3-4785-adc0-5ddd89293842-kube-api-access-xcbr5\") pod \"9850fba4-11d3-4785-adc0-5ddd89293842\" (UID: \"9850fba4-11d3-4785-adc0-5ddd89293842\") " Oct 31 05:00:23.006174 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:23.006154 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9850fba4-11d3-4785-adc0-5ddd89293842-maas-api-service-ca" pod "9850fba4-11d3-4785-adc0-5ddd89293842" (UID: "9850fba4-11d3-4785-adc0-5ddd89293842"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:00:23.006681 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:23.006659 4376 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9850fba4-11d3-4785-adc0-5ddd89293842-maas-api-service-ca\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:00:23.009432 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:23.009404 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9850fba4-11d3-4785-adc0-5ddd89293842-kube-api-access-xcbr5" pod "9850fba4-11d3-4785-adc0-5ddd89293842" (UID: "9850fba4-11d3-4785-adc0-5ddd89293842"). InnerVolumeSpecName "kube-api-access-xcbr5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:00:23.108135 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:23.108060 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xcbr5\" (UniqueName: \"kubernetes.io/projected/9850fba4-11d3-4785-adc0-5ddd89293842-kube-api-access-xcbr5\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:00:23.782343 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:23.782296 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-766728-29890380-9xphp" Oct 31 05:00:23.782343 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:23.782317 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-766728-29890380-9xphp" event={"ID":"9850fba4-11d3-4785-adc0-5ddd89293842","Type":"ContainerDied","Data":"ded170c2b38c7241da47302b565dbf738496cb6720aab7dc89bc4ec7bf7f5329"} Oct 31 05:00:23.782574 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:23.782406 4376 scope.go:122] "RemoveContainer" containerID="bcdfb085235b310bbf44c39ac9fbb77046280f24301a055b6afe3b54cc25b3e0" Oct 31 05:00:23.846208 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:23.846178 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-766728-29890380-9xphp"] Oct 31 05:00:23.850408 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:23.850386 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-766728-29890380-9xphp"] Oct 31 05:00:24.453242 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:24.453206 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9850fba4-11d3-4785-adc0-5ddd89293842" path="/var/lib/kubelet/pods/9850fba4-11d3-4785-adc0-5ddd89293842/volumes" Oct 31 05:00:31.728302 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:31.728271 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-xwqm7"] Oct 31 05:00:31.728829 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:31.728513 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-xwqm7" podUID="69c1b903-0719-4813-9e1e-11e87818ec69" containerName="istio-proxy" containerID="cri-o://ff8b6101a553eebb447b95c7ef365f91f985170deab1f4fc7a18aeeb428a8e50" gracePeriod=30 Oct 31 05:00:36.896744 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:36.896710 4376 generic.go:356] "Generic (PLEG): container finished" podID="69c1b903-0719-4813-9e1e-11e87818ec69" containerID="ff8b6101a553eebb447b95c7ef365f91f985170deab1f4fc7a18aeeb428a8e50" exitCode=0 Oct 31 05:00:36.897165 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:36.896786 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-xwqm7" event={"ID":"69c1b903-0719-4813-9e1e-11e87818ec69","Type":"ContainerDied","Data":"ff8b6101a553eebb447b95c7ef365f91f985170deab1f4fc7a18aeeb428a8e50"} Oct 31 05:00:37.023590 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:37.023526 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-xwqm7" Oct 31 05:00:37.072167 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:37.072140 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/69c1b903-0719-4813-9e1e-11e87818ec69-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69c1b903-0719-4813-9e1e-11e87818ec69-istio-podinfo\") pod \"69c1b903-0719-4813-9e1e-11e87818ec69\" (UID: \"69c1b903-0719-4813-9e1e-11e87818ec69\") " Oct 31 05:00:37.072370 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:37.072286 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69c1b903-0719-4813-9e1e-11e87818ec69-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69c1b903-0719-4813-9e1e-11e87818ec69-istio-envoy\") pod \"69c1b903-0719-4813-9e1e-11e87818ec69\" (UID: \"69c1b903-0719-4813-9e1e-11e87818ec69\") " Oct 31 05:00:37.072453 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:37.072368 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69c1b903-0719-4813-9e1e-11e87818ec69-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69c1b903-0719-4813-9e1e-11e87818ec69-istio-data\") pod \"69c1b903-0719-4813-9e1e-11e87818ec69\" (UID: \"69c1b903-0719-4813-9e1e-11e87818ec69\") " Oct 31 05:00:37.072453 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:37.072436 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69c1b903-0719-4813-9e1e-11e87818ec69-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69c1b903-0719-4813-9e1e-11e87818ec69-workload-certs\") pod \"69c1b903-0719-4813-9e1e-11e87818ec69\" (UID: \"69c1b903-0719-4813-9e1e-11e87818ec69\") " Oct 31 05:00:37.072584 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:37.072478 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69c1b903-0719-4813-9e1e-11e87818ec69-kube-api-access-7q7hv\" (UniqueName: \"kubernetes.io/projected/69c1b903-0719-4813-9e1e-11e87818ec69-kube-api-access-7q7hv\") pod \"69c1b903-0719-4813-9e1e-11e87818ec69\" (UID: \"69c1b903-0719-4813-9e1e-11e87818ec69\") " Oct 31 05:00:37.072584 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:37.072552 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69c1b903-0719-4813-9e1e-11e87818ec69-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69c1b903-0719-4813-9e1e-11e87818ec69-credential-socket\") pod \"69c1b903-0719-4813-9e1e-11e87818ec69\" (UID: \"69c1b903-0719-4813-9e1e-11e87818ec69\") " Oct 31 05:00:37.072689 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:37.072604 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69c1b903-0719-4813-9e1e-11e87818ec69-istio-token\" (UniqueName: \"kubernetes.io/projected/69c1b903-0719-4813-9e1e-11e87818ec69-istio-token\") pod \"69c1b903-0719-4813-9e1e-11e87818ec69\" (UID: \"69c1b903-0719-4813-9e1e-11e87818ec69\") " Oct 31 05:00:37.072689 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:37.072670 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69c1b903-0719-4813-9e1e-11e87818ec69-istio-data" pod "69c1b903-0719-4813-9e1e-11e87818ec69" (UID: "69c1b903-0719-4813-9e1e-11e87818ec69"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:00:37.072798 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:37.072688 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69c1b903-0719-4813-9e1e-11e87818ec69-workload-certs" pod "69c1b903-0719-4813-9e1e-11e87818ec69" (UID: "69c1b903-0719-4813-9e1e-11e87818ec69"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:00:37.072798 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:37.072703 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69c1b903-0719-4813-9e1e-11e87818ec69-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69c1b903-0719-4813-9e1e-11e87818ec69-istiod-ca-cert\") pod \"69c1b903-0719-4813-9e1e-11e87818ec69\" (UID: \"69c1b903-0719-4813-9e1e-11e87818ec69\") " Oct 31 05:00:37.072921 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:37.072838 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69c1b903-0719-4813-9e1e-11e87818ec69-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69c1b903-0719-4813-9e1e-11e87818ec69-workload-socket\") pod \"69c1b903-0719-4813-9e1e-11e87818ec69\" (UID: \"69c1b903-0719-4813-9e1e-11e87818ec69\") " Oct 31 05:00:37.073013 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:37.072910 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69c1b903-0719-4813-9e1e-11e87818ec69-credential-socket" pod "69c1b903-0719-4813-9e1e-11e87818ec69" (UID: "69c1b903-0719-4813-9e1e-11e87818ec69"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:00:37.073111 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:37.073088 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69c1b903-0719-4813-9e1e-11e87818ec69-workload-socket" pod "69c1b903-0719-4813-9e1e-11e87818ec69" (UID: "69c1b903-0719-4813-9e1e-11e87818ec69"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:00:37.073383 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:37.073359 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69c1b903-0719-4813-9e1e-11e87818ec69-istiod-ca-cert" pod "69c1b903-0719-4813-9e1e-11e87818ec69" (UID: "69c1b903-0719-4813-9e1e-11e87818ec69"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:00:37.073728 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:37.073707 4376 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69c1b903-0719-4813-9e1e-11e87818ec69-workload-certs\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:00:37.073813 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:37.073735 4376 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69c1b903-0719-4813-9e1e-11e87818ec69-credential-socket\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:00:37.073813 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:37.073751 4376 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69c1b903-0719-4813-9e1e-11e87818ec69-istiod-ca-cert\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:00:37.073813 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:37.073766 4376 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69c1b903-0719-4813-9e1e-11e87818ec69-workload-socket\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:00:37.073813 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:37.073780 4376 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69c1b903-0719-4813-9e1e-11e87818ec69-istio-data\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:00:37.074558 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:37.074531 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69c1b903-0719-4813-9e1e-11e87818ec69-istio-envoy" pod "69c1b903-0719-4813-9e1e-11e87818ec69" (UID: "69c1b903-0719-4813-9e1e-11e87818ec69"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:00:37.074708 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:37.074692 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/69c1b903-0719-4813-9e1e-11e87818ec69-istio-podinfo" pod "69c1b903-0719-4813-9e1e-11e87818ec69" (UID: "69c1b903-0719-4813-9e1e-11e87818ec69"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 05:00:37.074770 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:37.074720 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69c1b903-0719-4813-9e1e-11e87818ec69-kube-api-access-7q7hv" pod "69c1b903-0719-4813-9e1e-11e87818ec69" (UID: "69c1b903-0719-4813-9e1e-11e87818ec69"). InnerVolumeSpecName "kube-api-access-7q7hv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:00:37.074770 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:37.074740 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69c1b903-0719-4813-9e1e-11e87818ec69-istio-token" pod "69c1b903-0719-4813-9e1e-11e87818ec69" (UID: "69c1b903-0719-4813-9e1e-11e87818ec69"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:00:37.174964 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:37.174935 4376 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69c1b903-0719-4813-9e1e-11e87818ec69-istio-token\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:00:37.174964 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:37.174961 4376 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69c1b903-0719-4813-9e1e-11e87818ec69-istio-podinfo\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:00:37.174964 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:37.174970 4376 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69c1b903-0719-4813-9e1e-11e87818ec69-istio-envoy\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:00:37.175203 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:37.174978 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7q7hv\" (UniqueName: \"kubernetes.io/projected/69c1b903-0719-4813-9e1e-11e87818ec69-kube-api-access-7q7hv\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:00:37.905661 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:37.905630 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-xwqm7" Oct 31 05:00:37.906069 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:37.905676 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-xwqm7" event={"ID":"69c1b903-0719-4813-9e1e-11e87818ec69","Type":"ContainerDied","Data":"8073b58d3d7901f5e70a386d8de5513664e81a103d3f9eeba253286871d67a76"} Oct 31 05:00:37.906069 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:37.905741 4376 scope.go:122] "RemoveContainer" containerID="ff8b6101a553eebb447b95c7ef365f91f985170deab1f4fc7a18aeeb428a8e50" Oct 31 05:00:37.974932 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:37.974901 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-xwqm7"] Oct 31 05:00:37.990739 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:37.990713 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-xwqm7"] Oct 31 05:00:38.448399 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:00:38.448358 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69c1b903-0719-4813-9e1e-11e87818ec69" path="/var/lib/kubelet/pods/69c1b903-0719-4813-9e1e-11e87818ec69/volumes" Oct 31 05:00:41.000559 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:00:41.000520 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 05:01:10.358738 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:01:10.358702 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-99pz4"] Oct 31 05:01:10.359252 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:01:10.358942 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-99pz4" podUID="2fa45335-eb01-48ef-bdd2-2a32785614d5" containerName="istio-proxy" containerID="cri-o://85be22f43c677d624717f8f64f0de22274b17c67b6e946db24ce0084c6a66446" gracePeriod=30 Oct 31 05:01:11.001075 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:01:11.001038 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 05:01:15.066286 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:01:15.066254 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-kc5qw"] Oct 31 05:01:15.066738 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:01:15.066512 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-kc5qw" podUID="19a7ea83-1d36-494a-ab52-f24535365ff9" containerName="istio-proxy" containerID="cri-o://ff3dbd23adb8f169e27436691ba272353a719f518e28fe4cb18bcbb7ab5e317d" gracePeriod=30 Oct 31 05:01:15.643603 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:01:15.643545 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-99pz4" Oct 31 05:01:15.689020 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:01:15.688991 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fa45335-eb01-48ef-bdd2-2a32785614d5-kube-api-access-5nc2r\" (UniqueName: \"kubernetes.io/projected/2fa45335-eb01-48ef-bdd2-2a32785614d5-kube-api-access-5nc2r\") pod \"2fa45335-eb01-48ef-bdd2-2a32785614d5\" (UID: \"2fa45335-eb01-48ef-bdd2-2a32785614d5\") " Oct 31 05:01:15.689169 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:01:15.689054 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fa45335-eb01-48ef-bdd2-2a32785614d5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fa45335-eb01-48ef-bdd2-2a32785614d5-istio-envoy\") pod \"2fa45335-eb01-48ef-bdd2-2a32785614d5\" (UID: \"2fa45335-eb01-48ef-bdd2-2a32785614d5\") " Oct 31 05:01:15.689169 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:01:15.689088 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fa45335-eb01-48ef-bdd2-2a32785614d5-istio-token\" (UniqueName: \"kubernetes.io/projected/2fa45335-eb01-48ef-bdd2-2a32785614d5-istio-token\") pod \"2fa45335-eb01-48ef-bdd2-2a32785614d5\" (UID: \"2fa45335-eb01-48ef-bdd2-2a32785614d5\") " Oct 31 05:01:15.689169 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:01:15.689116 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fa45335-eb01-48ef-bdd2-2a32785614d5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fa45335-eb01-48ef-bdd2-2a32785614d5-istio-data\") pod \"2fa45335-eb01-48ef-bdd2-2a32785614d5\" (UID: \"2fa45335-eb01-48ef-bdd2-2a32785614d5\") " Oct 31 05:01:15.689346 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:01:15.689175 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fa45335-eb01-48ef-bdd2-2a32785614d5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fa45335-eb01-48ef-bdd2-2a32785614d5-credential-socket\") pod \"2fa45335-eb01-48ef-bdd2-2a32785614d5\" (UID: \"2fa45335-eb01-48ef-bdd2-2a32785614d5\") " Oct 31 05:01:15.689346 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:01:15.689228 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2fa45335-eb01-48ef-bdd2-2a32785614d5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fa45335-eb01-48ef-bdd2-2a32785614d5-istiod-ca-cert\") pod \"2fa45335-eb01-48ef-bdd2-2a32785614d5\" (UID: \"2fa45335-eb01-48ef-bdd2-2a32785614d5\") " Oct 31 05:01:15.689346 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:01:15.689297 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2fa45335-eb01-48ef-bdd2-2a32785614d5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fa45335-eb01-48ef-bdd2-2a32785614d5-istio-podinfo\") pod \"2fa45335-eb01-48ef-bdd2-2a32785614d5\" (UID: \"2fa45335-eb01-48ef-bdd2-2a32785614d5\") " Oct 31 05:01:15.689635 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:01:15.689435 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fa45335-eb01-48ef-bdd2-2a32785614d5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fa45335-eb01-48ef-bdd2-2a32785614d5-workload-socket\") pod \"2fa45335-eb01-48ef-bdd2-2a32785614d5\" (UID: \"2fa45335-eb01-48ef-bdd2-2a32785614d5\") " Oct 31 05:01:15.689635 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:01:15.689477 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fa45335-eb01-48ef-bdd2-2a32785614d5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fa45335-eb01-48ef-bdd2-2a32785614d5-workload-certs\") pod \"2fa45335-eb01-48ef-bdd2-2a32785614d5\" (UID: \"2fa45335-eb01-48ef-bdd2-2a32785614d5\") " Oct 31 05:01:15.689635 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:01:15.689511 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fa45335-eb01-48ef-bdd2-2a32785614d5-istio-data" pod "2fa45335-eb01-48ef-bdd2-2a32785614d5" (UID: "2fa45335-eb01-48ef-bdd2-2a32785614d5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:01:15.689635 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:01:15.689496 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fa45335-eb01-48ef-bdd2-2a32785614d5-credential-socket" pod "2fa45335-eb01-48ef-bdd2-2a32785614d5" (UID: "2fa45335-eb01-48ef-bdd2-2a32785614d5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:01:15.689869 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:01:15.689752 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fa45335-eb01-48ef-bdd2-2a32785614d5-workload-socket" pod "2fa45335-eb01-48ef-bdd2-2a32785614d5" (UID: "2fa45335-eb01-48ef-bdd2-2a32785614d5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:01:15.689869 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:01:15.689831 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fa45335-eb01-48ef-bdd2-2a32785614d5-workload-certs" pod "2fa45335-eb01-48ef-bdd2-2a32785614d5" (UID: "2fa45335-eb01-48ef-bdd2-2a32785614d5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:01:15.689995 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:01:15.689969 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fa45335-eb01-48ef-bdd2-2a32785614d5-istiod-ca-cert" pod "2fa45335-eb01-48ef-bdd2-2a32785614d5" (UID: "2fa45335-eb01-48ef-bdd2-2a32785614d5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:01:15.690523 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:01:15.690430 4376 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fa45335-eb01-48ef-bdd2-2a32785614d5-istio-data\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:01:15.690523 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:01:15.690458 4376 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fa45335-eb01-48ef-bdd2-2a32785614d5-credential-socket\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:01:15.690523 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:01:15.690474 4376 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fa45335-eb01-48ef-bdd2-2a32785614d5-istiod-ca-cert\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:01:15.690523 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:01:15.690486 4376 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fa45335-eb01-48ef-bdd2-2a32785614d5-workload-socket\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:01:15.690523 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:01:15.690500 4376 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fa45335-eb01-48ef-bdd2-2a32785614d5-workload-certs\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:01:15.693045 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:01:15.692988 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fa45335-eb01-48ef-bdd2-2a32785614d5-istio-token" pod "2fa45335-eb01-48ef-bdd2-2a32785614d5" (UID: "2fa45335-eb01-48ef-bdd2-2a32785614d5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:01:15.693546 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:01:15.693243 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fa45335-eb01-48ef-bdd2-2a32785614d5-istio-envoy" pod "2fa45335-eb01-48ef-bdd2-2a32785614d5" (UID: "2fa45335-eb01-48ef-bdd2-2a32785614d5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:01:15.693546 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:01:15.693364 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2fa45335-eb01-48ef-bdd2-2a32785614d5-istio-podinfo" pod "2fa45335-eb01-48ef-bdd2-2a32785614d5" (UID: "2fa45335-eb01-48ef-bdd2-2a32785614d5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 05:01:15.693546 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:01:15.693404 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fa45335-eb01-48ef-bdd2-2a32785614d5-kube-api-access-5nc2r" pod "2fa45335-eb01-48ef-bdd2-2a32785614d5" (UID: "2fa45335-eb01-48ef-bdd2-2a32785614d5"). InnerVolumeSpecName "kube-api-access-5nc2r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:01:15.792486 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:01:15.792446 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5nc2r\" (UniqueName: \"kubernetes.io/projected/2fa45335-eb01-48ef-bdd2-2a32785614d5-kube-api-access-5nc2r\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:01:15.792486 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:01:15.792475 4376 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fa45335-eb01-48ef-bdd2-2a32785614d5-istio-envoy\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:01:15.792486 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:01:15.792488 4376 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2fa45335-eb01-48ef-bdd2-2a32785614d5-istio-token\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:01:15.792800 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:01:15.792501 4376 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fa45335-eb01-48ef-bdd2-2a32785614d5-istio-podinfo\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:01:16.232177 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:01:16.232141 4376 generic.go:356] "Generic (PLEG): container finished" podID="2fa45335-eb01-48ef-bdd2-2a32785614d5" containerID="85be22f43c677d624717f8f64f0de22274b17c67b6e946db24ce0084c6a66446" exitCode=0 Oct 31 05:01:16.232682 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:01:16.232193 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-99pz4" Oct 31 05:01:16.232682 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:01:16.232221 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-99pz4" event={"ID":"2fa45335-eb01-48ef-bdd2-2a32785614d5","Type":"ContainerDied","Data":"85be22f43c677d624717f8f64f0de22274b17c67b6e946db24ce0084c6a66446"} Oct 31 05:01:16.232682 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:01:16.232274 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-99pz4" event={"ID":"2fa45335-eb01-48ef-bdd2-2a32785614d5","Type":"ContainerDied","Data":"6b947bb817e26268dcd3dca981a709cb7189f4e7d63d1e1cac55eeca6794e802"} Oct 31 05:01:16.232682 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:01:16.232296 4376 scope.go:122] "RemoveContainer" containerID="85be22f43c677d624717f8f64f0de22274b17c67b6e946db24ce0084c6a66446" Oct 31 05:01:16.246807 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:01:16.246789 4376 scope.go:122] "RemoveContainer" containerID="85be22f43c677d624717f8f64f0de22274b17c67b6e946db24ce0084c6a66446" Oct 31 05:01:16.247089 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:01:16.247072 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85be22f43c677d624717f8f64f0de22274b17c67b6e946db24ce0084c6a66446\": container with ID starting with 85be22f43c677d624717f8f64f0de22274b17c67b6e946db24ce0084c6a66446 not found: ID does not exist" containerID="85be22f43c677d624717f8f64f0de22274b17c67b6e946db24ce0084c6a66446" Oct 31 05:01:16.247139 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:01:16.247099 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85be22f43c677d624717f8f64f0de22274b17c67b6e946db24ce0084c6a66446"} err="failed to get container status \"85be22f43c677d624717f8f64f0de22274b17c67b6e946db24ce0084c6a66446\": rpc error: code = NotFound desc = could not find container \"85be22f43c677d624717f8f64f0de22274b17c67b6e946db24ce0084c6a66446\": container with ID starting with 85be22f43c677d624717f8f64f0de22274b17c67b6e946db24ce0084c6a66446 not found: ID does not exist" Oct 31 05:01:16.292855 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:01:16.292831 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-99pz4"] Oct 31 05:01:16.296810 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:01:16.296790 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-99pz4"] Oct 31 05:01:16.446207 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:01:16.446175 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2fa45335-eb01-48ef-bdd2-2a32785614d5" path="/var/lib/kubelet/pods/2fa45335-eb01-48ef-bdd2-2a32785614d5/volumes" Oct 31 05:01:18.046071 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:01:18.046038 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-8973e2-openshift-default-6bd79597f7-pdlqd"] Oct 31 05:01:18.046569 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:01:18.046207 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-8973e2-openshift-default-6bd79597f7-pdlqd" podUID="69b79bb1-ccf1-4322-a0bd-1a307906c83a" containerName="istio-proxy" containerID="cri-o://4d4fa08898e985e3990b3c6be18814e244301190358a2585b835da704a133ab7" gracePeriod=30 Oct 31 05:01:19.801141 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:01:19.801110 4376 patch_prober.go:29] interesting pod/e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-kc5qw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 05:01:19.801555 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:01:19.801177 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-kc5qw" podUID="19a7ea83-1d36-494a-ab52-f24535365ff9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 05:01:20.287985 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:01:20.287944 4376 generic.go:356] "Generic (PLEG): container finished" podID="19a7ea83-1d36-494a-ab52-f24535365ff9" containerID="ff3dbd23adb8f169e27436691ba272353a719f518e28fe4cb18bcbb7ab5e317d" exitCode=0 Oct 31 05:01:20.288147 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:01:20.288028 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-kc5qw" event={"ID":"19a7ea83-1d36-494a-ab52-f24535365ff9","Type":"ContainerDied","Data":"ff3dbd23adb8f169e27436691ba272353a719f518e28fe4cb18bcbb7ab5e317d"} Oct 31 05:01:20.354219 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:01:20.354191 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-kc5qw" Oct 31 05:01:20.480625 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:01:20.480597 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19a7ea83-1d36-494a-ab52-f24535365ff9-kube-api-access-hp8rf\" (UniqueName: \"kubernetes.io/projected/19a7ea83-1d36-494a-ab52-f24535365ff9-kube-api-access-hp8rf\") pod \"19a7ea83-1d36-494a-ab52-f24535365ff9\" (UID: \"19a7ea83-1d36-494a-ab52-f24535365ff9\") " Oct 31 05:01:20.480801 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:01:20.480654 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19a7ea83-1d36-494a-ab52-f24535365ff9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19a7ea83-1d36-494a-ab52-f24535365ff9-credential-socket\") pod \"19a7ea83-1d36-494a-ab52-f24535365ff9\" (UID: \"19a7ea83-1d36-494a-ab52-f24535365ff9\") " Oct 31 05:01:20.480801 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:01:20.480723 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19a7ea83-1d36-494a-ab52-f24535365ff9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19a7ea83-1d36-494a-ab52-f24535365ff9-istiod-ca-cert\") pod \"19a7ea83-1d36-494a-ab52-f24535365ff9\" (UID: \"19a7ea83-1d36-494a-ab52-f24535365ff9\") " Oct 31 05:01:20.480889 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:01:20.480873 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19a7ea83-1d36-494a-ab52-f24535365ff9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19a7ea83-1d36-494a-ab52-f24535365ff9-istio-envoy\") pod \"19a7ea83-1d36-494a-ab52-f24535365ff9\" (UID: \"19a7ea83-1d36-494a-ab52-f24535365ff9\") " Oct 31 05:01:20.480950 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:01:20.480907 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19a7ea83-1d36-494a-ab52-f24535365ff9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19a7ea83-1d36-494a-ab52-f24535365ff9-workload-certs\") pod \"19a7ea83-1d36-494a-ab52-f24535365ff9\" (UID: \"19a7ea83-1d36-494a-ab52-f24535365ff9\") " Oct 31 05:01:20.481034 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:01:20.481018 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19a7ea83-1d36-494a-ab52-f24535365ff9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19a7ea83-1d36-494a-ab52-f24535365ff9-workload-socket\") pod \"19a7ea83-1d36-494a-ab52-f24535365ff9\" (UID: \"19a7ea83-1d36-494a-ab52-f24535365ff9\") " Oct 31 05:01:20.481099 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:01:20.481046 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19a7ea83-1d36-494a-ab52-f24535365ff9-credential-socket" pod "19a7ea83-1d36-494a-ab52-f24535365ff9" (UID: "19a7ea83-1d36-494a-ab52-f24535365ff9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:01:20.481099 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:01:20.481069 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19a7ea83-1d36-494a-ab52-f24535365ff9-istio-token\" (UniqueName: \"kubernetes.io/projected/19a7ea83-1d36-494a-ab52-f24535365ff9-istio-token\") pod \"19a7ea83-1d36-494a-ab52-f24535365ff9\" (UID: \"19a7ea83-1d36-494a-ab52-f24535365ff9\") " Oct 31 05:01:20.481213 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:01:20.481140 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19a7ea83-1d36-494a-ab52-f24535365ff9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19a7ea83-1d36-494a-ab52-f24535365ff9-istio-data\") pod \"19a7ea83-1d36-494a-ab52-f24535365ff9\" (UID: \"19a7ea83-1d36-494a-ab52-f24535365ff9\") " Oct 31 05:01:20.481213 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:01:20.481159 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19a7ea83-1d36-494a-ab52-f24535365ff9-istiod-ca-cert" pod "19a7ea83-1d36-494a-ab52-f24535365ff9" (UID: "19a7ea83-1d36-494a-ab52-f24535365ff9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:01:20.481314 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:01:20.481206 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19a7ea83-1d36-494a-ab52-f24535365ff9-workload-certs" pod "19a7ea83-1d36-494a-ab52-f24535365ff9" (UID: "19a7ea83-1d36-494a-ab52-f24535365ff9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:01:20.481314 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:01:20.481252 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/19a7ea83-1d36-494a-ab52-f24535365ff9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19a7ea83-1d36-494a-ab52-f24535365ff9-istio-podinfo\") pod \"19a7ea83-1d36-494a-ab52-f24535365ff9\" (UID: \"19a7ea83-1d36-494a-ab52-f24535365ff9\") " Oct 31 05:01:20.481540 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:01:20.481519 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19a7ea83-1d36-494a-ab52-f24535365ff9-istio-data" pod "19a7ea83-1d36-494a-ab52-f24535365ff9" (UID: "19a7ea83-1d36-494a-ab52-f24535365ff9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:01:20.481622 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:01:20.481606 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19a7ea83-1d36-494a-ab52-f24535365ff9-workload-socket" pod "19a7ea83-1d36-494a-ab52-f24535365ff9" (UID: "19a7ea83-1d36-494a-ab52-f24535365ff9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:01:20.482013 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:01:20.481990 4376 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19a7ea83-1d36-494a-ab52-f24535365ff9-credential-socket\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:01:20.482121 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:01:20.482017 4376 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19a7ea83-1d36-494a-ab52-f24535365ff9-istiod-ca-cert\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:01:20.482121 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:01:20.482033 4376 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19a7ea83-1d36-494a-ab52-f24535365ff9-workload-certs\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:01:20.482121 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:01:20.482047 4376 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19a7ea83-1d36-494a-ab52-f24535365ff9-workload-socket\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:01:20.482121 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:01:20.482061 4376 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19a7ea83-1d36-494a-ab52-f24535365ff9-istio-data\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:01:20.483409 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:01:20.483377 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19a7ea83-1d36-494a-ab52-f24535365ff9-istio-envoy" pod "19a7ea83-1d36-494a-ab52-f24535365ff9" (UID: "19a7ea83-1d36-494a-ab52-f24535365ff9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:01:20.483528 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:01:20.483420 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19a7ea83-1d36-494a-ab52-f24535365ff9-kube-api-access-hp8rf" pod "19a7ea83-1d36-494a-ab52-f24535365ff9" (UID: "19a7ea83-1d36-494a-ab52-f24535365ff9"). InnerVolumeSpecName "kube-api-access-hp8rf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:01:20.483528 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:01:20.483430 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/19a7ea83-1d36-494a-ab52-f24535365ff9-istio-podinfo" pod "19a7ea83-1d36-494a-ab52-f24535365ff9" (UID: "19a7ea83-1d36-494a-ab52-f24535365ff9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 05:01:20.483528 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:01:20.483463 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19a7ea83-1d36-494a-ab52-f24535365ff9-istio-token" pod "19a7ea83-1d36-494a-ab52-f24535365ff9" (UID: "19a7ea83-1d36-494a-ab52-f24535365ff9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:01:20.583805 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:01:20.583725 4376 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19a7ea83-1d36-494a-ab52-f24535365ff9-istio-envoy\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:01:20.583805 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:01:20.583750 4376 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19a7ea83-1d36-494a-ab52-f24535365ff9-istio-token\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:01:20.583805 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:01:20.583761 4376 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19a7ea83-1d36-494a-ab52-f24535365ff9-istio-podinfo\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:01:20.583805 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:01:20.583773 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hp8rf\" (UniqueName: \"kubernetes.io/projected/19a7ea83-1d36-494a-ab52-f24535365ff9-kube-api-access-hp8rf\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:01:21.302428 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:01:21.302397 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-kc5qw" Oct 31 05:01:21.303173 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:01:21.302393 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-kc5qw" event={"ID":"19a7ea83-1d36-494a-ab52-f24535365ff9","Type":"ContainerDied","Data":"59e64d053c8c3efcf67e8c460a60a67502db9e5b9f32ac6d7b5f1bdcacb50ff1"} Oct 31 05:01:21.303173 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:01:21.302549 4376 scope.go:122] "RemoveContainer" containerID="ff3dbd23adb8f169e27436691ba272353a719f518e28fe4cb18bcbb7ab5e317d" Oct 31 05:01:21.378017 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:01:21.377994 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-kc5qw"] Oct 31 05:01:21.387607 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:01:21.387575 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-kc5qw"] Oct 31 05:01:22.446102 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:01:22.446070 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19a7ea83-1d36-494a-ab52-f24535365ff9" path="/var/lib/kubelet/pods/19a7ea83-1d36-494a-ab52-f24535365ff9/volumes" Oct 31 05:01:23.321027 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:01:23.320999 4376 generic.go:356] "Generic (PLEG): container finished" podID="69b79bb1-ccf1-4322-a0bd-1a307906c83a" containerID="4d4fa08898e985e3990b3c6be18814e244301190358a2585b835da704a133ab7" exitCode=0 Oct 31 05:01:23.321170 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:01:23.321076 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-8973e2-openshift-default-6bd79597f7-pdlqd" event={"ID":"69b79bb1-ccf1-4322-a0bd-1a307906c83a","Type":"ContainerDied","Data":"4d4fa08898e985e3990b3c6be18814e244301190358a2585b835da704a133ab7"} Oct 31 05:01:23.321170 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:01:23.321119 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-8973e2-openshift-default-6bd79597f7-pdlqd" event={"ID":"69b79bb1-ccf1-4322-a0bd-1a307906c83a","Type":"ContainerDied","Data":"858291b244b1069df9bf6cf6e3c042c0ed0b2f012258e03d5047303d8b35ef47"} Oct 31 05:01:23.321170 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:01:23.321139 4376 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="858291b244b1069df9bf6cf6e3c042c0ed0b2f012258e03d5047303d8b35ef47" Oct 31 05:01:23.338148 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:01:23.338125 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-8973e2-openshift-default-6bd79597f7-pdlqd" Oct 31 05:01:23.422731 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:01:23.422699 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69b79bb1-ccf1-4322-a0bd-1a307906c83a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69b79bb1-ccf1-4322-a0bd-1a307906c83a-credential-socket\") pod \"69b79bb1-ccf1-4322-a0bd-1a307906c83a\" (UID: \"69b79bb1-ccf1-4322-a0bd-1a307906c83a\") " Oct 31 05:01:23.422921 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:01:23.422796 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69b79bb1-ccf1-4322-a0bd-1a307906c83a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69b79bb1-ccf1-4322-a0bd-1a307906c83a-workload-socket\") pod \"69b79bb1-ccf1-4322-a0bd-1a307906c83a\" (UID: \"69b79bb1-ccf1-4322-a0bd-1a307906c83a\") " Oct 31 05:01:23.422921 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:01:23.422860 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69b79bb1-ccf1-4322-a0bd-1a307906c83a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69b79bb1-ccf1-4322-a0bd-1a307906c83a-istio-envoy\") pod \"69b79bb1-ccf1-4322-a0bd-1a307906c83a\" (UID: \"69b79bb1-ccf1-4322-a0bd-1a307906c83a\") " Oct 31 05:01:23.423104 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:01:23.422972 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69b79bb1-ccf1-4322-a0bd-1a307906c83a-kube-api-access-m6pbm\" (UniqueName: \"kubernetes.io/projected/69b79bb1-ccf1-4322-a0bd-1a307906c83a-kube-api-access-m6pbm\") pod \"69b79bb1-ccf1-4322-a0bd-1a307906c83a\" (UID: \"69b79bb1-ccf1-4322-a0bd-1a307906c83a\") " Oct 31 05:01:23.423104 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:01:23.422976 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69b79bb1-ccf1-4322-a0bd-1a307906c83a-credential-socket" pod "69b79bb1-ccf1-4322-a0bd-1a307906c83a" (UID: "69b79bb1-ccf1-4322-a0bd-1a307906c83a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:01:23.423104 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:01:23.423039 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69b79bb1-ccf1-4322-a0bd-1a307906c83a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69b79bb1-ccf1-4322-a0bd-1a307906c83a-istio-data\") pod \"69b79bb1-ccf1-4322-a0bd-1a307906c83a\" (UID: \"69b79bb1-ccf1-4322-a0bd-1a307906c83a\") " Oct 31 05:01:23.423104 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:01:23.423058 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69b79bb1-ccf1-4322-a0bd-1a307906c83a-istio-token\" (UniqueName: \"kubernetes.io/projected/69b79bb1-ccf1-4322-a0bd-1a307906c83a-istio-token\") pod \"69b79bb1-ccf1-4322-a0bd-1a307906c83a\" (UID: \"69b79bb1-ccf1-4322-a0bd-1a307906c83a\") " Oct 31 05:01:23.423104 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:01:23.423091 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69b79bb1-ccf1-4322-a0bd-1a307906c83a-workload-socket" pod "69b79bb1-ccf1-4322-a0bd-1a307906c83a" (UID: "69b79bb1-ccf1-4322-a0bd-1a307906c83a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:01:23.423379 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:01:23.423163 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/69b79bb1-ccf1-4322-a0bd-1a307906c83a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69b79bb1-ccf1-4322-a0bd-1a307906c83a-istio-podinfo\") pod \"69b79bb1-ccf1-4322-a0bd-1a307906c83a\" (UID: \"69b79bb1-ccf1-4322-a0bd-1a307906c83a\") " Oct 31 05:01:23.423379 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:01:23.423212 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69b79bb1-ccf1-4322-a0bd-1a307906c83a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69b79bb1-ccf1-4322-a0bd-1a307906c83a-workload-certs\") pod \"69b79bb1-ccf1-4322-a0bd-1a307906c83a\" (UID: \"69b79bb1-ccf1-4322-a0bd-1a307906c83a\") " Oct 31 05:01:23.423379 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:01:23.423258 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69b79bb1-ccf1-4322-a0bd-1a307906c83a-istio-data" pod "69b79bb1-ccf1-4322-a0bd-1a307906c83a" (UID: "69b79bb1-ccf1-4322-a0bd-1a307906c83a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:01:23.423379 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:01:23.423298 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69b79bb1-ccf1-4322-a0bd-1a307906c83a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69b79bb1-ccf1-4322-a0bd-1a307906c83a-istiod-ca-cert\") pod \"69b79bb1-ccf1-4322-a0bd-1a307906c83a\" (UID: \"69b79bb1-ccf1-4322-a0bd-1a307906c83a\") " Oct 31 05:01:23.423817 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:01:23.423787 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69b79bb1-ccf1-4322-a0bd-1a307906c83a-workload-certs" pod "69b79bb1-ccf1-4322-a0bd-1a307906c83a" (UID: "69b79bb1-ccf1-4322-a0bd-1a307906c83a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:01:23.424192 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:01:23.424166 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69b79bb1-ccf1-4322-a0bd-1a307906c83a-istiod-ca-cert" pod "69b79bb1-ccf1-4322-a0bd-1a307906c83a" (UID: "69b79bb1-ccf1-4322-a0bd-1a307906c83a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:01:23.425077 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:01:23.425053 4376 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69b79bb1-ccf1-4322-a0bd-1a307906c83a-credential-socket\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:01:23.425188 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:01:23.425085 4376 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69b79bb1-ccf1-4322-a0bd-1a307906c83a-workload-socket\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:01:23.425188 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:01:23.425101 4376 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69b79bb1-ccf1-4322-a0bd-1a307906c83a-istio-data\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:01:23.425188 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:01:23.425116 4376 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69b79bb1-ccf1-4322-a0bd-1a307906c83a-workload-certs\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:01:23.425188 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:01:23.425129 4376 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69b79bb1-ccf1-4322-a0bd-1a307906c83a-istiod-ca-cert\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:01:23.425946 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:01:23.425903 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69b79bb1-ccf1-4322-a0bd-1a307906c83a-istio-envoy" pod "69b79bb1-ccf1-4322-a0bd-1a307906c83a" (UID: "69b79bb1-ccf1-4322-a0bd-1a307906c83a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:01:23.426202 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:01:23.426174 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/69b79bb1-ccf1-4322-a0bd-1a307906c83a-istio-podinfo" pod "69b79bb1-ccf1-4322-a0bd-1a307906c83a" (UID: "69b79bb1-ccf1-4322-a0bd-1a307906c83a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 05:01:23.427766 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:01:23.427740 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69b79bb1-ccf1-4322-a0bd-1a307906c83a-istio-token" pod "69b79bb1-ccf1-4322-a0bd-1a307906c83a" (UID: "69b79bb1-ccf1-4322-a0bd-1a307906c83a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:01:23.428730 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:01:23.428705 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69b79bb1-ccf1-4322-a0bd-1a307906c83a-kube-api-access-m6pbm" pod "69b79bb1-ccf1-4322-a0bd-1a307906c83a" (UID: "69b79bb1-ccf1-4322-a0bd-1a307906c83a"). InnerVolumeSpecName "kube-api-access-m6pbm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:01:23.526316 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:01:23.526281 4376 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69b79bb1-ccf1-4322-a0bd-1a307906c83a-istio-envoy\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:01:23.526316 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:01:23.526316 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6pbm\" (UniqueName: \"kubernetes.io/projected/69b79bb1-ccf1-4322-a0bd-1a307906c83a-kube-api-access-m6pbm\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:01:23.526316 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:01:23.526344 4376 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69b79bb1-ccf1-4322-a0bd-1a307906c83a-istio-token\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:01:23.526961 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:01:23.526359 4376 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69b79bb1-ccf1-4322-a0bd-1a307906c83a-istio-podinfo\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:01:24.328126 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:01:24.328092 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-8973e2-openshift-default-6bd79597f7-pdlqd" Oct 31 05:01:24.402484 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:01:24.402459 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-8973e2-openshift-default-6bd79597f7-pdlqd"] Oct 31 05:01:24.406366 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:01:24.406322 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-8973e2-openshift-default-6bd79597f7-pdlqd"] Oct 31 05:01:24.445854 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:01:24.445825 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69b79bb1-ccf1-4322-a0bd-1a307906c83a" path="/var/lib/kubelet/pods/69b79bb1-ccf1-4322-a0bd-1a307906c83a/volumes" Oct 31 05:01:32.586944 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:01:32.586903 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 05:01:41.001426 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:01:41.001384 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 05:02:11.000790 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:02:11.000740 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 05:02:16.947380 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:02:16.947350 4376 scope.go:122] "RemoveContainer" containerID="4d4fa08898e985e3990b3c6be18814e244301190358a2585b835da704a133ab7" Oct 31 05:02:41.001661 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:02:41.001622 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 05:02:54.123932 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:02:54.123900 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 05:03:11.000697 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:03:11.000654 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 05:03:41.002766 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:03:41.002734 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 05:04:11.002691 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:04:11.002652 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 05:04:12.822153 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:04:12.822116 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 05:04:41.001925 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:04:41.001887 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 05:05:11.008436 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:05:11.008395 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 05:05:17.541579 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:17.541535 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lj6bl"] Oct 31 05:05:17.542040 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:17.541789 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lj6bl" podUID="a9435ec7-da60-49e4-9219-1a3cdb485843" containerName="istio-proxy" containerID="cri-o://a297566bb97bdba22ad788cfe14bbca8cb77f6c5f4e5c83358f4bda5fad0a4e9" gracePeriod=30 Oct 31 05:05:21.274135 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:21.274097 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-47e9af-openshift-default-5b7fdd8749-hb7v2"] Oct 31 05:05:21.275076 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:21.275050 4376 cpu_manager.go:414] "containerMap: removing container" podUID="978b4c58-1c25-4e04-bfbf-5cd4e0a7001d" containerName="cleanup" Oct 31 05:05:21.275076 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:21.275076 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="978b4c58-1c25-4e04-bfbf-5cd4e0a7001d" containerName="cleanup" Oct 31 05:05:21.275253 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:21.275089 4376 cpu_manager.go:414] "containerMap: removing container" podUID="a9853a85-21be-4351-a76c-78d4da0aa7dc" containerName="cleanup" Oct 31 05:05:21.275253 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:21.275098 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9853a85-21be-4351-a76c-78d4da0aa7dc" containerName="cleanup" Oct 31 05:05:21.275253 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:21.275107 4376 cpu_manager.go:414] "containerMap: removing container" podUID="a63f4216-a71c-45b7-98b9-3b08500503fd" containerName="cleanup" Oct 31 05:05:21.275253 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:21.275115 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a63f4216-a71c-45b7-98b9-3b08500503fd" containerName="cleanup" Oct 31 05:05:21.275253 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:21.275129 4376 cpu_manager.go:414] "containerMap: removing container" podUID="9850fba4-11d3-4785-adc0-5ddd89293842" containerName="cleanup" Oct 31 05:05:21.275253 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:21.275136 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9850fba4-11d3-4785-adc0-5ddd89293842" containerName="cleanup" Oct 31 05:05:21.275253 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:21.275145 4376 cpu_manager.go:414] "containerMap: removing container" podUID="a63f4216-a71c-45b7-98b9-3b08500503fd" containerName="cleanup" Oct 31 05:05:21.275253 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:21.275152 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a63f4216-a71c-45b7-98b9-3b08500503fd" containerName="cleanup" Oct 31 05:05:21.275253 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:21.275171 4376 cpu_manager.go:414] "containerMap: removing container" podUID="8dfadf3e-94d3-4e77-bd97-5bb554b03d2f" containerName="cleanup" Oct 31 05:05:21.275253 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:21.275178 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8dfadf3e-94d3-4e77-bd97-5bb554b03d2f" containerName="cleanup" Oct 31 05:05:21.275253 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:21.275191 4376 cpu_manager.go:414] "containerMap: removing container" podUID="69b79bb1-ccf1-4322-a0bd-1a307906c83a" containerName="istio-proxy" Oct 31 05:05:21.275253 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:21.275197 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69b79bb1-ccf1-4322-a0bd-1a307906c83a" containerName="istio-proxy" Oct 31 05:05:21.275253 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:21.275212 4376 cpu_manager.go:414] "containerMap: removing container" podUID="5f6a15d6-d47f-47de-817f-fc70feb236e8" containerName="cleanup" Oct 31 05:05:21.275253 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:21.275219 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f6a15d6-d47f-47de-817f-fc70feb236e8" containerName="cleanup" Oct 31 05:05:21.275253 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:21.275232 4376 cpu_manager.go:414] "containerMap: removing container" podUID="9850fba4-11d3-4785-adc0-5ddd89293842" containerName="cleanup" Oct 31 05:05:21.275253 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:21.275238 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9850fba4-11d3-4785-adc0-5ddd89293842" containerName="cleanup" Oct 31 05:05:21.275253 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:21.275250 4376 cpu_manager.go:414] "containerMap: removing container" podUID="7212f966-42ad-4ed2-8e45-fde6a7908c2c" containerName="cleanup" Oct 31 05:05:21.275253 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:21.275257 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7212f966-42ad-4ed2-8e45-fde6a7908c2c" containerName="cleanup" Oct 31 05:05:21.276212 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:21.275268 4376 cpu_manager.go:414] "containerMap: removing container" podUID="ecbbd44a-efa7-4e45-8514-1a57eccc71e5" containerName="cleanup" Oct 31 05:05:21.276212 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:21.275274 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecbbd44a-efa7-4e45-8514-1a57eccc71e5" containerName="cleanup" Oct 31 05:05:21.276212 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:21.275284 4376 cpu_manager.go:414] "containerMap: removing container" podUID="7b953e1c-6983-456f-b2c8-77e9314aa758" containerName="cleanup" Oct 31 05:05:21.276212 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:21.275291 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b953e1c-6983-456f-b2c8-77e9314aa758" containerName="cleanup" Oct 31 05:05:21.276212 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:21.275306 4376 cpu_manager.go:414] "containerMap: removing container" podUID="09f6c274-8499-4b06-91af-6af516c3e56a" containerName="cleanup" Oct 31 05:05:21.276212 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:21.275314 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09f6c274-8499-4b06-91af-6af516c3e56a" containerName="cleanup" Oct 31 05:05:21.276212 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:21.275324 4376 cpu_manager.go:414] "containerMap: removing container" podUID="7b953e1c-6983-456f-b2c8-77e9314aa758" containerName="cleanup" Oct 31 05:05:21.276212 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:21.275347 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b953e1c-6983-456f-b2c8-77e9314aa758" containerName="cleanup" Oct 31 05:05:21.276212 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:21.275361 4376 cpu_manager.go:414] "containerMap: removing container" podUID="09f6c274-8499-4b06-91af-6af516c3e56a" containerName="cleanup" Oct 31 05:05:21.276212 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:21.275368 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09f6c274-8499-4b06-91af-6af516c3e56a" containerName="cleanup" Oct 31 05:05:21.276212 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:21.275381 4376 cpu_manager.go:414] "containerMap: removing container" podUID="ecbbd44a-efa7-4e45-8514-1a57eccc71e5" containerName="cleanup" Oct 31 05:05:21.276212 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:21.275388 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecbbd44a-efa7-4e45-8514-1a57eccc71e5" containerName="cleanup" Oct 31 05:05:21.276212 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:21.275400 4376 cpu_manager.go:414] "containerMap: removing container" podUID="a9853a85-21be-4351-a76c-78d4da0aa7dc" containerName="cleanup" Oct 31 05:05:21.276212 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:21.275407 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9853a85-21be-4351-a76c-78d4da0aa7dc" containerName="cleanup" Oct 31 05:05:21.276212 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:21.275417 4376 cpu_manager.go:414] "containerMap: removing container" podUID="7b953e1c-6983-456f-b2c8-77e9314aa758" containerName="cleanup" Oct 31 05:05:21.276212 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:21.275424 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b953e1c-6983-456f-b2c8-77e9314aa758" containerName="cleanup" Oct 31 05:05:21.276212 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:21.275433 4376 cpu_manager.go:414] "containerMap: removing container" podUID="ecbbd44a-efa7-4e45-8514-1a57eccc71e5" containerName="cleanup" Oct 31 05:05:21.276212 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:21.275439 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecbbd44a-efa7-4e45-8514-1a57eccc71e5" containerName="cleanup" Oct 31 05:05:21.276212 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:21.275453 4376 cpu_manager.go:414] "containerMap: removing container" podUID="79b8c49d-28ac-49fa-b101-5f553fdac035" containerName="cleanup" Oct 31 05:05:21.276212 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:21.275459 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79b8c49d-28ac-49fa-b101-5f553fdac035" containerName="cleanup" Oct 31 05:05:21.276212 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:21.275468 4376 cpu_manager.go:414] "containerMap: removing container" podUID="7212f966-42ad-4ed2-8e45-fde6a7908c2c" containerName="cleanup" Oct 31 05:05:21.276212 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:21.275474 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7212f966-42ad-4ed2-8e45-fde6a7908c2c" containerName="cleanup" Oct 31 05:05:21.276212 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:21.275497 4376 cpu_manager.go:414] "containerMap: removing container" podUID="19a7ea83-1d36-494a-ab52-f24535365ff9" containerName="istio-proxy" Oct 31 05:05:21.276212 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:21.275504 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19a7ea83-1d36-494a-ab52-f24535365ff9" containerName="istio-proxy" Oct 31 05:05:21.276212 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:21.275514 4376 cpu_manager.go:414] "containerMap: removing container" podUID="2ae817be-98c3-4760-b1ed-b545d5b24e0f" containerName="cleanup" Oct 31 05:05:21.276212 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:21.275520 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ae817be-98c3-4760-b1ed-b545d5b24e0f" containerName="cleanup" Oct 31 05:05:21.276212 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:21.275534 4376 cpu_manager.go:414] "containerMap: removing container" podUID="69c1b903-0719-4813-9e1e-11e87818ec69" containerName="istio-proxy" Oct 31 05:05:21.276212 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:21.275541 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69c1b903-0719-4813-9e1e-11e87818ec69" containerName="istio-proxy" Oct 31 05:05:21.276212 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:21.275554 4376 cpu_manager.go:414] "containerMap: removing container" podUID="5f6a15d6-d47f-47de-817f-fc70feb236e8" containerName="cleanup" Oct 31 05:05:21.276212 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:21.275561 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f6a15d6-d47f-47de-817f-fc70feb236e8" containerName="cleanup" Oct 31 05:05:21.276212 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:21.275573 4376 cpu_manager.go:414] "containerMap: removing container" podUID="8dfadf3e-94d3-4e77-bd97-5bb554b03d2f" containerName="cleanup" Oct 31 05:05:21.276212 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:21.275581 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8dfadf3e-94d3-4e77-bd97-5bb554b03d2f" containerName="cleanup" Oct 31 05:05:21.276212 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:21.275592 4376 cpu_manager.go:414] "containerMap: removing container" podUID="978b4c58-1c25-4e04-bfbf-5cd4e0a7001d" containerName="cleanup" Oct 31 05:05:21.276212 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:21.275598 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="978b4c58-1c25-4e04-bfbf-5cd4e0a7001d" containerName="cleanup" Oct 31 05:05:21.276212 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:21.275610 4376 cpu_manager.go:414] "containerMap: removing container" podUID="a9853a85-21be-4351-a76c-78d4da0aa7dc" containerName="cleanup" Oct 31 05:05:21.276212 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:21.275617 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9853a85-21be-4351-a76c-78d4da0aa7dc" containerName="cleanup" Oct 31 05:05:21.276212 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:21.275637 4376 cpu_manager.go:414] "containerMap: removing container" podUID="2fa45335-eb01-48ef-bdd2-2a32785614d5" containerName="istio-proxy" Oct 31 05:05:21.276212 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:21.275644 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fa45335-eb01-48ef-bdd2-2a32785614d5" containerName="istio-proxy" Oct 31 05:05:21.276212 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:21.275662 4376 cpu_manager.go:414] "containerMap: removing container" podUID="8dfadf3e-94d3-4e77-bd97-5bb554b03d2f" containerName="cleanup" Oct 31 05:05:21.276212 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:21.275668 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8dfadf3e-94d3-4e77-bd97-5bb554b03d2f" containerName="cleanup" Oct 31 05:05:21.276212 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:21.275683 4376 cpu_manager.go:414] "containerMap: removing container" podUID="2ae817be-98c3-4760-b1ed-b545d5b24e0f" containerName="cleanup" Oct 31 05:05:21.276212 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:21.275690 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ae817be-98c3-4760-b1ed-b545d5b24e0f" containerName="cleanup" Oct 31 05:05:21.276212 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:21.275703 4376 cpu_manager.go:414] "containerMap: removing container" podUID="79b8c49d-28ac-49fa-b101-5f553fdac035" containerName="cleanup" Oct 31 05:05:21.276212 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:21.275710 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79b8c49d-28ac-49fa-b101-5f553fdac035" containerName="cleanup" Oct 31 05:05:21.276212 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:21.275721 4376 cpu_manager.go:414] "containerMap: removing container" podUID="2ae817be-98c3-4760-b1ed-b545d5b24e0f" containerName="cleanup" Oct 31 05:05:21.276212 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:21.275728 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ae817be-98c3-4760-b1ed-b545d5b24e0f" containerName="cleanup" Oct 31 05:05:21.276212 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:21.275740 4376 cpu_manager.go:414] "containerMap: removing container" podUID="79b8c49d-28ac-49fa-b101-5f553fdac035" containerName="cleanup" Oct 31 05:05:21.276212 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:21.275747 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79b8c49d-28ac-49fa-b101-5f553fdac035" containerName="cleanup" Oct 31 05:05:21.276212 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:21.275943 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b953e1c-6983-456f-b2c8-77e9314aa758" containerName="cleanup" Oct 31 05:05:21.276212 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:21.275957 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecbbd44a-efa7-4e45-8514-1a57eccc71e5" containerName="cleanup" Oct 31 05:05:21.276212 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:21.275967 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="978b4c58-1c25-4e04-bfbf-5cd4e0a7001d" containerName="cleanup" Oct 31 05:05:21.276212 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:21.275976 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="79b8c49d-28ac-49fa-b101-5f553fdac035" containerName="cleanup" Oct 31 05:05:21.276212 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:21.275984 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="7212f966-42ad-4ed2-8e45-fde6a7908c2c" containerName="cleanup" Oct 31 05:05:21.276212 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:21.275995 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="19a7ea83-1d36-494a-ab52-f24535365ff9" containerName="istio-proxy" Oct 31 05:05:21.276212 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:21.276003 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="09f6c274-8499-4b06-91af-6af516c3e56a" containerName="cleanup" Oct 31 05:05:21.276212 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:21.276013 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ae817be-98c3-4760-b1ed-b545d5b24e0f" containerName="cleanup" Oct 31 05:05:21.276212 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:21.276021 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="9850fba4-11d3-4785-adc0-5ddd89293842" containerName="cleanup" Oct 31 05:05:21.276212 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:21.276030 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="69c1b903-0719-4813-9e1e-11e87818ec69" containerName="istio-proxy" Oct 31 05:05:21.276212 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:21.276043 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f6a15d6-d47f-47de-817f-fc70feb236e8" containerName="cleanup" Oct 31 05:05:21.276212 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:21.276051 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="8dfadf3e-94d3-4e77-bd97-5bb554b03d2f" containerName="cleanup" Oct 31 05:05:21.276212 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:21.276060 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9853a85-21be-4351-a76c-78d4da0aa7dc" containerName="cleanup" Oct 31 05:05:21.276212 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:21.276069 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="a63f4216-a71c-45b7-98b9-3b08500503fd" containerName="cleanup" Oct 31 05:05:21.276212 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:21.276079 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fa45335-eb01-48ef-bdd2-2a32785614d5" containerName="istio-proxy" Oct 31 05:05:21.276212 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:21.276093 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ae817be-98c3-4760-b1ed-b545d5b24e0f" containerName="cleanup" Oct 31 05:05:21.276212 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:21.276102 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ae817be-98c3-4760-b1ed-b545d5b24e0f" containerName="cleanup" Oct 31 05:05:21.276212 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:21.276112 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="79b8c49d-28ac-49fa-b101-5f553fdac035" containerName="cleanup" Oct 31 05:05:21.276212 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:21.276122 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="978b4c58-1c25-4e04-bfbf-5cd4e0a7001d" containerName="cleanup" Oct 31 05:05:21.276212 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:21.276128 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9853a85-21be-4351-a76c-78d4da0aa7dc" containerName="cleanup" Oct 31 05:05:21.276212 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:21.276137 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="a63f4216-a71c-45b7-98b9-3b08500503fd" containerName="cleanup" Oct 31 05:05:21.276212 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:21.276146 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="9850fba4-11d3-4785-adc0-5ddd89293842" containerName="cleanup" Oct 31 05:05:21.276212 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:21.276154 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="a63f4216-a71c-45b7-98b9-3b08500503fd" containerName="cleanup" Oct 31 05:05:21.276212 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:21.276163 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="8dfadf3e-94d3-4e77-bd97-5bb554b03d2f" containerName="cleanup" Oct 31 05:05:21.276212 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:21.276173 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="69b79bb1-ccf1-4322-a0bd-1a307906c83a" containerName="istio-proxy" Oct 31 05:05:21.276212 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:21.276184 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f6a15d6-d47f-47de-817f-fc70feb236e8" containerName="cleanup" Oct 31 05:05:21.276212 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:21.276194 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="9850fba4-11d3-4785-adc0-5ddd89293842" containerName="cleanup" Oct 31 05:05:21.276212 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:21.276205 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="7212f966-42ad-4ed2-8e45-fde6a7908c2c" containerName="cleanup" Oct 31 05:05:21.276212 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:21.276215 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b953e1c-6983-456f-b2c8-77e9314aa758" containerName="cleanup" Oct 31 05:05:21.276212 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:21.276225 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="09f6c274-8499-4b06-91af-6af516c3e56a" containerName="cleanup" Oct 31 05:05:21.276212 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:21.276233 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecbbd44a-efa7-4e45-8514-1a57eccc71e5" containerName="cleanup" Oct 31 05:05:21.280710 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:21.280689 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-47e9af-openshift-default-5b7fdd8749-hb7v2" Oct 31 05:05:21.299088 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:21.299065 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-47e9af-openshift-default-dockercfg-k45vf\"" Oct 31 05:05:21.312489 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:21.312459 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/341d12d1-db55-4078-acb0-47169d00ac7e-istio-podinfo\") pod \"e2e-inf-b-47e9af-openshift-default-5b7fdd8749-hb7v2\" (UID: \"341d12d1-db55-4078-acb0-47169d00ac7e\") " pod="openshift-ingress/e2e-inf-b-47e9af-openshift-default-5b7fdd8749-hb7v2" Oct 31 05:05:21.312663 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:21.312646 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/341d12d1-db55-4078-acb0-47169d00ac7e-workload-certs\") pod \"e2e-inf-b-47e9af-openshift-default-5b7fdd8749-hb7v2\" (UID: \"341d12d1-db55-4078-acb0-47169d00ac7e\") " pod="openshift-ingress/e2e-inf-b-47e9af-openshift-default-5b7fdd8749-hb7v2" Oct 31 05:05:21.312722 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:21.312683 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/341d12d1-db55-4078-acb0-47169d00ac7e-istio-envoy\") pod \"e2e-inf-b-47e9af-openshift-default-5b7fdd8749-hb7v2\" (UID: \"341d12d1-db55-4078-acb0-47169d00ac7e\") " pod="openshift-ingress/e2e-inf-b-47e9af-openshift-default-5b7fdd8749-hb7v2" Oct 31 05:05:21.312828 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:21.312806 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/341d12d1-db55-4078-acb0-47169d00ac7e-istiod-ca-cert\") pod \"e2e-inf-b-47e9af-openshift-default-5b7fdd8749-hb7v2\" (UID: \"341d12d1-db55-4078-acb0-47169d00ac7e\") " pod="openshift-ingress/e2e-inf-b-47e9af-openshift-default-5b7fdd8749-hb7v2" Oct 31 05:05:21.312915 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:21.312901 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/341d12d1-db55-4078-acb0-47169d00ac7e-credential-socket\") pod \"e2e-inf-b-47e9af-openshift-default-5b7fdd8749-hb7v2\" (UID: \"341d12d1-db55-4078-acb0-47169d00ac7e\") " pod="openshift-ingress/e2e-inf-b-47e9af-openshift-default-5b7fdd8749-hb7v2" Oct 31 05:05:21.312979 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:21.312933 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/341d12d1-db55-4078-acb0-47169d00ac7e-istio-token\") pod \"e2e-inf-b-47e9af-openshift-default-5b7fdd8749-hb7v2\" (UID: \"341d12d1-db55-4078-acb0-47169d00ac7e\") " pod="openshift-ingress/e2e-inf-b-47e9af-openshift-default-5b7fdd8749-hb7v2" Oct 31 05:05:21.313073 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:21.313051 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frwnx\" (UniqueName: \"kubernetes.io/projected/341d12d1-db55-4078-acb0-47169d00ac7e-kube-api-access-frwnx\") pod \"e2e-inf-b-47e9af-openshift-default-5b7fdd8749-hb7v2\" (UID: \"341d12d1-db55-4078-acb0-47169d00ac7e\") " pod="openshift-ingress/e2e-inf-b-47e9af-openshift-default-5b7fdd8749-hb7v2" Oct 31 05:05:21.313203 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:21.313150 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/341d12d1-db55-4078-acb0-47169d00ac7e-istio-data\") pod \"e2e-inf-b-47e9af-openshift-default-5b7fdd8749-hb7v2\" (UID: \"341d12d1-db55-4078-acb0-47169d00ac7e\") " pod="openshift-ingress/e2e-inf-b-47e9af-openshift-default-5b7fdd8749-hb7v2" Oct 31 05:05:21.313367 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:21.313319 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/341d12d1-db55-4078-acb0-47169d00ac7e-workload-socket\") pod \"e2e-inf-b-47e9af-openshift-default-5b7fdd8749-hb7v2\" (UID: \"341d12d1-db55-4078-acb0-47169d00ac7e\") " pod="openshift-ingress/e2e-inf-b-47e9af-openshift-default-5b7fdd8749-hb7v2" Oct 31 05:05:21.318608 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:21.318582 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-47e9af-openshift-default-5b7fdd8749-hb7v2"] Oct 31 05:05:21.414366 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:21.414321 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/341d12d1-db55-4078-acb0-47169d00ac7e-credential-socket\") pod \"e2e-inf-b-47e9af-openshift-default-5b7fdd8749-hb7v2\" (UID: \"341d12d1-db55-4078-acb0-47169d00ac7e\") " pod="openshift-ingress/e2e-inf-b-47e9af-openshift-default-5b7fdd8749-hb7v2" Oct 31 05:05:21.414508 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:21.414386 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/341d12d1-db55-4078-acb0-47169d00ac7e-istio-token\") pod \"e2e-inf-b-47e9af-openshift-default-5b7fdd8749-hb7v2\" (UID: \"341d12d1-db55-4078-acb0-47169d00ac7e\") " pod="openshift-ingress/e2e-inf-b-47e9af-openshift-default-5b7fdd8749-hb7v2" Oct 31 05:05:21.414508 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:21.414454 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frwnx\" (UniqueName: \"kubernetes.io/projected/341d12d1-db55-4078-acb0-47169d00ac7e-kube-api-access-frwnx\") pod \"e2e-inf-b-47e9af-openshift-default-5b7fdd8749-hb7v2\" (UID: \"341d12d1-db55-4078-acb0-47169d00ac7e\") " pod="openshift-ingress/e2e-inf-b-47e9af-openshift-default-5b7fdd8749-hb7v2" Oct 31 05:05:21.414508 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:21.414498 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/341d12d1-db55-4078-acb0-47169d00ac7e-istio-data\") pod \"e2e-inf-b-47e9af-openshift-default-5b7fdd8749-hb7v2\" (UID: \"341d12d1-db55-4078-acb0-47169d00ac7e\") " pod="openshift-ingress/e2e-inf-b-47e9af-openshift-default-5b7fdd8749-hb7v2" Oct 31 05:05:21.414696 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:21.414550 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/341d12d1-db55-4078-acb0-47169d00ac7e-workload-socket\") pod \"e2e-inf-b-47e9af-openshift-default-5b7fdd8749-hb7v2\" (UID: \"341d12d1-db55-4078-acb0-47169d00ac7e\") " pod="openshift-ingress/e2e-inf-b-47e9af-openshift-default-5b7fdd8749-hb7v2" Oct 31 05:05:21.414696 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:21.414610 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/341d12d1-db55-4078-acb0-47169d00ac7e-istio-podinfo\") pod \"e2e-inf-b-47e9af-openshift-default-5b7fdd8749-hb7v2\" (UID: \"341d12d1-db55-4078-acb0-47169d00ac7e\") " pod="openshift-ingress/e2e-inf-b-47e9af-openshift-default-5b7fdd8749-hb7v2" Oct 31 05:05:21.414814 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:21.414721 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/341d12d1-db55-4078-acb0-47169d00ac7e-workload-certs\") pod \"e2e-inf-b-47e9af-openshift-default-5b7fdd8749-hb7v2\" (UID: \"341d12d1-db55-4078-acb0-47169d00ac7e\") " pod="openshift-ingress/e2e-inf-b-47e9af-openshift-default-5b7fdd8749-hb7v2" Oct 31 05:05:21.414814 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:21.414754 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/341d12d1-db55-4078-acb0-47169d00ac7e-istio-envoy\") pod \"e2e-inf-b-47e9af-openshift-default-5b7fdd8749-hb7v2\" (UID: \"341d12d1-db55-4078-acb0-47169d00ac7e\") " pod="openshift-ingress/e2e-inf-b-47e9af-openshift-default-5b7fdd8749-hb7v2" Oct 31 05:05:21.414814 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:21.414802 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/341d12d1-db55-4078-acb0-47169d00ac7e-credential-socket\") pod \"e2e-inf-b-47e9af-openshift-default-5b7fdd8749-hb7v2\" (UID: \"341d12d1-db55-4078-acb0-47169d00ac7e\") " pod="openshift-ingress/e2e-inf-b-47e9af-openshift-default-5b7fdd8749-hb7v2" Oct 31 05:05:21.414979 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:21.414805 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/341d12d1-db55-4078-acb0-47169d00ac7e-istiod-ca-cert\") pod \"e2e-inf-b-47e9af-openshift-default-5b7fdd8749-hb7v2\" (UID: \"341d12d1-db55-4078-acb0-47169d00ac7e\") " pod="openshift-ingress/e2e-inf-b-47e9af-openshift-default-5b7fdd8749-hb7v2" Oct 31 05:05:21.414979 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:21.414867 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/341d12d1-db55-4078-acb0-47169d00ac7e-istio-data\") pod \"e2e-inf-b-47e9af-openshift-default-5b7fdd8749-hb7v2\" (UID: \"341d12d1-db55-4078-acb0-47169d00ac7e\") " pod="openshift-ingress/e2e-inf-b-47e9af-openshift-default-5b7fdd8749-hb7v2" Oct 31 05:05:21.414979 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:21.414940 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/341d12d1-db55-4078-acb0-47169d00ac7e-workload-socket\") pod \"e2e-inf-b-47e9af-openshift-default-5b7fdd8749-hb7v2\" (UID: \"341d12d1-db55-4078-acb0-47169d00ac7e\") " pod="openshift-ingress/e2e-inf-b-47e9af-openshift-default-5b7fdd8749-hb7v2" Oct 31 05:05:21.415196 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:21.415175 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/341d12d1-db55-4078-acb0-47169d00ac7e-workload-certs\") pod \"e2e-inf-b-47e9af-openshift-default-5b7fdd8749-hb7v2\" (UID: \"341d12d1-db55-4078-acb0-47169d00ac7e\") " pod="openshift-ingress/e2e-inf-b-47e9af-openshift-default-5b7fdd8749-hb7v2" Oct 31 05:05:21.415317 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:21.415300 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/341d12d1-db55-4078-acb0-47169d00ac7e-istiod-ca-cert\") pod \"e2e-inf-b-47e9af-openshift-default-5b7fdd8749-hb7v2\" (UID: \"341d12d1-db55-4078-acb0-47169d00ac7e\") " pod="openshift-ingress/e2e-inf-b-47e9af-openshift-default-5b7fdd8749-hb7v2" Oct 31 05:05:21.416791 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:21.416762 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/341d12d1-db55-4078-acb0-47169d00ac7e-istio-envoy\") pod \"e2e-inf-b-47e9af-openshift-default-5b7fdd8749-hb7v2\" (UID: \"341d12d1-db55-4078-acb0-47169d00ac7e\") " pod="openshift-ingress/e2e-inf-b-47e9af-openshift-default-5b7fdd8749-hb7v2" Oct 31 05:05:21.416893 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:21.416876 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/341d12d1-db55-4078-acb0-47169d00ac7e-istio-podinfo\") pod \"e2e-inf-b-47e9af-openshift-default-5b7fdd8749-hb7v2\" (UID: \"341d12d1-db55-4078-acb0-47169d00ac7e\") " pod="openshift-ingress/e2e-inf-b-47e9af-openshift-default-5b7fdd8749-hb7v2" Oct 31 05:05:21.439725 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:21.439702 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/341d12d1-db55-4078-acb0-47169d00ac7e-istio-token\") pod \"e2e-inf-b-47e9af-openshift-default-5b7fdd8749-hb7v2\" (UID: \"341d12d1-db55-4078-acb0-47169d00ac7e\") " pod="openshift-ingress/e2e-inf-b-47e9af-openshift-default-5b7fdd8749-hb7v2" Oct 31 05:05:21.440163 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:21.440144 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frwnx\" (UniqueName: \"kubernetes.io/projected/341d12d1-db55-4078-acb0-47169d00ac7e-kube-api-access-frwnx\") pod \"e2e-inf-b-47e9af-openshift-default-5b7fdd8749-hb7v2\" (UID: \"341d12d1-db55-4078-acb0-47169d00ac7e\") " pod="openshift-ingress/e2e-inf-b-47e9af-openshift-default-5b7fdd8749-hb7v2" Oct 31 05:05:21.594891 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:21.594829 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-47e9af-openshift-default-5b7fdd8749-hb7v2" Oct 31 05:05:21.919715 ip-10-0-134-155 kubenswrapper[4376]: W1031 05:05:21.919684 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod341d12d1_db55_4078_acb0_47169d00ac7e.slice/crio-e33dd1ebd203db805ee2e4d23b76fa616daeb43fc535f107e5c8baddc5cff4b7 WatchSource:0}: Error finding container e33dd1ebd203db805ee2e4d23b76fa616daeb43fc535f107e5c8baddc5cff4b7: Status 404 returned error can't find the container with id e33dd1ebd203db805ee2e4d23b76fa616daeb43fc535f107e5c8baddc5cff4b7 Oct 31 05:05:21.922177 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:21.922150 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-47e9af-openshift-default-5b7fdd8749-hb7v2"] Oct 31 05:05:21.922985 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:21.922951 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 05:05:21.923041 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:21.923021 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 05:05:21.923081 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:21.923051 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 05:05:22.347744 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:22.347670 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-108ad8/test-model-108ad8-kserve-7c99b696dc-jh5fg"] Oct 31 05:05:22.633777 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:22.633746 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-47e9af-openshift-default-5b7fdd8749-hb7v2" event={"ID":"341d12d1-db55-4078-acb0-47169d00ac7e","Type":"ContainerStarted","Data":"dfa799656a62f2e3dcf83ed2c25668b201a65551e0b0a4f37972ceafcf588755"} Oct 31 05:05:22.633953 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:22.633800 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-47e9af-openshift-default-5b7fdd8749-hb7v2" event={"ID":"341d12d1-db55-4078-acb0-47169d00ac7e","Type":"ContainerStarted","Data":"e33dd1ebd203db805ee2e4d23b76fa616daeb43fc535f107e5c8baddc5cff4b7"} Oct 31 05:05:22.692899 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:22.692845 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-47e9af-openshift-default-5b7fdd8749-hb7v2" podStartSLOduration=1.692827943 podStartE2EDuration="1.692827943s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 05:05:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 05:05:22.686213168 +0000 UTC m=+3670.400547780" watchObservedRunningTime="2026-10-31 05:05:22.692827943 +0000 UTC m=+3670.407162643" Oct 31 05:05:22.839779 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:22.839745 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lj6bl" Oct 31 05:05:22.907654 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:22.905898 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9435ec7-da60-49e4-9219-1a3cdb485843-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9435ec7-da60-49e4-9219-1a3cdb485843-workload-certs\") pod \"a9435ec7-da60-49e4-9219-1a3cdb485843\" (UID: \"a9435ec7-da60-49e4-9219-1a3cdb485843\") " Oct 31 05:05:22.907654 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:22.905986 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9435ec7-da60-49e4-9219-1a3cdb485843-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9435ec7-da60-49e4-9219-1a3cdb485843-credential-socket\") pod \"a9435ec7-da60-49e4-9219-1a3cdb485843\" (UID: \"a9435ec7-da60-49e4-9219-1a3cdb485843\") " Oct 31 05:05:22.907654 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:22.906045 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9435ec7-da60-49e4-9219-1a3cdb485843-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9435ec7-da60-49e4-9219-1a3cdb485843-istio-envoy\") pod \"a9435ec7-da60-49e4-9219-1a3cdb485843\" (UID: \"a9435ec7-da60-49e4-9219-1a3cdb485843\") " Oct 31 05:05:22.907654 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:22.906190 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9435ec7-da60-49e4-9219-1a3cdb485843-istio-token\" (UniqueName: \"kubernetes.io/projected/a9435ec7-da60-49e4-9219-1a3cdb485843-istio-token\") pod \"a9435ec7-da60-49e4-9219-1a3cdb485843\" (UID: \"a9435ec7-da60-49e4-9219-1a3cdb485843\") " Oct 31 05:05:22.907654 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:22.906340 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9435ec7-da60-49e4-9219-1a3cdb485843-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9435ec7-da60-49e4-9219-1a3cdb485843-workload-socket\") pod \"a9435ec7-da60-49e4-9219-1a3cdb485843\" (UID: \"a9435ec7-da60-49e4-9219-1a3cdb485843\") " Oct 31 05:05:22.907654 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:22.906392 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9435ec7-da60-49e4-9219-1a3cdb485843-kube-api-access-wfvm9\" (UniqueName: \"kubernetes.io/projected/a9435ec7-da60-49e4-9219-1a3cdb485843-kube-api-access-wfvm9\") pod \"a9435ec7-da60-49e4-9219-1a3cdb485843\" (UID: \"a9435ec7-da60-49e4-9219-1a3cdb485843\") " Oct 31 05:05:22.907654 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:22.906472 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9435ec7-da60-49e4-9219-1a3cdb485843-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9435ec7-da60-49e4-9219-1a3cdb485843-istiod-ca-cert\") pod \"a9435ec7-da60-49e4-9219-1a3cdb485843\" (UID: \"a9435ec7-da60-49e4-9219-1a3cdb485843\") " Oct 31 05:05:22.907654 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:22.906473 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9435ec7-da60-49e4-9219-1a3cdb485843-credential-socket" pod "a9435ec7-da60-49e4-9219-1a3cdb485843" (UID: "a9435ec7-da60-49e4-9219-1a3cdb485843"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:05:22.907654 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:22.906516 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9435ec7-da60-49e4-9219-1a3cdb485843-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9435ec7-da60-49e4-9219-1a3cdb485843-istio-data\") pod \"a9435ec7-da60-49e4-9219-1a3cdb485843\" (UID: \"a9435ec7-da60-49e4-9219-1a3cdb485843\") " Oct 31 05:05:22.907654 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:22.906548 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a9435ec7-da60-49e4-9219-1a3cdb485843-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9435ec7-da60-49e4-9219-1a3cdb485843-istio-podinfo\") pod \"a9435ec7-da60-49e4-9219-1a3cdb485843\" (UID: \"a9435ec7-da60-49e4-9219-1a3cdb485843\") " Oct 31 05:05:22.907654 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:22.906705 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9435ec7-da60-49e4-9219-1a3cdb485843-workload-certs" pod "a9435ec7-da60-49e4-9219-1a3cdb485843" (UID: "a9435ec7-da60-49e4-9219-1a3cdb485843"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:05:22.907654 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:22.906891 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9435ec7-da60-49e4-9219-1a3cdb485843-workload-socket" pod "a9435ec7-da60-49e4-9219-1a3cdb485843" (UID: "a9435ec7-da60-49e4-9219-1a3cdb485843"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:05:22.907654 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:22.907240 4376 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9435ec7-da60-49e4-9219-1a3cdb485843-workload-certs\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:05:22.907654 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:22.907261 4376 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9435ec7-da60-49e4-9219-1a3cdb485843-credential-socket\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:05:22.907654 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:22.907275 4376 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9435ec7-da60-49e4-9219-1a3cdb485843-workload-socket\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:05:22.914170 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:22.914144 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9435ec7-da60-49e4-9219-1a3cdb485843-istio-token" pod "a9435ec7-da60-49e4-9219-1a3cdb485843" (UID: "a9435ec7-da60-49e4-9219-1a3cdb485843"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:05:22.914346 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:22.914301 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9435ec7-da60-49e4-9219-1a3cdb485843-istio-data" pod "a9435ec7-da60-49e4-9219-1a3cdb485843" (UID: "a9435ec7-da60-49e4-9219-1a3cdb485843"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:05:22.914520 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:22.914504 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9435ec7-da60-49e4-9219-1a3cdb485843-istiod-ca-cert" pod "a9435ec7-da60-49e4-9219-1a3cdb485843" (UID: "a9435ec7-da60-49e4-9219-1a3cdb485843"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:05:22.916287 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:22.916265 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a9435ec7-da60-49e4-9219-1a3cdb485843-istio-podinfo" pod "a9435ec7-da60-49e4-9219-1a3cdb485843" (UID: "a9435ec7-da60-49e4-9219-1a3cdb485843"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 05:05:22.923412 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:22.923388 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9435ec7-da60-49e4-9219-1a3cdb485843-istio-envoy" pod "a9435ec7-da60-49e4-9219-1a3cdb485843" (UID: "a9435ec7-da60-49e4-9219-1a3cdb485843"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:05:22.923988 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:22.923968 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9435ec7-da60-49e4-9219-1a3cdb485843-kube-api-access-wfvm9" pod "a9435ec7-da60-49e4-9219-1a3cdb485843" (UID: "a9435ec7-da60-49e4-9219-1a3cdb485843"). InnerVolumeSpecName "kube-api-access-wfvm9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:05:23.014524 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:23.014492 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wfvm9\" (UniqueName: \"kubernetes.io/projected/a9435ec7-da60-49e4-9219-1a3cdb485843-kube-api-access-wfvm9\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:05:23.014771 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:23.014751 4376 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9435ec7-da60-49e4-9219-1a3cdb485843-istiod-ca-cert\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:05:23.014898 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:23.014886 4376 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9435ec7-da60-49e4-9219-1a3cdb485843-istio-data\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:05:23.014995 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:23.014984 4376 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9435ec7-da60-49e4-9219-1a3cdb485843-istio-podinfo\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:05:23.015081 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:23.015069 4376 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9435ec7-da60-49e4-9219-1a3cdb485843-istio-envoy\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:05:23.015177 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:23.015166 4376 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9435ec7-da60-49e4-9219-1a3cdb485843-istio-token\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:05:23.429363 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:23.429308 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-04d111/test-model-04d111-kserve-5d676b5997-f7fqj"] Oct 31 05:05:23.498418 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:23.498382 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-c1fe26-openshift-default-88665c78c-frz7j"] Oct 31 05:05:23.499978 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:23.499949 4376 cpu_manager.go:414] "containerMap: removing container" podUID="978b4c58-1c25-4e04-bfbf-5cd4e0a7001d" containerName="cleanup" Oct 31 05:05:23.499978 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:23.499973 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="978b4c58-1c25-4e04-bfbf-5cd4e0a7001d" containerName="cleanup" Oct 31 05:05:23.500181 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:23.499988 4376 cpu_manager.go:414] "containerMap: removing container" podUID="a63f4216-a71c-45b7-98b9-3b08500503fd" containerName="cleanup" Oct 31 05:05:23.500181 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:23.499995 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a63f4216-a71c-45b7-98b9-3b08500503fd" containerName="cleanup" Oct 31 05:05:23.500181 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:23.500008 4376 cpu_manager.go:414] "containerMap: removing container" podUID="a9435ec7-da60-49e4-9219-1a3cdb485843" containerName="istio-proxy" Oct 31 05:05:23.500181 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:23.500013 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9435ec7-da60-49e4-9219-1a3cdb485843" containerName="istio-proxy" Oct 31 05:05:23.500181 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:23.500039 4376 cpu_manager.go:414] "containerMap: removing container" podUID="9850fba4-11d3-4785-adc0-5ddd89293842" containerName="cleanup" Oct 31 05:05:23.500181 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:23.500045 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9850fba4-11d3-4785-adc0-5ddd89293842" containerName="cleanup" Oct 31 05:05:23.500181 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:23.500064 4376 cpu_manager.go:414] "containerMap: removing container" podUID="09f6c274-8499-4b06-91af-6af516c3e56a" containerName="cleanup" Oct 31 05:05:23.500181 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:23.500071 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09f6c274-8499-4b06-91af-6af516c3e56a" containerName="cleanup" Oct 31 05:05:23.500181 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:23.500083 4376 cpu_manager.go:414] "containerMap: removing container" podUID="7212f966-42ad-4ed2-8e45-fde6a7908c2c" containerName="cleanup" Oct 31 05:05:23.500181 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:23.500090 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7212f966-42ad-4ed2-8e45-fde6a7908c2c" containerName="cleanup" Oct 31 05:05:23.500181 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:23.500110 4376 cpu_manager.go:414] "containerMap: removing container" podUID="5f6a15d6-d47f-47de-817f-fc70feb236e8" containerName="cleanup" Oct 31 05:05:23.500181 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:23.500114 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f6a15d6-d47f-47de-817f-fc70feb236e8" containerName="cleanup" Oct 31 05:05:23.500673 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:23.500291 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="09f6c274-8499-4b06-91af-6af516c3e56a" containerName="cleanup" Oct 31 05:05:23.500673 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:23.500307 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b953e1c-6983-456f-b2c8-77e9314aa758" containerName="cleanup" Oct 31 05:05:23.500673 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:23.500319 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecbbd44a-efa7-4e45-8514-1a57eccc71e5" containerName="cleanup" Oct 31 05:05:23.500673 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:23.500341 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="79b8c49d-28ac-49fa-b101-5f553fdac035" containerName="cleanup" Oct 31 05:05:23.500673 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:23.500351 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="7212f966-42ad-4ed2-8e45-fde6a7908c2c" containerName="cleanup" Oct 31 05:05:23.500673 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:23.500366 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f6a15d6-d47f-47de-817f-fc70feb236e8" containerName="cleanup" Oct 31 05:05:23.500673 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:23.500382 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="978b4c58-1c25-4e04-bfbf-5cd4e0a7001d" containerName="cleanup" Oct 31 05:05:23.500673 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:23.500390 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9853a85-21be-4351-a76c-78d4da0aa7dc" containerName="cleanup" Oct 31 05:05:23.500673 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:23.500399 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9435ec7-da60-49e4-9219-1a3cdb485843" containerName="istio-proxy" Oct 31 05:05:23.500673 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:23.500407 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="8dfadf3e-94d3-4e77-bd97-5bb554b03d2f" containerName="cleanup" Oct 31 05:05:23.504652 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:23.504629 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c1fe26-openshift-default-88665c78c-frz7j" Oct 31 05:05:23.542699 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:23.542667 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-c1fe26-openshift-default-dockercfg-dmvwf\"" Oct 31 05:05:23.613362 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:23.598083 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-47e9af-openshift-default-5b7fdd8749-hb7v2" Oct 31 05:05:23.672559 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:23.672513 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/babafa7c-43f1-4ca0-aee3-6b8b4bda5dd7-credential-socket\") pod \"e2e-inf-b-c1fe26-openshift-default-88665c78c-frz7j\" (UID: \"babafa7c-43f1-4ca0-aee3-6b8b4bda5dd7\") " pod="openshift-ingress/e2e-inf-b-c1fe26-openshift-default-88665c78c-frz7j" Oct 31 05:05:23.672765 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:23.672612 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/babafa7c-43f1-4ca0-aee3-6b8b4bda5dd7-istiod-ca-cert\") pod \"e2e-inf-b-c1fe26-openshift-default-88665c78c-frz7j\" (UID: \"babafa7c-43f1-4ca0-aee3-6b8b4bda5dd7\") " pod="openshift-ingress/e2e-inf-b-c1fe26-openshift-default-88665c78c-frz7j" Oct 31 05:05:23.672765 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:23.672688 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/babafa7c-43f1-4ca0-aee3-6b8b4bda5dd7-workload-certs\") pod \"e2e-inf-b-c1fe26-openshift-default-88665c78c-frz7j\" (UID: \"babafa7c-43f1-4ca0-aee3-6b8b4bda5dd7\") " pod="openshift-ingress/e2e-inf-b-c1fe26-openshift-default-88665c78c-frz7j" Oct 31 05:05:23.672765 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:23.672712 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/babafa7c-43f1-4ca0-aee3-6b8b4bda5dd7-istio-envoy\") pod \"e2e-inf-b-c1fe26-openshift-default-88665c78c-frz7j\" (UID: \"babafa7c-43f1-4ca0-aee3-6b8b4bda5dd7\") " pod="openshift-ingress/e2e-inf-b-c1fe26-openshift-default-88665c78c-frz7j" Oct 31 05:05:23.672969 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:23.672793 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/babafa7c-43f1-4ca0-aee3-6b8b4bda5dd7-istio-data\") pod \"e2e-inf-b-c1fe26-openshift-default-88665c78c-frz7j\" (UID: \"babafa7c-43f1-4ca0-aee3-6b8b4bda5dd7\") " pod="openshift-ingress/e2e-inf-b-c1fe26-openshift-default-88665c78c-frz7j" Oct 31 05:05:23.672969 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:23.672826 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/babafa7c-43f1-4ca0-aee3-6b8b4bda5dd7-workload-socket\") pod \"e2e-inf-b-c1fe26-openshift-default-88665c78c-frz7j\" (UID: \"babafa7c-43f1-4ca0-aee3-6b8b4bda5dd7\") " pod="openshift-ingress/e2e-inf-b-c1fe26-openshift-default-88665c78c-frz7j" Oct 31 05:05:23.672969 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:23.672950 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sw6tj\" (UniqueName: \"kubernetes.io/projected/babafa7c-43f1-4ca0-aee3-6b8b4bda5dd7-kube-api-access-sw6tj\") pod \"e2e-inf-b-c1fe26-openshift-default-88665c78c-frz7j\" (UID: \"babafa7c-43f1-4ca0-aee3-6b8b4bda5dd7\") " pod="openshift-ingress/e2e-inf-b-c1fe26-openshift-default-88665c78c-frz7j" Oct 31 05:05:23.673136 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:23.672999 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/babafa7c-43f1-4ca0-aee3-6b8b4bda5dd7-istio-podinfo\") pod \"e2e-inf-b-c1fe26-openshift-default-88665c78c-frz7j\" (UID: \"babafa7c-43f1-4ca0-aee3-6b8b4bda5dd7\") " pod="openshift-ingress/e2e-inf-b-c1fe26-openshift-default-88665c78c-frz7j" Oct 31 05:05:23.673136 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:23.673044 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/babafa7c-43f1-4ca0-aee3-6b8b4bda5dd7-istio-token\") pod \"e2e-inf-b-c1fe26-openshift-default-88665c78c-frz7j\" (UID: \"babafa7c-43f1-4ca0-aee3-6b8b4bda5dd7\") " pod="openshift-ingress/e2e-inf-b-c1fe26-openshift-default-88665c78c-frz7j" Oct 31 05:05:23.701447 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:23.696568 4376 patch_prober.go:29] interesting pod/e2e-inf-b-47e9af-openshift-default-5b7fdd8749-hb7v2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused" start-of-body= Oct 31 05:05:23.701447 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:23.696613 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-47e9af-openshift-default-5b7fdd8749-hb7v2" podUID="341d12d1-db55-4078-acb0-47169d00ac7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused" Oct 31 05:05:23.704367 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:23.703153 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-c1fe26-openshift-default-88665c78c-frz7j"] Oct 31 05:05:23.719501 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:23.719467 4376 generic.go:356] "Generic (PLEG): container finished" podID="a9435ec7-da60-49e4-9219-1a3cdb485843" containerID="a297566bb97bdba22ad788cfe14bbca8cb77f6c5f4e5c83358f4bda5fad0a4e9" exitCode=0 Oct 31 05:05:23.719910 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:23.719881 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lj6bl" Oct 31 05:05:23.720241 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:23.720216 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lj6bl" event={"ID":"a9435ec7-da60-49e4-9219-1a3cdb485843","Type":"ContainerDied","Data":"a297566bb97bdba22ad788cfe14bbca8cb77f6c5f4e5c83358f4bda5fad0a4e9"} Oct 31 05:05:23.720307 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:23.720268 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lj6bl" event={"ID":"a9435ec7-da60-49e4-9219-1a3cdb485843","Type":"ContainerDied","Data":"81605d6611605b7eeb8b5f1bce40c39a11e24ab1dc30937509249d733ab5a669"} Oct 31 05:05:23.720307 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:23.720292 4376 scope.go:122] "RemoveContainer" containerID="a297566bb97bdba22ad788cfe14bbca8cb77f6c5f4e5c83358f4bda5fad0a4e9" Oct 31 05:05:23.733570 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:23.733552 4376 scope.go:122] "RemoveContainer" containerID="a297566bb97bdba22ad788cfe14bbca8cb77f6c5f4e5c83358f4bda5fad0a4e9" Oct 31 05:05:23.733826 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:05:23.733802 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a297566bb97bdba22ad788cfe14bbca8cb77f6c5f4e5c83358f4bda5fad0a4e9\": container with ID starting with a297566bb97bdba22ad788cfe14bbca8cb77f6c5f4e5c83358f4bda5fad0a4e9 not found: ID does not exist" containerID="a297566bb97bdba22ad788cfe14bbca8cb77f6c5f4e5c83358f4bda5fad0a4e9" Oct 31 05:05:23.733876 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:23.733836 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a297566bb97bdba22ad788cfe14bbca8cb77f6c5f4e5c83358f4bda5fad0a4e9"} err="failed to get container status \"a297566bb97bdba22ad788cfe14bbca8cb77f6c5f4e5c83358f4bda5fad0a4e9\": rpc error: code = NotFound desc = could not find container \"a297566bb97bdba22ad788cfe14bbca8cb77f6c5f4e5c83358f4bda5fad0a4e9\": container with ID starting with a297566bb97bdba22ad788cfe14bbca8cb77f6c5f4e5c83358f4bda5fad0a4e9 not found: ID does not exist" Oct 31 05:05:23.775159 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:23.775121 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/babafa7c-43f1-4ca0-aee3-6b8b4bda5dd7-workload-certs\") pod \"e2e-inf-b-c1fe26-openshift-default-88665c78c-frz7j\" (UID: \"babafa7c-43f1-4ca0-aee3-6b8b4bda5dd7\") " pod="openshift-ingress/e2e-inf-b-c1fe26-openshift-default-88665c78c-frz7j" Oct 31 05:05:23.775159 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:23.775163 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/babafa7c-43f1-4ca0-aee3-6b8b4bda5dd7-istio-envoy\") pod \"e2e-inf-b-c1fe26-openshift-default-88665c78c-frz7j\" (UID: \"babafa7c-43f1-4ca0-aee3-6b8b4bda5dd7\") " pod="openshift-ingress/e2e-inf-b-c1fe26-openshift-default-88665c78c-frz7j" Oct 31 05:05:23.775465 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:23.775230 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/babafa7c-43f1-4ca0-aee3-6b8b4bda5dd7-istio-data\") pod \"e2e-inf-b-c1fe26-openshift-default-88665c78c-frz7j\" (UID: \"babafa7c-43f1-4ca0-aee3-6b8b4bda5dd7\") " pod="openshift-ingress/e2e-inf-b-c1fe26-openshift-default-88665c78c-frz7j" Oct 31 05:05:23.775465 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:23.775261 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/babafa7c-43f1-4ca0-aee3-6b8b4bda5dd7-workload-socket\") pod \"e2e-inf-b-c1fe26-openshift-default-88665c78c-frz7j\" (UID: \"babafa7c-43f1-4ca0-aee3-6b8b4bda5dd7\") " pod="openshift-ingress/e2e-inf-b-c1fe26-openshift-default-88665c78c-frz7j" Oct 31 05:05:23.775465 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:23.775397 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sw6tj\" (UniqueName: \"kubernetes.io/projected/babafa7c-43f1-4ca0-aee3-6b8b4bda5dd7-kube-api-access-sw6tj\") pod \"e2e-inf-b-c1fe26-openshift-default-88665c78c-frz7j\" (UID: \"babafa7c-43f1-4ca0-aee3-6b8b4bda5dd7\") " pod="openshift-ingress/e2e-inf-b-c1fe26-openshift-default-88665c78c-frz7j" Oct 31 05:05:23.775465 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:23.775444 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/babafa7c-43f1-4ca0-aee3-6b8b4bda5dd7-istio-podinfo\") pod \"e2e-inf-b-c1fe26-openshift-default-88665c78c-frz7j\" (UID: \"babafa7c-43f1-4ca0-aee3-6b8b4bda5dd7\") " pod="openshift-ingress/e2e-inf-b-c1fe26-openshift-default-88665c78c-frz7j" Oct 31 05:05:23.775727 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:23.775490 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/babafa7c-43f1-4ca0-aee3-6b8b4bda5dd7-istio-token\") pod \"e2e-inf-b-c1fe26-openshift-default-88665c78c-frz7j\" (UID: \"babafa7c-43f1-4ca0-aee3-6b8b4bda5dd7\") " pod="openshift-ingress/e2e-inf-b-c1fe26-openshift-default-88665c78c-frz7j" Oct 31 05:05:23.775727 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:23.775594 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/babafa7c-43f1-4ca0-aee3-6b8b4bda5dd7-workload-certs\") pod \"e2e-inf-b-c1fe26-openshift-default-88665c78c-frz7j\" (UID: \"babafa7c-43f1-4ca0-aee3-6b8b4bda5dd7\") " pod="openshift-ingress/e2e-inf-b-c1fe26-openshift-default-88665c78c-frz7j" Oct 31 05:05:23.775975 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:23.775947 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/babafa7c-43f1-4ca0-aee3-6b8b4bda5dd7-credential-socket\") pod \"e2e-inf-b-c1fe26-openshift-default-88665c78c-frz7j\" (UID: \"babafa7c-43f1-4ca0-aee3-6b8b4bda5dd7\") " pod="openshift-ingress/e2e-inf-b-c1fe26-openshift-default-88665c78c-frz7j" Oct 31 05:05:23.776080 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:23.776043 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/babafa7c-43f1-4ca0-aee3-6b8b4bda5dd7-istiod-ca-cert\") pod \"e2e-inf-b-c1fe26-openshift-default-88665c78c-frz7j\" (UID: \"babafa7c-43f1-4ca0-aee3-6b8b4bda5dd7\") " pod="openshift-ingress/e2e-inf-b-c1fe26-openshift-default-88665c78c-frz7j" Oct 31 05:05:23.776166 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:23.776143 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/babafa7c-43f1-4ca0-aee3-6b8b4bda5dd7-workload-socket\") pod \"e2e-inf-b-c1fe26-openshift-default-88665c78c-frz7j\" (UID: \"babafa7c-43f1-4ca0-aee3-6b8b4bda5dd7\") " pod="openshift-ingress/e2e-inf-b-c1fe26-openshift-default-88665c78c-frz7j" Oct 31 05:05:23.776438 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:23.776415 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/babafa7c-43f1-4ca0-aee3-6b8b4bda5dd7-istio-data\") pod \"e2e-inf-b-c1fe26-openshift-default-88665c78c-frz7j\" (UID: \"babafa7c-43f1-4ca0-aee3-6b8b4bda5dd7\") " pod="openshift-ingress/e2e-inf-b-c1fe26-openshift-default-88665c78c-frz7j" Oct 31 05:05:23.776658 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:23.776638 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/babafa7c-43f1-4ca0-aee3-6b8b4bda5dd7-credential-socket\") pod \"e2e-inf-b-c1fe26-openshift-default-88665c78c-frz7j\" (UID: \"babafa7c-43f1-4ca0-aee3-6b8b4bda5dd7\") " pod="openshift-ingress/e2e-inf-b-c1fe26-openshift-default-88665c78c-frz7j" Oct 31 05:05:23.776865 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:23.776843 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/babafa7c-43f1-4ca0-aee3-6b8b4bda5dd7-istiod-ca-cert\") pod \"e2e-inf-b-c1fe26-openshift-default-88665c78c-frz7j\" (UID: \"babafa7c-43f1-4ca0-aee3-6b8b4bda5dd7\") " pod="openshift-ingress/e2e-inf-b-c1fe26-openshift-default-88665c78c-frz7j" Oct 31 05:05:23.780626 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:23.780603 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/babafa7c-43f1-4ca0-aee3-6b8b4bda5dd7-istio-envoy\") pod \"e2e-inf-b-c1fe26-openshift-default-88665c78c-frz7j\" (UID: \"babafa7c-43f1-4ca0-aee3-6b8b4bda5dd7\") " pod="openshift-ingress/e2e-inf-b-c1fe26-openshift-default-88665c78c-frz7j" Oct 31 05:05:23.789167 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:23.789112 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lj6bl"] Oct 31 05:05:23.791768 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:23.791749 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/babafa7c-43f1-4ca0-aee3-6b8b4bda5dd7-istio-podinfo\") pod \"e2e-inf-b-c1fe26-openshift-default-88665c78c-frz7j\" (UID: \"babafa7c-43f1-4ca0-aee3-6b8b4bda5dd7\") " pod="openshift-ingress/e2e-inf-b-c1fe26-openshift-default-88665c78c-frz7j" Oct 31 05:05:23.796151 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:23.796061 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lj6bl"] Oct 31 05:05:23.802170 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:23.801869 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/babafa7c-43f1-4ca0-aee3-6b8b4bda5dd7-istio-token\") pod \"e2e-inf-b-c1fe26-openshift-default-88665c78c-frz7j\" (UID: \"babafa7c-43f1-4ca0-aee3-6b8b4bda5dd7\") " pod="openshift-ingress/e2e-inf-b-c1fe26-openshift-default-88665c78c-frz7j" Oct 31 05:05:23.803226 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:23.803184 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sw6tj\" (UniqueName: \"kubernetes.io/projected/babafa7c-43f1-4ca0-aee3-6b8b4bda5dd7-kube-api-access-sw6tj\") pod \"e2e-inf-b-c1fe26-openshift-default-88665c78c-frz7j\" (UID: \"babafa7c-43f1-4ca0-aee3-6b8b4bda5dd7\") " pod="openshift-ingress/e2e-inf-b-c1fe26-openshift-default-88665c78c-frz7j" Oct 31 05:05:23.967810 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:23.966388 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c1fe26-openshift-default-88665c78c-frz7j" Oct 31 05:05:24.335562 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:24.335533 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-c1fe26-openshift-default-88665c78c-frz7j"] Oct 31 05:05:24.336504 ip-10-0-134-155 kubenswrapper[4376]: W1031 05:05:24.336475 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbabafa7c_43f1_4ca0_aee3_6b8b4bda5dd7.slice/crio-64545fb67fec6056521fe84d017f9feed8e1cb21e595976a7595d01fde152f37 WatchSource:0}: Error finding container 64545fb67fec6056521fe84d017f9feed8e1cb21e595976a7595d01fde152f37: Status 404 returned error can't find the container with id 64545fb67fec6056521fe84d017f9feed8e1cb21e595976a7595d01fde152f37 Oct 31 05:05:24.368284 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:24.365176 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 05:05:24.368284 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:24.365273 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 05:05:24.368284 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:24.365320 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 05:05:24.472290 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:24.472264 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9435ec7-da60-49e4-9219-1a3cdb485843" path="/var/lib/kubelet/pods/a9435ec7-da60-49e4-9219-1a3cdb485843/volumes" Oct 31 05:05:24.596141 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:24.596063 4376 patch_prober.go:29] interesting pod/e2e-inf-b-47e9af-openshift-default-5b7fdd8749-hb7v2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused" start-of-body= Oct 31 05:05:24.596141 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:24.596126 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-47e9af-openshift-default-5b7fdd8749-hb7v2" podUID="341d12d1-db55-4078-acb0-47169d00ac7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused" Oct 31 05:05:24.731639 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:24.731606 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c1fe26-openshift-default-88665c78c-frz7j" event={"ID":"babafa7c-43f1-4ca0-aee3-6b8b4bda5dd7","Type":"ContainerStarted","Data":"0a8065bdd935048ca341ded4bc921cb4e4f6f813e828f3ddf717c3b72e65ed87"} Oct 31 05:05:24.731838 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:24.731669 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c1fe26-openshift-default-88665c78c-frz7j" event={"ID":"babafa7c-43f1-4ca0-aee3-6b8b4bda5dd7","Type":"ContainerStarted","Data":"64545fb67fec6056521fe84d017f9feed8e1cb21e595976a7595d01fde152f37"} Oct 31 05:05:24.787064 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:24.787008 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-c1fe26-openshift-default-88665c78c-frz7j" podStartSLOduration=1.786995734 podStartE2EDuration="1.786995734s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 05:05:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 05:05:24.784215931 +0000 UTC m=+3672.498550575" watchObservedRunningTime="2026-10-31 05:05:24.786995734 +0000 UTC m=+3672.501330349" Oct 31 05:05:24.967269 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:24.967193 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-c1fe26-openshift-default-88665c78c-frz7j" Oct 31 05:05:25.595621 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:25.595582 4376 patch_prober.go:29] interesting pod/e2e-inf-b-47e9af-openshift-default-5b7fdd8749-hb7v2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused" start-of-body= Oct 31 05:05:25.596104 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:25.595647 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-47e9af-openshift-default-5b7fdd8749-hb7v2" podUID="341d12d1-db55-4078-acb0-47169d00ac7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused" Oct 31 05:05:25.974351 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:25.973902 4376 patch_prober.go:29] interesting pod/e2e-inf-b-c1fe26-openshift-default-88665c78c-frz7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.239:15021/healthz/ready\": dial tcp 10.133.0.239:15021: connect: connection refused" start-of-body= Oct 31 05:05:25.974351 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:25.973964 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c1fe26-openshift-default-88665c78c-frz7j" podUID="babafa7c-43f1-4ca0-aee3-6b8b4bda5dd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.239:15021/healthz/ready\": dial tcp 10.133.0.239:15021: connect: connection refused" Oct 31 05:05:26.598424 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:26.597774 4376 patch_prober.go:29] interesting pod/e2e-inf-b-47e9af-openshift-default-5b7fdd8749-hb7v2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused" start-of-body= Oct 31 05:05:26.598424 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:26.597836 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-47e9af-openshift-default-5b7fdd8749-hb7v2" podUID="341d12d1-db55-4078-acb0-47169d00ac7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused" Oct 31 05:05:26.967689 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:26.967651 4376 patch_prober.go:29] interesting pod/e2e-inf-b-c1fe26-openshift-default-88665c78c-frz7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.239:15021/healthz/ready\": dial tcp 10.133.0.239:15021: connect: connection refused" start-of-body= Oct 31 05:05:26.968013 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:26.967973 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c1fe26-openshift-default-88665c78c-frz7j" podUID="babafa7c-43f1-4ca0-aee3-6b8b4bda5dd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.239:15021/healthz/ready\": dial tcp 10.133.0.239:15021: connect: connection refused" Oct 31 05:05:27.602355 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:27.599747 4376 patch_prober.go:29] interesting pod/e2e-inf-b-47e9af-openshift-default-5b7fdd8749-hb7v2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused" start-of-body= Oct 31 05:05:27.602355 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:27.599810 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-47e9af-openshift-default-5b7fdd8749-hb7v2" podUID="341d12d1-db55-4078-acb0-47169d00ac7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused" Oct 31 05:05:27.972437 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:27.967659 4376 patch_prober.go:29] interesting pod/e2e-inf-b-c1fe26-openshift-default-88665c78c-frz7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.239:15021/healthz/ready\": dial tcp 10.133.0.239:15021: connect: connection refused" start-of-body= Oct 31 05:05:27.977853 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:27.977813 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c1fe26-openshift-default-88665c78c-frz7j" podUID="babafa7c-43f1-4ca0-aee3-6b8b4bda5dd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.239:15021/healthz/ready\": dial tcp 10.133.0.239:15021: connect: connection refused" Oct 31 05:05:28.595169 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:28.595130 4376 patch_prober.go:29] interesting pod/e2e-inf-b-47e9af-openshift-default-5b7fdd8749-hb7v2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused" start-of-body= Oct 31 05:05:28.595381 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:28.595188 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-47e9af-openshift-default-5b7fdd8749-hb7v2" podUID="341d12d1-db55-4078-acb0-47169d00ac7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused" Oct 31 05:05:28.966950 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:28.966912 4376 patch_prober.go:29] interesting pod/e2e-inf-b-c1fe26-openshift-default-88665c78c-frz7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.239:15021/healthz/ready\": dial tcp 10.133.0.239:15021: connect: connection refused" start-of-body= Oct 31 05:05:28.967359 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:28.966967 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c1fe26-openshift-default-88665c78c-frz7j" podUID="babafa7c-43f1-4ca0-aee3-6b8b4bda5dd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.239:15021/healthz/ready\": dial tcp 10.133.0.239:15021: connect: connection refused" Oct 31 05:05:29.600518 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:29.600489 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-47e9af-openshift-default-5b7fdd8749-hb7v2" Oct 31 05:05:29.601022 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:29.601003 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-47e9af-openshift-default-5b7fdd8749-hb7v2" Oct 31 05:05:29.601723 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:29.601702 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-47e9af-openshift-default-5b7fdd8749-hb7v2" Oct 31 05:05:29.967677 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:29.967642 4376 patch_prober.go:29] interesting pod/e2e-inf-b-c1fe26-openshift-default-88665c78c-frz7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.239:15021/healthz/ready\": dial tcp 10.133.0.239:15021: connect: connection refused" start-of-body= Oct 31 05:05:29.968310 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:29.968281 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c1fe26-openshift-default-88665c78c-frz7j" podUID="babafa7c-43f1-4ca0-aee3-6b8b4bda5dd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.239:15021/healthz/ready\": dial tcp 10.133.0.239:15021: connect: connection refused" Oct 31 05:05:31.098917 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:31.098880 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-c1fe26-openshift-default-88665c78c-frz7j" Oct 31 05:05:31.099727 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:31.099699 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-c1fe26-openshift-default-88665c78c-frz7j" Oct 31 05:05:31.100403 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:31.100382 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-c1fe26-openshift-default-88665c78c-frz7j" Oct 31 05:05:34.875865 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:34.875829 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 05:05:37.373006 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:37.372957 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-108ad8/test-model-108ad8-kserve-7c99b696dc-jh5fg" podUID="70f810de-025a-402f-8c1b-5d70a156789b" containerName="main" containerID="cri-o://f9d2a3c8e0cff254e8d34de1a8c8ec77e644854505334f8f0d16df1a412183bb" gracePeriod=45 Oct 31 05:05:37.697346 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:37.697309 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-108ad8/test-model-108ad8-kserve-7c99b696dc-jh5fg" Oct 31 05:05:37.878353 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:37.873005 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70f810de-025a-402f-8c1b-5d70a156789b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/70f810de-025a-402f-8c1b-5d70a156789b-dshm\") pod \"70f810de-025a-402f-8c1b-5d70a156789b\" (UID: \"70f810de-025a-402f-8c1b-5d70a156789b\") " Oct 31 05:05:37.878353 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:37.873057 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70f810de-025a-402f-8c1b-5d70a156789b-home\" (UniqueName: \"kubernetes.io/empty-dir/70f810de-025a-402f-8c1b-5d70a156789b-home\") pod \"70f810de-025a-402f-8c1b-5d70a156789b\" (UID: \"70f810de-025a-402f-8c1b-5d70a156789b\") " Oct 31 05:05:37.878353 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:37.873276 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70f810de-025a-402f-8c1b-5d70a156789b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70f810de-025a-402f-8c1b-5d70a156789b-model-cache\") pod \"70f810de-025a-402f-8c1b-5d70a156789b\" (UID: \"70f810de-025a-402f-8c1b-5d70a156789b\") " Oct 31 05:05:37.878353 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:37.873361 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/70f810de-025a-402f-8c1b-5d70a156789b-tls-certs\" (UniqueName: \"kubernetes.io/secret/70f810de-025a-402f-8c1b-5d70a156789b-tls-certs\") pod \"70f810de-025a-402f-8c1b-5d70a156789b\" (UID: \"70f810de-025a-402f-8c1b-5d70a156789b\") " Oct 31 05:05:37.878353 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:37.873455 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70f810de-025a-402f-8c1b-5d70a156789b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70f810de-025a-402f-8c1b-5d70a156789b-tmp-dir\") pod \"70f810de-025a-402f-8c1b-5d70a156789b\" (UID: \"70f810de-025a-402f-8c1b-5d70a156789b\") " Oct 31 05:05:37.878353 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:37.873639 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70f810de-025a-402f-8c1b-5d70a156789b-kube-api-access-njvcp\" (UniqueName: \"kubernetes.io/projected/70f810de-025a-402f-8c1b-5d70a156789b-kube-api-access-njvcp\") pod \"70f810de-025a-402f-8c1b-5d70a156789b\" (UID: \"70f810de-025a-402f-8c1b-5d70a156789b\") " Oct 31 05:05:37.894210 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:37.893988 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70f810de-025a-402f-8c1b-5d70a156789b-model-cache" pod "70f810de-025a-402f-8c1b-5d70a156789b" (UID: "70f810de-025a-402f-8c1b-5d70a156789b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:05:37.894787 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:37.894473 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70f810de-025a-402f-8c1b-5d70a156789b-home" pod "70f810de-025a-402f-8c1b-5d70a156789b" (UID: "70f810de-025a-402f-8c1b-5d70a156789b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:05:37.898393 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:37.898356 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70f810de-025a-402f-8c1b-5d70a156789b-tmp-dir" pod "70f810de-025a-402f-8c1b-5d70a156789b" (UID: "70f810de-025a-402f-8c1b-5d70a156789b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:05:37.906162 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:37.904938 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70f810de-025a-402f-8c1b-5d70a156789b-tls-certs" pod "70f810de-025a-402f-8c1b-5d70a156789b" (UID: "70f810de-025a-402f-8c1b-5d70a156789b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 05:05:37.917160 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:37.908659 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70f810de-025a-402f-8c1b-5d70a156789b-dshm" pod "70f810de-025a-402f-8c1b-5d70a156789b" (UID: "70f810de-025a-402f-8c1b-5d70a156789b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:05:37.917160 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:37.915130 4376 generic.go:356] "Generic (PLEG): container finished" podID="70f810de-025a-402f-8c1b-5d70a156789b" containerID="f9d2a3c8e0cff254e8d34de1a8c8ec77e644854505334f8f0d16df1a412183bb" exitCode=0 Oct 31 05:05:37.917160 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:37.915206 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-108ad8/test-model-108ad8-kserve-7c99b696dc-jh5fg" event={"ID":"70f810de-025a-402f-8c1b-5d70a156789b","Type":"ContainerDied","Data":"f9d2a3c8e0cff254e8d34de1a8c8ec77e644854505334f8f0d16df1a412183bb"} Oct 31 05:05:37.917160 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:37.915258 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-108ad8/test-model-108ad8-kserve-7c99b696dc-jh5fg" event={"ID":"70f810de-025a-402f-8c1b-5d70a156789b","Type":"ContainerDied","Data":"e868c8ebc5e34cd4a8f8eaf641e97569798f2eb36d594d97c496f98b701b82c6"} Oct 31 05:05:37.917160 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:37.915286 4376 scope.go:122] "RemoveContainer" containerID="f9d2a3c8e0cff254e8d34de1a8c8ec77e644854505334f8f0d16df1a412183bb" Oct 31 05:05:37.917160 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:37.915584 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-108ad8/test-model-108ad8-kserve-7c99b696dc-jh5fg" Oct 31 05:05:37.921825 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:37.921798 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70f810de-025a-402f-8c1b-5d70a156789b-kube-api-access-njvcp" pod "70f810de-025a-402f-8c1b-5d70a156789b" (UID: "70f810de-025a-402f-8c1b-5d70a156789b"). InnerVolumeSpecName "kube-api-access-njvcp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:05:37.943430 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:37.943410 4376 scope.go:122] "RemoveContainer" containerID="f9d2a3c8e0cff254e8d34de1a8c8ec77e644854505334f8f0d16df1a412183bb" Oct 31 05:05:37.943771 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:05:37.943744 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9d2a3c8e0cff254e8d34de1a8c8ec77e644854505334f8f0d16df1a412183bb\": container with ID starting with f9d2a3c8e0cff254e8d34de1a8c8ec77e644854505334f8f0d16df1a412183bb not found: ID does not exist" containerID="f9d2a3c8e0cff254e8d34de1a8c8ec77e644854505334f8f0d16df1a412183bb" Oct 31 05:05:37.943863 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:37.943777 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9d2a3c8e0cff254e8d34de1a8c8ec77e644854505334f8f0d16df1a412183bb"} err="failed to get container status \"f9d2a3c8e0cff254e8d34de1a8c8ec77e644854505334f8f0d16df1a412183bb\": rpc error: code = NotFound desc = could not find container \"f9d2a3c8e0cff254e8d34de1a8c8ec77e644854505334f8f0d16df1a412183bb\": container with ID starting with f9d2a3c8e0cff254e8d34de1a8c8ec77e644854505334f8f0d16df1a412183bb not found: ID does not exist" Oct 31 05:05:37.976076 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:37.976034 4376 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70f810de-025a-402f-8c1b-5d70a156789b-model-cache\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:05:37.976076 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:37.976075 4376 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70f810de-025a-402f-8c1b-5d70a156789b-tls-certs\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:05:37.976274 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:37.976089 4376 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70f810de-025a-402f-8c1b-5d70a156789b-tmp-dir\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:05:37.976274 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:37.976106 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njvcp\" (UniqueName: \"kubernetes.io/projected/70f810de-025a-402f-8c1b-5d70a156789b-kube-api-access-njvcp\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:05:37.976274 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:37.976120 4376 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70f810de-025a-402f-8c1b-5d70a156789b-dshm\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:05:37.976274 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:37.976132 4376 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70f810de-025a-402f-8c1b-5d70a156789b-home\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:05:38.277980 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:38.277940 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-108ad8/test-model-108ad8-kserve-7c99b696dc-jh5fg"] Oct 31 05:05:38.291537 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:38.291506 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-108ad8/test-model-108ad8-kserve-7c99b696dc-jh5fg"] Oct 31 05:05:38.445482 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:38.445448 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70f810de-025a-402f-8c1b-5d70a156789b" path="/var/lib/kubelet/pods/70f810de-025a-402f-8c1b-5d70a156789b/volumes" Oct 31 05:05:38.482213 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:38.482132 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-04d111/test-model-04d111-kserve-5d676b5997-f7fqj" podUID="c48d91ef-0b60-4cad-8f97-7405a26337f0" containerName="main" containerID="cri-o://a699f7dfc0299c485ac4f6d7874ad40698576cf2ccd15921104919d194e5bbc6" gracePeriod=45 Oct 31 05:05:38.804178 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:38.804146 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-04d111/test-model-04d111-kserve-5d676b5997-f7fqj" Oct 31 05:05:38.892505 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:38.892388 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c48d91ef-0b60-4cad-8f97-7405a26337f0-tls-certs\" (UniqueName: \"kubernetes.io/secret/c48d91ef-0b60-4cad-8f97-7405a26337f0-tls-certs\") pod \"c48d91ef-0b60-4cad-8f97-7405a26337f0\" (UID: \"c48d91ef-0b60-4cad-8f97-7405a26337f0\") " Oct 31 05:05:38.892705 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:38.892634 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c48d91ef-0b60-4cad-8f97-7405a26337f0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c48d91ef-0b60-4cad-8f97-7405a26337f0-dshm\") pod \"c48d91ef-0b60-4cad-8f97-7405a26337f0\" (UID: \"c48d91ef-0b60-4cad-8f97-7405a26337f0\") " Oct 31 05:05:38.892705 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:38.892673 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c48d91ef-0b60-4cad-8f97-7405a26337f0-kube-api-access-hsv9l\" (UniqueName: \"kubernetes.io/projected/c48d91ef-0b60-4cad-8f97-7405a26337f0-kube-api-access-hsv9l\") pod \"c48d91ef-0b60-4cad-8f97-7405a26337f0\" (UID: \"c48d91ef-0b60-4cad-8f97-7405a26337f0\") " Oct 31 05:05:38.892864 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:38.892705 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c48d91ef-0b60-4cad-8f97-7405a26337f0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c48d91ef-0b60-4cad-8f97-7405a26337f0-model-cache\") pod \"c48d91ef-0b60-4cad-8f97-7405a26337f0\" (UID: \"c48d91ef-0b60-4cad-8f97-7405a26337f0\") " Oct 31 05:05:38.892864 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:38.892765 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c48d91ef-0b60-4cad-8f97-7405a26337f0-home\" (UniqueName: \"kubernetes.io/empty-dir/c48d91ef-0b60-4cad-8f97-7405a26337f0-home\") pod \"c48d91ef-0b60-4cad-8f97-7405a26337f0\" (UID: \"c48d91ef-0b60-4cad-8f97-7405a26337f0\") " Oct 31 05:05:38.893052 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:38.892917 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c48d91ef-0b60-4cad-8f97-7405a26337f0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c48d91ef-0b60-4cad-8f97-7405a26337f0-tmp-dir\") pod \"c48d91ef-0b60-4cad-8f97-7405a26337f0\" (UID: \"c48d91ef-0b60-4cad-8f97-7405a26337f0\") " Oct 31 05:05:38.894968 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:38.893796 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c48d91ef-0b60-4cad-8f97-7405a26337f0-model-cache" pod "c48d91ef-0b60-4cad-8f97-7405a26337f0" (UID: "c48d91ef-0b60-4cad-8f97-7405a26337f0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:05:38.894968 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:38.893907 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c48d91ef-0b60-4cad-8f97-7405a26337f0-tmp-dir" pod "c48d91ef-0b60-4cad-8f97-7405a26337f0" (UID: "c48d91ef-0b60-4cad-8f97-7405a26337f0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:05:38.895410 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:38.895387 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c48d91ef-0b60-4cad-8f97-7405a26337f0-home" pod "c48d91ef-0b60-4cad-8f97-7405a26337f0" (UID: "c48d91ef-0b60-4cad-8f97-7405a26337f0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:05:38.897440 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:38.897409 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c48d91ef-0b60-4cad-8f97-7405a26337f0-kube-api-access-hsv9l" pod "c48d91ef-0b60-4cad-8f97-7405a26337f0" (UID: "c48d91ef-0b60-4cad-8f97-7405a26337f0"). InnerVolumeSpecName "kube-api-access-hsv9l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:05:38.904840 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:38.904793 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c48d91ef-0b60-4cad-8f97-7405a26337f0-tls-certs" pod "c48d91ef-0b60-4cad-8f97-7405a26337f0" (UID: "c48d91ef-0b60-4cad-8f97-7405a26337f0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 05:05:38.905305 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:38.905276 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c48d91ef-0b60-4cad-8f97-7405a26337f0-dshm" pod "c48d91ef-0b60-4cad-8f97-7405a26337f0" (UID: "c48d91ef-0b60-4cad-8f97-7405a26337f0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:05:38.926765 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:38.926444 4376 generic.go:356] "Generic (PLEG): container finished" podID="c48d91ef-0b60-4cad-8f97-7405a26337f0" containerID="a699f7dfc0299c485ac4f6d7874ad40698576cf2ccd15921104919d194e5bbc6" exitCode=0 Oct 31 05:05:38.926765 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:38.926508 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-04d111/test-model-04d111-kserve-5d676b5997-f7fqj" Oct 31 05:05:38.926765 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:38.926556 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-04d111/test-model-04d111-kserve-5d676b5997-f7fqj" event={"ID":"c48d91ef-0b60-4cad-8f97-7405a26337f0","Type":"ContainerDied","Data":"a699f7dfc0299c485ac4f6d7874ad40698576cf2ccd15921104919d194e5bbc6"} Oct 31 05:05:38.926765 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:38.926608 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-04d111/test-model-04d111-kserve-5d676b5997-f7fqj" event={"ID":"c48d91ef-0b60-4cad-8f97-7405a26337f0","Type":"ContainerDied","Data":"e633b5f912630429e5292cd9babb5238dc0830470dbb049a048ce4e76bb88f45"} Oct 31 05:05:38.926765 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:38.926631 4376 scope.go:122] "RemoveContainer" containerID="a699f7dfc0299c485ac4f6d7874ad40698576cf2ccd15921104919d194e5bbc6" Oct 31 05:05:38.947524 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:38.943126 4376 scope.go:122] "RemoveContainer" containerID="a699f7dfc0299c485ac4f6d7874ad40698576cf2ccd15921104919d194e5bbc6" Oct 31 05:05:38.947524 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:05:38.943459 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a699f7dfc0299c485ac4f6d7874ad40698576cf2ccd15921104919d194e5bbc6\": container with ID starting with a699f7dfc0299c485ac4f6d7874ad40698576cf2ccd15921104919d194e5bbc6 not found: ID does not exist" containerID="a699f7dfc0299c485ac4f6d7874ad40698576cf2ccd15921104919d194e5bbc6" Oct 31 05:05:38.947524 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:38.943487 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a699f7dfc0299c485ac4f6d7874ad40698576cf2ccd15921104919d194e5bbc6"} err="failed to get container status \"a699f7dfc0299c485ac4f6d7874ad40698576cf2ccd15921104919d194e5bbc6\": rpc error: code = NotFound desc = could not find container \"a699f7dfc0299c485ac4f6d7874ad40698576cf2ccd15921104919d194e5bbc6\": container with ID starting with a699f7dfc0299c485ac4f6d7874ad40698576cf2ccd15921104919d194e5bbc6 not found: ID does not exist" Oct 31 05:05:38.989583 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:38.989556 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-04d111/test-model-04d111-kserve-5d676b5997-f7fqj"] Oct 31 05:05:38.995343 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:38.995308 4376 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c48d91ef-0b60-4cad-8f97-7405a26337f0-dshm\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:05:38.995472 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:38.995354 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hsv9l\" (UniqueName: \"kubernetes.io/projected/c48d91ef-0b60-4cad-8f97-7405a26337f0-kube-api-access-hsv9l\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:05:38.995472 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:38.995370 4376 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c48d91ef-0b60-4cad-8f97-7405a26337f0-model-cache\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:05:38.995472 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:38.995384 4376 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c48d91ef-0b60-4cad-8f97-7405a26337f0-home\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:05:38.995472 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:38.995396 4376 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c48d91ef-0b60-4cad-8f97-7405a26337f0-tmp-dir\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:05:38.995472 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:38.995408 4376 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c48d91ef-0b60-4cad-8f97-7405a26337f0-tls-certs\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:05:38.996160 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:38.996146 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-04d111/test-model-04d111-kserve-5d676b5997-f7fqj"] Oct 31 05:05:40.446059 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:40.446023 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c48d91ef-0b60-4cad-8f97-7405a26337f0" path="/var/lib/kubelet/pods/c48d91ef-0b60-4cad-8f97-7405a26337f0/volumes" Oct 31 05:05:41.002037 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:05:41.001991 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 05:05:45.827047 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:05:45.827010 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-47e9af/test-model-47e9af-kserve-5557db6747-hkddj"] Oct 31 05:06:00.852443 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:00.852403 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-47e9af/test-model-47e9af-kserve-5557db6747-hkddj" podUID="981a91d9-8a9c-4d3e-a2fb-1e69d12624f8" containerName="main" containerID="cri-o://c52464818f34c56acd634f3f426fee2521fd6af54da97f2d8ac8bad35c166a77" gracePeriod=45 Oct 31 05:06:01.365875 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:01.326843 4376 generic.go:356] "Generic (PLEG): container finished" podID="981a91d9-8a9c-4d3e-a2fb-1e69d12624f8" containerID="c52464818f34c56acd634f3f426fee2521fd6af54da97f2d8ac8bad35c166a77" exitCode=0 Oct 31 05:06:01.365875 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:01.327863 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-47e9af/test-model-47e9af-kserve-5557db6747-hkddj" event={"ID":"981a91d9-8a9c-4d3e-a2fb-1e69d12624f8","Type":"ContainerDied","Data":"c52464818f34c56acd634f3f426fee2521fd6af54da97f2d8ac8bad35c166a77"} Oct 31 05:06:01.412260 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:01.412232 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-47e9af/test-model-47e9af-kserve-5557db6747-hkddj" Oct 31 05:06:01.436569 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:01.436535 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/981a91d9-8a9c-4d3e-a2fb-1e69d12624f8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/981a91d9-8a9c-4d3e-a2fb-1e69d12624f8-dshm\") pod \"981a91d9-8a9c-4d3e-a2fb-1e69d12624f8\" (UID: \"981a91d9-8a9c-4d3e-a2fb-1e69d12624f8\") " Oct 31 05:06:01.436569 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:01.436570 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/981a91d9-8a9c-4d3e-a2fb-1e69d12624f8-home\" (UniqueName: \"kubernetes.io/empty-dir/981a91d9-8a9c-4d3e-a2fb-1e69d12624f8-home\") pod \"981a91d9-8a9c-4d3e-a2fb-1e69d12624f8\" (UID: \"981a91d9-8a9c-4d3e-a2fb-1e69d12624f8\") " Oct 31 05:06:01.436827 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:01.436638 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/981a91d9-8a9c-4d3e-a2fb-1e69d12624f8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/981a91d9-8a9c-4d3e-a2fb-1e69d12624f8-tmp-dir\") pod \"981a91d9-8a9c-4d3e-a2fb-1e69d12624f8\" (UID: \"981a91d9-8a9c-4d3e-a2fb-1e69d12624f8\") " Oct 31 05:06:01.436827 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:01.436680 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/981a91d9-8a9c-4d3e-a2fb-1e69d12624f8-tls-certs\" (UniqueName: \"kubernetes.io/secret/981a91d9-8a9c-4d3e-a2fb-1e69d12624f8-tls-certs\") pod \"981a91d9-8a9c-4d3e-a2fb-1e69d12624f8\" (UID: \"981a91d9-8a9c-4d3e-a2fb-1e69d12624f8\") " Oct 31 05:06:01.436958 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:01.436872 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/981a91d9-8a9c-4d3e-a2fb-1e69d12624f8-home" pod "981a91d9-8a9c-4d3e-a2fb-1e69d12624f8" (UID: "981a91d9-8a9c-4d3e-a2fb-1e69d12624f8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:06:01.437020 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:01.436971 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/981a91d9-8a9c-4d3e-a2fb-1e69d12624f8-tmp-dir" pod "981a91d9-8a9c-4d3e-a2fb-1e69d12624f8" (UID: "981a91d9-8a9c-4d3e-a2fb-1e69d12624f8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:06:01.437248 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:01.437226 4376 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/981a91d9-8a9c-4d3e-a2fb-1e69d12624f8-home\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:06:01.437248 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:01.437247 4376 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/981a91d9-8a9c-4d3e-a2fb-1e69d12624f8-tmp-dir\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:06:01.438919 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:01.438894 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/981a91d9-8a9c-4d3e-a2fb-1e69d12624f8-dshm" pod "981a91d9-8a9c-4d3e-a2fb-1e69d12624f8" (UID: "981a91d9-8a9c-4d3e-a2fb-1e69d12624f8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:06:01.439098 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:01.439079 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/981a91d9-8a9c-4d3e-a2fb-1e69d12624f8-tls-certs" pod "981a91d9-8a9c-4d3e-a2fb-1e69d12624f8" (UID: "981a91d9-8a9c-4d3e-a2fb-1e69d12624f8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 05:06:01.555355 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:01.543370 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/981a91d9-8a9c-4d3e-a2fb-1e69d12624f8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/981a91d9-8a9c-4d3e-a2fb-1e69d12624f8-model-cache\") pod \"981a91d9-8a9c-4d3e-a2fb-1e69d12624f8\" (UID: \"981a91d9-8a9c-4d3e-a2fb-1e69d12624f8\") " Oct 31 05:06:01.555355 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:01.543446 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/981a91d9-8a9c-4d3e-a2fb-1e69d12624f8-kube-api-access-cdmjj\" (UniqueName: \"kubernetes.io/projected/981a91d9-8a9c-4d3e-a2fb-1e69d12624f8-kube-api-access-cdmjj\") pod \"981a91d9-8a9c-4d3e-a2fb-1e69d12624f8\" (UID: \"981a91d9-8a9c-4d3e-a2fb-1e69d12624f8\") " Oct 31 05:06:01.555355 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:01.544654 4376 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/981a91d9-8a9c-4d3e-a2fb-1e69d12624f8-tls-certs\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:06:01.555355 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:01.544676 4376 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/981a91d9-8a9c-4d3e-a2fb-1e69d12624f8-dshm\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:06:01.555355 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:01.545292 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/981a91d9-8a9c-4d3e-a2fb-1e69d12624f8-model-cache" pod "981a91d9-8a9c-4d3e-a2fb-1e69d12624f8" (UID: "981a91d9-8a9c-4d3e-a2fb-1e69d12624f8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:06:01.571362 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:01.568612 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/981a91d9-8a9c-4d3e-a2fb-1e69d12624f8-kube-api-access-cdmjj" pod "981a91d9-8a9c-4d3e-a2fb-1e69d12624f8" (UID: "981a91d9-8a9c-4d3e-a2fb-1e69d12624f8"). InnerVolumeSpecName "kube-api-access-cdmjj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:06:01.645814 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:01.645739 4376 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/981a91d9-8a9c-4d3e-a2fb-1e69d12624f8-model-cache\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:06:01.645814 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:01.645767 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cdmjj\" (UniqueName: \"kubernetes.io/projected/981a91d9-8a9c-4d3e-a2fb-1e69d12624f8-kube-api-access-cdmjj\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:06:02.197596 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:02.197546 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-47e9af/test-model-47e9af-kserve-5557db6747-hkddj" podUID="981a91d9-8a9c-4d3e-a2fb-1e69d12624f8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.224:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 05:06:02.346876 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:02.346837 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-47e9af/test-model-47e9af-kserve-5557db6747-hkddj" Oct 31 05:06:02.347533 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:02.346837 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-47e9af/test-model-47e9af-kserve-5557db6747-hkddj" event={"ID":"981a91d9-8a9c-4d3e-a2fb-1e69d12624f8","Type":"ContainerDied","Data":"fdc25705f477129d9363afba30e262bf1f510ceb116ff7334fc18e34a574275b"} Oct 31 05:06:02.347704 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:02.347596 4376 scope.go:122] "RemoveContainer" containerID="c52464818f34c56acd634f3f426fee2521fd6af54da97f2d8ac8bad35c166a77" Oct 31 05:06:02.408449 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:02.408417 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-47e9af/test-model-47e9af-kserve-5557db6747-hkddj"] Oct 31 05:06:02.422342 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:02.422293 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-47e9af/test-model-47e9af-kserve-5557db6747-hkddj"] Oct 31 05:06:02.445226 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:02.445192 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="981a91d9-8a9c-4d3e-a2fb-1e69d12624f8" path="/var/lib/kubelet/pods/981a91d9-8a9c-4d3e-a2fb-1e69d12624f8/volumes" Oct 31 05:06:02.715211 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:02.715176 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-l4jlq"] Oct 31 05:06:02.716128 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:02.716102 4376 cpu_manager.go:414] "containerMap: removing container" podUID="981a91d9-8a9c-4d3e-a2fb-1e69d12624f8" containerName="main" Oct 31 05:06:02.716128 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:02.716126 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="981a91d9-8a9c-4d3e-a2fb-1e69d12624f8" containerName="main" Oct 31 05:06:02.716320 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:02.716150 4376 cpu_manager.go:414] "containerMap: removing container" podUID="c48d91ef-0b60-4cad-8f97-7405a26337f0" containerName="main" Oct 31 05:06:02.716320 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:02.716159 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c48d91ef-0b60-4cad-8f97-7405a26337f0" containerName="main" Oct 31 05:06:02.716320 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:02.716185 4376 cpu_manager.go:414] "containerMap: removing container" podUID="70f810de-025a-402f-8c1b-5d70a156789b" containerName="main" Oct 31 05:06:02.716320 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:02.716193 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70f810de-025a-402f-8c1b-5d70a156789b" containerName="main" Oct 31 05:06:02.716511 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:02.716463 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="981a91d9-8a9c-4d3e-a2fb-1e69d12624f8" containerName="main" Oct 31 05:06:02.716511 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:02.716480 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="c48d91ef-0b60-4cad-8f97-7405a26337f0" containerName="main" Oct 31 05:06:02.716511 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:02.716492 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="70f810de-025a-402f-8c1b-5d70a156789b" containerName="main" Oct 31 05:06:02.722175 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:02.722101 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-l4jlq" Oct 31 05:06:02.742565 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:02.742535 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-l4jlq"] Oct 31 05:06:02.762507 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:02.762472 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3e2da41-2d03-4b11-8ef0-40e3848109d3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-l4jlq\" (UID: \"d3e2da41-2d03-4b11-8ef0-40e3848109d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-l4jlq" Oct 31 05:06:02.762665 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:02.762531 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3e2da41-2d03-4b11-8ef0-40e3848109d3-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-l4jlq\" (UID: \"d3e2da41-2d03-4b11-8ef0-40e3848109d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-l4jlq" Oct 31 05:06:02.762716 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:02.762687 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3e2da41-2d03-4b11-8ef0-40e3848109d3-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-l4jlq\" (UID: \"d3e2da41-2d03-4b11-8ef0-40e3848109d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-l4jlq" Oct 31 05:06:02.762760 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:02.762744 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3e2da41-2d03-4b11-8ef0-40e3848109d3-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-l4jlq\" (UID: \"d3e2da41-2d03-4b11-8ef0-40e3848109d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-l4jlq" Oct 31 05:06:02.762951 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:02.762931 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfxqn\" (UniqueName: \"kubernetes.io/projected/d3e2da41-2d03-4b11-8ef0-40e3848109d3-kube-api-access-wfxqn\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-l4jlq\" (UID: \"d3e2da41-2d03-4b11-8ef0-40e3848109d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-l4jlq" Oct 31 05:06:02.763018 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:02.763000 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3e2da41-2d03-4b11-8ef0-40e3848109d3-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-l4jlq\" (UID: \"d3e2da41-2d03-4b11-8ef0-40e3848109d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-l4jlq" Oct 31 05:06:02.763079 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:02.763051 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3e2da41-2d03-4b11-8ef0-40e3848109d3-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-l4jlq\" (UID: \"d3e2da41-2d03-4b11-8ef0-40e3848109d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-l4jlq" Oct 31 05:06:02.763143 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:02.763102 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3e2da41-2d03-4b11-8ef0-40e3848109d3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-l4jlq\" (UID: \"d3e2da41-2d03-4b11-8ef0-40e3848109d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-l4jlq" Oct 31 05:06:02.763262 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:02.763176 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3e2da41-2d03-4b11-8ef0-40e3848109d3-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-l4jlq\" (UID: \"d3e2da41-2d03-4b11-8ef0-40e3848109d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-l4jlq" Oct 31 05:06:02.864464 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:02.864428 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfxqn\" (UniqueName: \"kubernetes.io/projected/d3e2da41-2d03-4b11-8ef0-40e3848109d3-kube-api-access-wfxqn\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-l4jlq\" (UID: \"d3e2da41-2d03-4b11-8ef0-40e3848109d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-l4jlq" Oct 31 05:06:02.864646 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:02.864485 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3e2da41-2d03-4b11-8ef0-40e3848109d3-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-l4jlq\" (UID: \"d3e2da41-2d03-4b11-8ef0-40e3848109d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-l4jlq" Oct 31 05:06:02.864646 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:02.864575 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3e2da41-2d03-4b11-8ef0-40e3848109d3-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-l4jlq\" (UID: \"d3e2da41-2d03-4b11-8ef0-40e3848109d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-l4jlq" Oct 31 05:06:02.864646 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:02.864601 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3e2da41-2d03-4b11-8ef0-40e3848109d3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-l4jlq\" (UID: \"d3e2da41-2d03-4b11-8ef0-40e3848109d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-l4jlq" Oct 31 05:06:02.864646 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:02.864623 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3e2da41-2d03-4b11-8ef0-40e3848109d3-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-l4jlq\" (UID: \"d3e2da41-2d03-4b11-8ef0-40e3848109d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-l4jlq" Oct 31 05:06:02.864891 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:02.864844 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3e2da41-2d03-4b11-8ef0-40e3848109d3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-l4jlq\" (UID: \"d3e2da41-2d03-4b11-8ef0-40e3848109d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-l4jlq" Oct 31 05:06:02.864891 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:02.864883 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3e2da41-2d03-4b11-8ef0-40e3848109d3-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-l4jlq\" (UID: \"d3e2da41-2d03-4b11-8ef0-40e3848109d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-l4jlq" Oct 31 05:06:02.865001 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:02.864933 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3e2da41-2d03-4b11-8ef0-40e3848109d3-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-l4jlq\" (UID: \"d3e2da41-2d03-4b11-8ef0-40e3848109d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-l4jlq" Oct 31 05:06:02.865065 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:02.864933 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3e2da41-2d03-4b11-8ef0-40e3848109d3-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-l4jlq\" (UID: \"d3e2da41-2d03-4b11-8ef0-40e3848109d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-l4jlq" Oct 31 05:06:02.865065 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:02.865046 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3e2da41-2d03-4b11-8ef0-40e3848109d3-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-l4jlq\" (UID: \"d3e2da41-2d03-4b11-8ef0-40e3848109d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-l4jlq" Oct 31 05:06:02.865249 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:02.865223 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3e2da41-2d03-4b11-8ef0-40e3848109d3-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-l4jlq\" (UID: \"d3e2da41-2d03-4b11-8ef0-40e3848109d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-l4jlq" Oct 31 05:06:02.865345 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:02.865309 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3e2da41-2d03-4b11-8ef0-40e3848109d3-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-l4jlq\" (UID: \"d3e2da41-2d03-4b11-8ef0-40e3848109d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-l4jlq" Oct 31 05:06:02.865506 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:02.865484 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3e2da41-2d03-4b11-8ef0-40e3848109d3-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-l4jlq\" (UID: \"d3e2da41-2d03-4b11-8ef0-40e3848109d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-l4jlq" Oct 31 05:06:02.865677 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:02.865659 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3e2da41-2d03-4b11-8ef0-40e3848109d3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-l4jlq\" (UID: \"d3e2da41-2d03-4b11-8ef0-40e3848109d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-l4jlq" Oct 31 05:06:02.867251 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:02.867223 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3e2da41-2d03-4b11-8ef0-40e3848109d3-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-l4jlq\" (UID: \"d3e2da41-2d03-4b11-8ef0-40e3848109d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-l4jlq" Oct 31 05:06:02.867381 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:02.867273 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3e2da41-2d03-4b11-8ef0-40e3848109d3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-l4jlq\" (UID: \"d3e2da41-2d03-4b11-8ef0-40e3848109d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-l4jlq" Oct 31 05:06:02.890807 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:02.890774 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3e2da41-2d03-4b11-8ef0-40e3848109d3-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-l4jlq\" (UID: \"d3e2da41-2d03-4b11-8ef0-40e3848109d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-l4jlq" Oct 31 05:06:02.890946 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:02.890897 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfxqn\" (UniqueName: \"kubernetes.io/projected/d3e2da41-2d03-4b11-8ef0-40e3848109d3-kube-api-access-wfxqn\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-l4jlq\" (UID: \"d3e2da41-2d03-4b11-8ef0-40e3848109d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-l4jlq" Oct 31 05:06:03.049405 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:03.048686 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-l4jlq" Oct 31 05:06:03.529208 ip-10-0-134-155 kubenswrapper[4376]: W1031 05:06:03.529179 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3e2da41_2d03_4b11_8ef0_40e3848109d3.slice/crio-265fad908e355e72b92f86ab10727f61646bb523e1cff6ca0a1cbeef5c8bcf01 WatchSource:0}: Error finding container 265fad908e355e72b92f86ab10727f61646bb523e1cff6ca0a1cbeef5c8bcf01: Status 404 returned error can't find the container with id 265fad908e355e72b92f86ab10727f61646bb523e1cff6ca0a1cbeef5c8bcf01 Oct 31 05:06:03.530210 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:03.530185 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-l4jlq"] Oct 31 05:06:03.532314 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:03.532279 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 05:06:03.532510 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:03.532377 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 05:06:03.532510 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:03.532452 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 05:06:04.373623 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:04.373579 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-l4jlq" event={"ID":"d3e2da41-2d03-4b11-8ef0-40e3848109d3","Type":"ContainerStarted","Data":"cd00a64840fe97d6665679e38f64ac5bbe38a3888ada568acb08b3bd09eabcdf"} Oct 31 05:06:04.373835 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:04.373647 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-l4jlq" event={"ID":"d3e2da41-2d03-4b11-8ef0-40e3848109d3","Type":"ContainerStarted","Data":"265fad908e355e72b92f86ab10727f61646bb523e1cff6ca0a1cbeef5c8bcf01"} Oct 31 05:06:04.464353 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:04.463807 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-l4jlq" podStartSLOduration=2.463783203 podStartE2EDuration="2.463783203s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 05:06:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 05:06:04.436151788 +0000 UTC m=+3712.150486404" watchObservedRunningTime="2026-10-31 05:06:04.463783203 +0000 UTC m=+3712.178117821" Oct 31 05:06:05.065035 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:05.062691 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-l4jlq" Oct 31 05:06:05.065035 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:05.064512 4376 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-l4jlq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused" start-of-body= Oct 31 05:06:05.065035 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:05.064591 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-l4jlq" podUID="d3e2da41-2d03-4b11-8ef0-40e3848109d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused" Oct 31 05:06:06.059447 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:06.053732 4376 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-l4jlq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused" start-of-body= Oct 31 05:06:06.059447 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:06.053792 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-l4jlq" podUID="d3e2da41-2d03-4b11-8ef0-40e3848109d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused" Oct 31 05:06:06.898812 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:06.897959 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-04d111-openshift-default-cd549dc77-f9kwz"] Oct 31 05:06:06.898812 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:06.898217 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-04d111-openshift-default-cd549dc77-f9kwz" podUID="96a130da-eaed-4765-905b-261822ce8014" containerName="istio-proxy" containerID="cri-o://7db18a134621d04c502edc286ab1e934c6242b6c8b8da9b42df1f2762948decf" gracePeriod=30 Oct 31 05:06:07.069386 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:07.056446 4376 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-l4jlq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused" start-of-body= Oct 31 05:06:07.069386 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:07.056509 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-l4jlq" podUID="d3e2da41-2d03-4b11-8ef0-40e3848109d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused" Oct 31 05:06:08.055352 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:08.049664 4376 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-l4jlq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused" start-of-body= Oct 31 05:06:08.055352 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:08.049735 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-l4jlq" podUID="d3e2da41-2d03-4b11-8ef0-40e3848109d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused" Oct 31 05:06:09.062352 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:09.062165 4376 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-l4jlq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused" start-of-body= Oct 31 05:06:09.062352 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:09.062223 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-l4jlq" podUID="d3e2da41-2d03-4b11-8ef0-40e3848109d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused" Oct 31 05:06:10.050081 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:10.050043 4376 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-l4jlq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused" start-of-body= Oct 31 05:06:10.050274 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:10.050107 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-l4jlq" podUID="d3e2da41-2d03-4b11-8ef0-40e3848109d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused" Oct 31 05:06:11.002107 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:06:11.002065 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 05:06:11.049421 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:11.049390 4376 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-l4jlq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused" start-of-body= Oct 31 05:06:11.049603 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:11.049464 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-l4jlq" podUID="d3e2da41-2d03-4b11-8ef0-40e3848109d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused" Oct 31 05:06:12.249418 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:12.249391 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-04d111-openshift-default-cd549dc77-f9kwz" Oct 31 05:06:12.345020 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:12.344988 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96a130da-eaed-4765-905b-261822ce8014-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96a130da-eaed-4765-905b-261822ce8014-istiod-ca-cert\") pod \"96a130da-eaed-4765-905b-261822ce8014\" (UID: \"96a130da-eaed-4765-905b-261822ce8014\") " Oct 31 05:06:12.345227 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:12.345081 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96a130da-eaed-4765-905b-261822ce8014-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96a130da-eaed-4765-905b-261822ce8014-istio-data\") pod \"96a130da-eaed-4765-905b-261822ce8014\" (UID: \"96a130da-eaed-4765-905b-261822ce8014\") " Oct 31 05:06:12.345227 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:12.345166 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96a130da-eaed-4765-905b-261822ce8014-istio-token\" (UniqueName: \"kubernetes.io/projected/96a130da-eaed-4765-905b-261822ce8014-istio-token\") pod \"96a130da-eaed-4765-905b-261822ce8014\" (UID: \"96a130da-eaed-4765-905b-261822ce8014\") " Oct 31 05:06:12.345227 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:12.345220 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96a130da-eaed-4765-905b-261822ce8014-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96a130da-eaed-4765-905b-261822ce8014-istio-envoy\") pod \"96a130da-eaed-4765-905b-261822ce8014\" (UID: \"96a130da-eaed-4765-905b-261822ce8014\") " Oct 31 05:06:12.345438 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:12.345254 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96a130da-eaed-4765-905b-261822ce8014-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96a130da-eaed-4765-905b-261822ce8014-workload-certs\") pod \"96a130da-eaed-4765-905b-261822ce8014\" (UID: \"96a130da-eaed-4765-905b-261822ce8014\") " Oct 31 05:06:12.345438 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:12.345276 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96a130da-eaed-4765-905b-261822ce8014-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96a130da-eaed-4765-905b-261822ce8014-credential-socket\") pod \"96a130da-eaed-4765-905b-261822ce8014\" (UID: \"96a130da-eaed-4765-905b-261822ce8014\") " Oct 31 05:06:12.345438 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:12.345394 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96a130da-eaed-4765-905b-261822ce8014-kube-api-access-d4xsf\" (UniqueName: \"kubernetes.io/projected/96a130da-eaed-4765-905b-261822ce8014-kube-api-access-d4xsf\") pod \"96a130da-eaed-4765-905b-261822ce8014\" (UID: \"96a130da-eaed-4765-905b-261822ce8014\") " Oct 31 05:06:12.345438 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:12.345419 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/96a130da-eaed-4765-905b-261822ce8014-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96a130da-eaed-4765-905b-261822ce8014-istio-podinfo\") pod \"96a130da-eaed-4765-905b-261822ce8014\" (UID: \"96a130da-eaed-4765-905b-261822ce8014\") " Oct 31 05:06:12.345672 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:12.345540 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96a130da-eaed-4765-905b-261822ce8014-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96a130da-eaed-4765-905b-261822ce8014-workload-socket\") pod \"96a130da-eaed-4765-905b-261822ce8014\" (UID: \"96a130da-eaed-4765-905b-261822ce8014\") " Oct 31 05:06:12.347063 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:12.346546 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96a130da-eaed-4765-905b-261822ce8014-istio-data" pod "96a130da-eaed-4765-905b-261822ce8014" (UID: "96a130da-eaed-4765-905b-261822ce8014"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:06:12.347299 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:12.346577 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96a130da-eaed-4765-905b-261822ce8014-workload-socket" pod "96a130da-eaed-4765-905b-261822ce8014" (UID: "96a130da-eaed-4765-905b-261822ce8014"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:06:12.347299 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:12.346665 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96a130da-eaed-4765-905b-261822ce8014-workload-certs" pod "96a130da-eaed-4765-905b-261822ce8014" (UID: "96a130da-eaed-4765-905b-261822ce8014"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:06:12.347299 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:12.346854 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96a130da-eaed-4765-905b-261822ce8014-credential-socket" pod "96a130da-eaed-4765-905b-261822ce8014" (UID: "96a130da-eaed-4765-905b-261822ce8014"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:06:12.347299 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:12.347017 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96a130da-eaed-4765-905b-261822ce8014-istiod-ca-cert" pod "96a130da-eaed-4765-905b-261822ce8014" (UID: "96a130da-eaed-4765-905b-261822ce8014"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:06:12.351701 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:12.351550 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/96a130da-eaed-4765-905b-261822ce8014-istio-podinfo" pod "96a130da-eaed-4765-905b-261822ce8014" (UID: "96a130da-eaed-4765-905b-261822ce8014"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 05:06:12.352612 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:12.352577 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96a130da-eaed-4765-905b-261822ce8014-istio-token" pod "96a130da-eaed-4765-905b-261822ce8014" (UID: "96a130da-eaed-4765-905b-261822ce8014"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:06:12.353034 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:12.352998 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96a130da-eaed-4765-905b-261822ce8014-istio-envoy" pod "96a130da-eaed-4765-905b-261822ce8014" (UID: "96a130da-eaed-4765-905b-261822ce8014"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:06:12.354641 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:12.354620 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96a130da-eaed-4765-905b-261822ce8014-kube-api-access-d4xsf" pod "96a130da-eaed-4765-905b-261822ce8014" (UID: "96a130da-eaed-4765-905b-261822ce8014"). InnerVolumeSpecName "kube-api-access-d4xsf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:06:12.358308 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:12.358280 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-l4jlq" Oct 31 05:06:12.359630 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:12.359611 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-l4jlq" Oct 31 05:06:12.360523 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:12.360500 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-l4jlq" Oct 31 05:06:12.437696 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:12.437665 4376 generic.go:356] "Generic (PLEG): container finished" podID="96a130da-eaed-4765-905b-261822ce8014" containerID="7db18a134621d04c502edc286ab1e934c6242b6c8b8da9b42df1f2762948decf" exitCode=0 Oct 31 05:06:12.437883 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:12.437864 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-04d111-openshift-default-cd549dc77-f9kwz" Oct 31 05:06:12.445305 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:12.445278 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-04d111-openshift-default-cd549dc77-f9kwz" event={"ID":"96a130da-eaed-4765-905b-261822ce8014","Type":"ContainerDied","Data":"7db18a134621d04c502edc286ab1e934c6242b6c8b8da9b42df1f2762948decf"} Oct 31 05:06:12.445476 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:12.445318 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-04d111-openshift-default-cd549dc77-f9kwz" event={"ID":"96a130da-eaed-4765-905b-261822ce8014","Type":"ContainerDied","Data":"292a75c68458e8bffd4786e8afa7a806e37bca7c9de146ce8335e0b9a4e877a0"} Oct 31 05:06:12.445476 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:12.445362 4376 scope.go:122] "RemoveContainer" containerID="7db18a134621d04c502edc286ab1e934c6242b6c8b8da9b42df1f2762948decf" Oct 31 05:06:12.447166 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:12.447123 4376 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96a130da-eaed-4765-905b-261822ce8014-workload-socket\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:06:12.447166 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:12.447147 4376 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96a130da-eaed-4765-905b-261822ce8014-istiod-ca-cert\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:06:12.447166 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:12.447161 4376 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96a130da-eaed-4765-905b-261822ce8014-istio-data\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:06:12.447410 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:12.447175 4376 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96a130da-eaed-4765-905b-261822ce8014-istio-token\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:06:12.447410 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:12.447187 4376 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96a130da-eaed-4765-905b-261822ce8014-istio-envoy\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:06:12.447410 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:12.447199 4376 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96a130da-eaed-4765-905b-261822ce8014-workload-certs\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:06:12.447410 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:12.447212 4376 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96a130da-eaed-4765-905b-261822ce8014-credential-socket\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:06:12.447410 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:12.447226 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4xsf\" (UniqueName: \"kubernetes.io/projected/96a130da-eaed-4765-905b-261822ce8014-kube-api-access-d4xsf\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:06:12.447410 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:12.447240 4376 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96a130da-eaed-4765-905b-261822ce8014-istio-podinfo\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:06:12.458063 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:12.458044 4376 scope.go:122] "RemoveContainer" containerID="1ff8d4c81bcb160a5fba84a6af6d911265299a4bfa28ad219feae5154305902f" Oct 31 05:06:12.469800 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:12.469776 4376 scope.go:122] "RemoveContainer" containerID="7db18a134621d04c502edc286ab1e934c6242b6c8b8da9b42df1f2762948decf" Oct 31 05:06:12.470151 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:06:12.470115 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7db18a134621d04c502edc286ab1e934c6242b6c8b8da9b42df1f2762948decf\": container with ID starting with 7db18a134621d04c502edc286ab1e934c6242b6c8b8da9b42df1f2762948decf not found: ID does not exist" containerID="7db18a134621d04c502edc286ab1e934c6242b6c8b8da9b42df1f2762948decf" Oct 31 05:06:12.470252 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:12.470158 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7db18a134621d04c502edc286ab1e934c6242b6c8b8da9b42df1f2762948decf"} err="failed to get container status \"7db18a134621d04c502edc286ab1e934c6242b6c8b8da9b42df1f2762948decf\": rpc error: code = NotFound desc = could not find container \"7db18a134621d04c502edc286ab1e934c6242b6c8b8da9b42df1f2762948decf\": container with ID starting with 7db18a134621d04c502edc286ab1e934c6242b6c8b8da9b42df1f2762948decf not found: ID does not exist" Oct 31 05:06:12.470252 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:12.470178 4376 scope.go:122] "RemoveContainer" containerID="1ff8d4c81bcb160a5fba84a6af6d911265299a4bfa28ad219feae5154305902f" Oct 31 05:06:12.470492 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:06:12.470473 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ff8d4c81bcb160a5fba84a6af6d911265299a4bfa28ad219feae5154305902f\": container with ID starting with 1ff8d4c81bcb160a5fba84a6af6d911265299a4bfa28ad219feae5154305902f not found: ID does not exist" containerID="1ff8d4c81bcb160a5fba84a6af6d911265299a4bfa28ad219feae5154305902f" Oct 31 05:06:12.470534 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:12.470497 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ff8d4c81bcb160a5fba84a6af6d911265299a4bfa28ad219feae5154305902f"} err="failed to get container status \"1ff8d4c81bcb160a5fba84a6af6d911265299a4bfa28ad219feae5154305902f\": rpc error: code = NotFound desc = could not find container \"1ff8d4c81bcb160a5fba84a6af6d911265299a4bfa28ad219feae5154305902f\": container with ID starting with 1ff8d4c81bcb160a5fba84a6af6d911265299a4bfa28ad219feae5154305902f not found: ID does not exist" Oct 31 05:06:12.545401 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:12.545369 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-04d111-openshift-default-cd549dc77-f9kwz"] Oct 31 05:06:12.545578 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:12.545519 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-04d111-openshift-default-cd549dc77-f9kwz"] Oct 31 05:06:14.445762 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:14.445730 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96a130da-eaed-4765-905b-261822ce8014" path="/var/lib/kubelet/pods/96a130da-eaed-4765-905b-261822ce8014/volumes" Oct 31 05:06:14.752378 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:14.752270 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c1fe26-7c97f95bc7-knlhj"] Oct 31 05:06:14.788384 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:14.788350 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c1fe26-7c889f576c-mptbr"] Oct 31 05:06:17.754999 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:17.754965 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-n5dnl"] Oct 31 05:06:17.756046 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:17.756022 4376 cpu_manager.go:414] "containerMap: removing container" podUID="96a130da-eaed-4765-905b-261822ce8014" containerName="istio-proxy" Oct 31 05:06:17.756046 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:17.756047 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96a130da-eaed-4765-905b-261822ce8014" containerName="istio-proxy" Oct 31 05:06:17.756189 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:17.756118 4376 cpu_manager.go:414] "containerMap: removing container" podUID="96a130da-eaed-4765-905b-261822ce8014" containerName="istio-proxy" Oct 31 05:06:17.756189 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:17.756127 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96a130da-eaed-4765-905b-261822ce8014" containerName="istio-proxy" Oct 31 05:06:17.756317 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:17.756303 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="96a130da-eaed-4765-905b-261822ce8014" containerName="istio-proxy" Oct 31 05:06:17.768555 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:17.768531 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-n5dnl" Oct 31 05:06:17.783716 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:17.783682 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-n5dnl"] Oct 31 05:06:17.786967 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:17.786939 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/577f4375-87fc-4850-a066-7f10cf8307b6-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-n5dnl\" (UID: \"577f4375-87fc-4850-a066-7f10cf8307b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-n5dnl" Oct 31 05:06:17.787092 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:17.786982 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dczl\" (UniqueName: \"kubernetes.io/projected/577f4375-87fc-4850-a066-7f10cf8307b6-kube-api-access-8dczl\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-n5dnl\" (UID: \"577f4375-87fc-4850-a066-7f10cf8307b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-n5dnl" Oct 31 05:06:17.787169 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:17.787121 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/577f4375-87fc-4850-a066-7f10cf8307b6-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-n5dnl\" (UID: \"577f4375-87fc-4850-a066-7f10cf8307b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-n5dnl" Oct 31 05:06:17.787231 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:17.787191 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/577f4375-87fc-4850-a066-7f10cf8307b6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-n5dnl\" (UID: \"577f4375-87fc-4850-a066-7f10cf8307b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-n5dnl" Oct 31 05:06:17.787286 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:17.787233 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/577f4375-87fc-4850-a066-7f10cf8307b6-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-n5dnl\" (UID: \"577f4375-87fc-4850-a066-7f10cf8307b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-n5dnl" Oct 31 05:06:17.787286 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:17.787277 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/577f4375-87fc-4850-a066-7f10cf8307b6-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-n5dnl\" (UID: \"577f4375-87fc-4850-a066-7f10cf8307b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-n5dnl" Oct 31 05:06:17.787417 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:17.787346 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/577f4375-87fc-4850-a066-7f10cf8307b6-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-n5dnl\" (UID: \"577f4375-87fc-4850-a066-7f10cf8307b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-n5dnl" Oct 31 05:06:17.787495 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:17.787473 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/577f4375-87fc-4850-a066-7f10cf8307b6-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-n5dnl\" (UID: \"577f4375-87fc-4850-a066-7f10cf8307b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-n5dnl" Oct 31 05:06:17.787588 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:17.787571 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/577f4375-87fc-4850-a066-7f10cf8307b6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-n5dnl\" (UID: \"577f4375-87fc-4850-a066-7f10cf8307b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-n5dnl" Oct 31 05:06:17.889011 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:17.888980 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/577f4375-87fc-4850-a066-7f10cf8307b6-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-n5dnl\" (UID: \"577f4375-87fc-4850-a066-7f10cf8307b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-n5dnl" Oct 31 05:06:17.889163 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:17.889060 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/577f4375-87fc-4850-a066-7f10cf8307b6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-n5dnl\" (UID: \"577f4375-87fc-4850-a066-7f10cf8307b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-n5dnl" Oct 31 05:06:17.889231 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:17.889184 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/577f4375-87fc-4850-a066-7f10cf8307b6-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-n5dnl\" (UID: \"577f4375-87fc-4850-a066-7f10cf8307b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-n5dnl" Oct 31 05:06:17.889231 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:17.889212 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dczl\" (UniqueName: \"kubernetes.io/projected/577f4375-87fc-4850-a066-7f10cf8307b6-kube-api-access-8dczl\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-n5dnl\" (UID: \"577f4375-87fc-4850-a066-7f10cf8307b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-n5dnl" Oct 31 05:06:17.889357 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:17.889310 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/577f4375-87fc-4850-a066-7f10cf8307b6-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-n5dnl\" (UID: \"577f4375-87fc-4850-a066-7f10cf8307b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-n5dnl" Oct 31 05:06:17.889423 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:17.889407 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/577f4375-87fc-4850-a066-7f10cf8307b6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-n5dnl\" (UID: \"577f4375-87fc-4850-a066-7f10cf8307b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-n5dnl" Oct 31 05:06:17.889483 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:17.889445 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/577f4375-87fc-4850-a066-7f10cf8307b6-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-n5dnl\" (UID: \"577f4375-87fc-4850-a066-7f10cf8307b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-n5dnl" Oct 31 05:06:17.889541 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:17.889487 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/577f4375-87fc-4850-a066-7f10cf8307b6-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-n5dnl\" (UID: \"577f4375-87fc-4850-a066-7f10cf8307b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-n5dnl" Oct 31 05:06:17.889541 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:17.889493 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/577f4375-87fc-4850-a066-7f10cf8307b6-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-n5dnl\" (UID: \"577f4375-87fc-4850-a066-7f10cf8307b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-n5dnl" Oct 31 05:06:17.889541 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:17.889526 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/577f4375-87fc-4850-a066-7f10cf8307b6-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-n5dnl\" (UID: \"577f4375-87fc-4850-a066-7f10cf8307b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-n5dnl" Oct 31 05:06:17.889800 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:17.889777 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/577f4375-87fc-4850-a066-7f10cf8307b6-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-n5dnl\" (UID: \"577f4375-87fc-4850-a066-7f10cf8307b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-n5dnl" Oct 31 05:06:17.889950 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:17.889814 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/577f4375-87fc-4850-a066-7f10cf8307b6-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-n5dnl\" (UID: \"577f4375-87fc-4850-a066-7f10cf8307b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-n5dnl" Oct 31 05:06:17.889950 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:17.889910 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/577f4375-87fc-4850-a066-7f10cf8307b6-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-n5dnl\" (UID: \"577f4375-87fc-4850-a066-7f10cf8307b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-n5dnl" Oct 31 05:06:17.890185 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:17.890157 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/577f4375-87fc-4850-a066-7f10cf8307b6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-n5dnl\" (UID: \"577f4375-87fc-4850-a066-7f10cf8307b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-n5dnl" Oct 31 05:06:17.891786 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:17.891762 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/577f4375-87fc-4850-a066-7f10cf8307b6-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-n5dnl\" (UID: \"577f4375-87fc-4850-a066-7f10cf8307b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-n5dnl" Oct 31 05:06:17.891885 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:17.891822 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/577f4375-87fc-4850-a066-7f10cf8307b6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-n5dnl\" (UID: \"577f4375-87fc-4850-a066-7f10cf8307b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-n5dnl" Oct 31 05:06:17.914744 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:17.914715 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/577f4375-87fc-4850-a066-7f10cf8307b6-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-n5dnl\" (UID: \"577f4375-87fc-4850-a066-7f10cf8307b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-n5dnl" Oct 31 05:06:17.914885 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:17.914813 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dczl\" (UniqueName: \"kubernetes.io/projected/577f4375-87fc-4850-a066-7f10cf8307b6-kube-api-access-8dczl\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-n5dnl\" (UID: \"577f4375-87fc-4850-a066-7f10cf8307b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-n5dnl" Oct 31 05:06:18.092190 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:18.092113 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-n5dnl" Oct 31 05:06:18.422487 ip-10-0-134-155 kubenswrapper[4376]: W1031 05:06:18.422449 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod577f4375_87fc_4850_a066_7f10cf8307b6.slice/crio-63d5d44fec161951bd1566338c230f954da0d30d72630c12c499b6e0582be07e WatchSource:0}: Error finding container 63d5d44fec161951bd1566338c230f954da0d30d72630c12c499b6e0582be07e: Status 404 returned error can't find the container with id 63d5d44fec161951bd1566338c230f954da0d30d72630c12c499b6e0582be07e Oct 31 05:06:18.428431 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:18.425407 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-n5dnl"] Oct 31 05:06:18.435670 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:18.435615 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 05:06:18.435791 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:18.435766 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 05:06:18.435866 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:18.435849 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 05:06:18.518906 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:18.518871 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-n5dnl" event={"ID":"577f4375-87fc-4850-a066-7f10cf8307b6","Type":"ContainerStarted","Data":"d3898aa45660ba8b16051cfca62058251c0de44e0fe8704e85516ef92a67b26a"} Oct 31 05:06:18.519042 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:18.518932 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-n5dnl" event={"ID":"577f4375-87fc-4850-a066-7f10cf8307b6","Type":"ContainerStarted","Data":"63d5d44fec161951bd1566338c230f954da0d30d72630c12c499b6e0582be07e"} Oct 31 05:06:18.576220 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:18.576156 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-n5dnl" podStartSLOduration=1.576138015 podStartE2EDuration="1.576138015s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 05:06:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 05:06:18.570193748 +0000 UTC m=+3726.284528391" watchObservedRunningTime="2026-10-31 05:06:18.576138015 +0000 UTC m=+3726.290472632" Oct 31 05:06:19.092474 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:19.092411 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-n5dnl" Oct 31 05:06:19.094037 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:19.094016 4376 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-n5dnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" start-of-body= Oct 31 05:06:19.094117 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:19.094087 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-n5dnl" podUID="577f4375-87fc-4850-a066-7f10cf8307b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" Oct 31 05:06:19.762352 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:19.761406 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-c1fe26-7c97f95bc7-knlhj" podUID="7d918c33-5fb2-4f29-a616-c29446e15ff2" containerName="payload-processing" containerID="cri-o://d57b3926ea3879d4a9753c3e1bbbdaef248a656063ec0fb53012bb2b44d84522" gracePeriod=25 Oct 31 05:06:19.789132 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:19.789091 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c1fe26-7c889f576c-mptbr" podUID="48d10907-ef2a-43b7-b481-746473b85ef6" containerName="payload-pre-processing" containerID="cri-o://09b3e5cb1833e3454e5c12041f09e7d184ec2384206f12e976c9a309f8443dad" gracePeriod=25 Oct 31 05:06:20.093267 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:20.093183 4376 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-n5dnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" start-of-body= Oct 31 05:06:20.093267 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:20.093251 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-n5dnl" podUID="577f4375-87fc-4850-a066-7f10cf8307b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" Oct 31 05:06:20.514376 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:20.514314 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c1fe26-7c889f576c-mptbr" podUID="48d10907-ef2a-43b7-b481-746473b85ef6" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 05:06:20.514376 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:20.514363 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-c1fe26-7c97f95bc7-knlhj" podUID="7d918c33-5fb2-4f29-a616-c29446e15ff2" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 05:06:21.093370 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:21.093304 4376 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-n5dnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" start-of-body= Oct 31 05:06:21.093857 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:21.093377 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-n5dnl" podUID="577f4375-87fc-4850-a066-7f10cf8307b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" Oct 31 05:06:22.093280 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:22.093248 4376 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-n5dnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" start-of-body= Oct 31 05:06:22.093588 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:22.093294 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-n5dnl" podUID="577f4375-87fc-4850-a066-7f10cf8307b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" Oct 31 05:06:23.093150 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:23.093114 4376 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-n5dnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" start-of-body= Oct 31 05:06:23.093368 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:23.093163 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-n5dnl" podUID="577f4375-87fc-4850-a066-7f10cf8307b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" Oct 31 05:06:24.102352 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:24.096989 4376 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-n5dnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" start-of-body= Oct 31 05:06:24.102352 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:24.097051 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-n5dnl" podUID="577f4375-87fc-4850-a066-7f10cf8307b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" Oct 31 05:06:25.092975 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:25.092892 4376 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-n5dnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" start-of-body= Oct 31 05:06:25.092975 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:25.092945 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-n5dnl" podUID="577f4375-87fc-4850-a066-7f10cf8307b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" Oct 31 05:06:26.093144 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:26.093112 4376 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-n5dnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" start-of-body= Oct 31 05:06:26.093561 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:26.093178 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-n5dnl" podUID="577f4375-87fc-4850-a066-7f10cf8307b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" Oct 31 05:06:27.218449 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:27.218414 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-n5dnl" Oct 31 05:06:27.219067 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:27.219047 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-n5dnl" Oct 31 05:06:27.220455 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:27.220432 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-n5dnl" Oct 31 05:06:30.513615 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:30.513573 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-c1fe26-7c97f95bc7-knlhj" podUID="7d918c33-5fb2-4f29-a616-c29446e15ff2" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 05:06:30.514005 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:30.513589 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c1fe26-7c889f576c-mptbr" podUID="48d10907-ef2a-43b7-b481-746473b85ef6" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 05:06:32.892434 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:32.892400 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 05:06:40.087993 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:40.087962 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-c1fe26-7c97f95bc7-knlhj" Oct 31 05:06:40.117366 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:40.117323 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c1fe26-7c889f576c-mptbr" Oct 31 05:06:40.207429 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:40.207387 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48d10907-ef2a-43b7-b481-746473b85ef6-kube-api-access-vbrvw\" (UniqueName: \"kubernetes.io/projected/48d10907-ef2a-43b7-b481-746473b85ef6-kube-api-access-vbrvw\") pod \"48d10907-ef2a-43b7-b481-746473b85ef6\" (UID: \"48d10907-ef2a-43b7-b481-746473b85ef6\") " Oct 31 05:06:40.210667 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:40.210628 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d918c33-5fb2-4f29-a616-c29446e15ff2-kube-api-access-t7x7k\" (UniqueName: \"kubernetes.io/projected/7d918c33-5fb2-4f29-a616-c29446e15ff2-kube-api-access-t7x7k\") pod \"7d918c33-5fb2-4f29-a616-c29446e15ff2\" (UID: \"7d918c33-5fb2-4f29-a616-c29446e15ff2\") " Oct 31 05:06:40.210846 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:40.210730 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48d10907-ef2a-43b7-b481-746473b85ef6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48d10907-ef2a-43b7-b481-746473b85ef6-plugins-config-volume\") pod \"48d10907-ef2a-43b7-b481-746473b85ef6\" (UID: \"48d10907-ef2a-43b7-b481-746473b85ef6\") " Oct 31 05:06:40.211108 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:40.211086 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d918c33-5fb2-4f29-a616-c29446e15ff2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d918c33-5fb2-4f29-a616-c29446e15ff2-plugins-config-volume\") pod \"7d918c33-5fb2-4f29-a616-c29446e15ff2\" (UID: \"7d918c33-5fb2-4f29-a616-c29446e15ff2\") " Oct 31 05:06:40.212629 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:40.212597 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d918c33-5fb2-4f29-a616-c29446e15ff2-plugins-config-volume" pod "7d918c33-5fb2-4f29-a616-c29446e15ff2" (UID: "7d918c33-5fb2-4f29-a616-c29446e15ff2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:06:40.216910 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:40.216832 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d918c33-5fb2-4f29-a616-c29446e15ff2-kube-api-access-t7x7k" pod "7d918c33-5fb2-4f29-a616-c29446e15ff2" (UID: "7d918c33-5fb2-4f29-a616-c29446e15ff2"). InnerVolumeSpecName "kube-api-access-t7x7k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:06:40.229783 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:40.229466 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48d10907-ef2a-43b7-b481-746473b85ef6-kube-api-access-vbrvw" pod "48d10907-ef2a-43b7-b481-746473b85ef6" (UID: "48d10907-ef2a-43b7-b481-746473b85ef6"). InnerVolumeSpecName "kube-api-access-vbrvw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:06:40.229783 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:40.229468 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48d10907-ef2a-43b7-b481-746473b85ef6-plugins-config-volume" pod "48d10907-ef2a-43b7-b481-746473b85ef6" (UID: "48d10907-ef2a-43b7-b481-746473b85ef6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:06:40.314615 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:40.314531 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7x7k\" (UniqueName: \"kubernetes.io/projected/7d918c33-5fb2-4f29-a616-c29446e15ff2-kube-api-access-t7x7k\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:06:40.314615 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:40.314567 4376 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48d10907-ef2a-43b7-b481-746473b85ef6-plugins-config-volume\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:06:40.314615 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:40.314582 4376 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d918c33-5fb2-4f29-a616-c29446e15ff2-plugins-config-volume\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:06:40.314615 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:40.314597 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbrvw\" (UniqueName: \"kubernetes.io/projected/48d10907-ef2a-43b7-b481-746473b85ef6-kube-api-access-vbrvw\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:06:40.737997 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:40.737957 4376 generic.go:356] "Generic (PLEG): container finished" podID="7d918c33-5fb2-4f29-a616-c29446e15ff2" containerID="d57b3926ea3879d4a9753c3e1bbbdaef248a656063ec0fb53012bb2b44d84522" exitCode=0 Oct 31 05:06:40.738184 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:40.738024 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-c1fe26-7c97f95bc7-knlhj" Oct 31 05:06:40.738184 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:40.738039 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-c1fe26-7c97f95bc7-knlhj" event={"ID":"7d918c33-5fb2-4f29-a616-c29446e15ff2","Type":"ContainerDied","Data":"d57b3926ea3879d4a9753c3e1bbbdaef248a656063ec0fb53012bb2b44d84522"} Oct 31 05:06:40.738184 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:40.738100 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-c1fe26-7c97f95bc7-knlhj" event={"ID":"7d918c33-5fb2-4f29-a616-c29446e15ff2","Type":"ContainerDied","Data":"75ffc981ec704512f834ec3ab1a5aa289d544de37deb3237fcf90e366a05ca16"} Oct 31 05:06:40.738184 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:40.738128 4376 scope.go:122] "RemoveContainer" containerID="d57b3926ea3879d4a9753c3e1bbbdaef248a656063ec0fb53012bb2b44d84522" Oct 31 05:06:40.739662 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:40.739611 4376 generic.go:356] "Generic (PLEG): container finished" podID="48d10907-ef2a-43b7-b481-746473b85ef6" containerID="09b3e5cb1833e3454e5c12041f09e7d184ec2384206f12e976c9a309f8443dad" exitCode=0 Oct 31 05:06:40.739760 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:40.739661 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c1fe26-7c889f576c-mptbr" event={"ID":"48d10907-ef2a-43b7-b481-746473b85ef6","Type":"ContainerDied","Data":"09b3e5cb1833e3454e5c12041f09e7d184ec2384206f12e976c9a309f8443dad"} Oct 31 05:06:40.739760 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:40.739676 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c1fe26-7c889f576c-mptbr" Oct 31 05:06:40.739760 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:40.739705 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c1fe26-7c889f576c-mptbr" event={"ID":"48d10907-ef2a-43b7-b481-746473b85ef6","Type":"ContainerDied","Data":"00699118d4af51c9f82f6b559a963358bb4a61fd80c0c4233c0d1d4e22ad6ba2"} Oct 31 05:06:40.749895 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:40.749877 4376 scope.go:122] "RemoveContainer" containerID="d57b3926ea3879d4a9753c3e1bbbdaef248a656063ec0fb53012bb2b44d84522" Oct 31 05:06:40.750173 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:06:40.750152 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d57b3926ea3879d4a9753c3e1bbbdaef248a656063ec0fb53012bb2b44d84522\": container with ID starting with d57b3926ea3879d4a9753c3e1bbbdaef248a656063ec0fb53012bb2b44d84522 not found: ID does not exist" containerID="d57b3926ea3879d4a9753c3e1bbbdaef248a656063ec0fb53012bb2b44d84522" Oct 31 05:06:40.750264 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:40.750183 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d57b3926ea3879d4a9753c3e1bbbdaef248a656063ec0fb53012bb2b44d84522"} err="failed to get container status \"d57b3926ea3879d4a9753c3e1bbbdaef248a656063ec0fb53012bb2b44d84522\": rpc error: code = NotFound desc = could not find container \"d57b3926ea3879d4a9753c3e1bbbdaef248a656063ec0fb53012bb2b44d84522\": container with ID starting with d57b3926ea3879d4a9753c3e1bbbdaef248a656063ec0fb53012bb2b44d84522 not found: ID does not exist" Oct 31 05:06:40.750264 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:40.750204 4376 scope.go:122] "RemoveContainer" containerID="09b3e5cb1833e3454e5c12041f09e7d184ec2384206f12e976c9a309f8443dad" Oct 31 05:06:40.760302 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:40.760286 4376 scope.go:122] "RemoveContainer" containerID="09b3e5cb1833e3454e5c12041f09e7d184ec2384206f12e976c9a309f8443dad" Oct 31 05:06:40.760590 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:06:40.760571 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09b3e5cb1833e3454e5c12041f09e7d184ec2384206f12e976c9a309f8443dad\": container with ID starting with 09b3e5cb1833e3454e5c12041f09e7d184ec2384206f12e976c9a309f8443dad not found: ID does not exist" containerID="09b3e5cb1833e3454e5c12041f09e7d184ec2384206f12e976c9a309f8443dad" Oct 31 05:06:40.760652 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:40.760596 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09b3e5cb1833e3454e5c12041f09e7d184ec2384206f12e976c9a309f8443dad"} err="failed to get container status \"09b3e5cb1833e3454e5c12041f09e7d184ec2384206f12e976c9a309f8443dad\": rpc error: code = NotFound desc = could not find container \"09b3e5cb1833e3454e5c12041f09e7d184ec2384206f12e976c9a309f8443dad\": container with ID starting with 09b3e5cb1833e3454e5c12041f09e7d184ec2384206f12e976c9a309f8443dad not found: ID does not exist" Oct 31 05:06:40.784771 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:40.784737 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-12c0bc-69fdd4b4ff-n6qfz"] Oct 31 05:06:40.800058 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:40.800031 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c1fe26-7c889f576c-mptbr"] Oct 31 05:06:40.805594 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:40.805571 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c1fe26-7c889f576c-mptbr"] Oct 31 05:06:40.846915 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:40.846883 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c1fe26-7c97f95bc7-knlhj"] Oct 31 05:06:40.853360 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:40.853316 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c1fe26-7c97f95bc7-knlhj"] Oct 31 05:06:41.080777 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:06:41.080646 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 05:06:42.447550 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:42.447516 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48d10907-ef2a-43b7-b481-746473b85ef6" path="/var/lib/kubelet/pods/48d10907-ef2a-43b7-b481-746473b85ef6/volumes" Oct 31 05:06:42.448105 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:42.448086 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d918c33-5fb2-4f29-a616-c29446e15ff2" path="/var/lib/kubelet/pods/7d918c33-5fb2-4f29-a616-c29446e15ff2/volumes" Oct 31 05:06:45.785133 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:45.785095 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-12c0bc-69fdd4b4ff-n6qfz" podUID="7b2dd0ee-880c-4538-b706-3412b77b7a53" containerName="payload-pre-processing" containerID="cri-o://9975fba59982c4c92ecf05c9f0e39b77d40553c133dc3bd3ba89b4d4b81e0ab9" gracePeriod=25 Oct 31 05:06:53.421860 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:53.421824 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-fb346b-openshift-default-744c56d4f9-mhkhz"] Oct 31 05:06:53.422552 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:53.422154 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-fb346b-openshift-default-744c56d4f9-mhkhz" podUID="a6c79800-45c4-4778-91ef-027ceba7bb8a" containerName="istio-proxy" containerID="cri-o://883f4122c1c076a8437e00ec21b3a649c13dc38046b2ca718b01132dfb4b1418" gracePeriod=30 Oct 31 05:06:54.924971 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:54.924928 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-12c0bc-69fdd4b4ff-n6qfz" podUID="7b2dd0ee-880c-4538-b706-3412b77b7a53" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 05:06:58.784731 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:58.784708 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-fb346b-openshift-default-744c56d4f9-mhkhz" Oct 31 05:06:58.882702 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:58.882673 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6c79800-45c4-4778-91ef-027ceba7bb8a-kube-api-access-vvbtj\" (UniqueName: \"kubernetes.io/projected/a6c79800-45c4-4778-91ef-027ceba7bb8a-kube-api-access-vvbtj\") pod \"a6c79800-45c4-4778-91ef-027ceba7bb8a\" (UID: \"a6c79800-45c4-4778-91ef-027ceba7bb8a\") " Oct 31 05:06:58.882850 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:58.882804 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a6c79800-45c4-4778-91ef-027ceba7bb8a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6c79800-45c4-4778-91ef-027ceba7bb8a-istio-podinfo\") pod \"a6c79800-45c4-4778-91ef-027ceba7bb8a\" (UID: \"a6c79800-45c4-4778-91ef-027ceba7bb8a\") " Oct 31 05:06:58.882900 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:58.882855 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6c79800-45c4-4778-91ef-027ceba7bb8a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6c79800-45c4-4778-91ef-027ceba7bb8a-istio-envoy\") pod \"a6c79800-45c4-4778-91ef-027ceba7bb8a\" (UID: \"a6c79800-45c4-4778-91ef-027ceba7bb8a\") " Oct 31 05:06:58.882900 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:58.882891 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6c79800-45c4-4778-91ef-027ceba7bb8a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6c79800-45c4-4778-91ef-027ceba7bb8a-workload-certs\") pod \"a6c79800-45c4-4778-91ef-027ceba7bb8a\" (UID: \"a6c79800-45c4-4778-91ef-027ceba7bb8a\") " Oct 31 05:06:58.882991 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:58.882957 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6c79800-45c4-4778-91ef-027ceba7bb8a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6c79800-45c4-4778-91ef-027ceba7bb8a-credential-socket\") pod \"a6c79800-45c4-4778-91ef-027ceba7bb8a\" (UID: \"a6c79800-45c4-4778-91ef-027ceba7bb8a\") " Oct 31 05:06:58.883189 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:58.883112 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6c79800-45c4-4778-91ef-027ceba7bb8a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6c79800-45c4-4778-91ef-027ceba7bb8a-workload-socket\") pod \"a6c79800-45c4-4778-91ef-027ceba7bb8a\" (UID: \"a6c79800-45c4-4778-91ef-027ceba7bb8a\") " Oct 31 05:06:58.883189 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:58.883154 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6c79800-45c4-4778-91ef-027ceba7bb8a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6c79800-45c4-4778-91ef-027ceba7bb8a-istio-data\") pod \"a6c79800-45c4-4778-91ef-027ceba7bb8a\" (UID: \"a6c79800-45c4-4778-91ef-027ceba7bb8a\") " Oct 31 05:06:58.883439 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:58.883206 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6c79800-45c4-4778-91ef-027ceba7bb8a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6c79800-45c4-4778-91ef-027ceba7bb8a-istiod-ca-cert\") pod \"a6c79800-45c4-4778-91ef-027ceba7bb8a\" (UID: \"a6c79800-45c4-4778-91ef-027ceba7bb8a\") " Oct 31 05:06:58.883439 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:58.883238 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6c79800-45c4-4778-91ef-027ceba7bb8a-istio-token\" (UniqueName: \"kubernetes.io/projected/a6c79800-45c4-4778-91ef-027ceba7bb8a-istio-token\") pod \"a6c79800-45c4-4778-91ef-027ceba7bb8a\" (UID: \"a6c79800-45c4-4778-91ef-027ceba7bb8a\") " Oct 31 05:06:58.883654 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:58.883230 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6c79800-45c4-4778-91ef-027ceba7bb8a-workload-certs" pod "a6c79800-45c4-4778-91ef-027ceba7bb8a" (UID: "a6c79800-45c4-4778-91ef-027ceba7bb8a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:06:58.883654 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:58.883255 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6c79800-45c4-4778-91ef-027ceba7bb8a-credential-socket" pod "a6c79800-45c4-4778-91ef-027ceba7bb8a" (UID: "a6c79800-45c4-4778-91ef-027ceba7bb8a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:06:58.883654 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:58.883421 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6c79800-45c4-4778-91ef-027ceba7bb8a-workload-socket" pod "a6c79800-45c4-4778-91ef-027ceba7bb8a" (UID: "a6c79800-45c4-4778-91ef-027ceba7bb8a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:06:58.883654 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:58.883519 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6c79800-45c4-4778-91ef-027ceba7bb8a-istio-data" pod "a6c79800-45c4-4778-91ef-027ceba7bb8a" (UID: "a6c79800-45c4-4778-91ef-027ceba7bb8a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:06:58.883654 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:58.883602 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6c79800-45c4-4778-91ef-027ceba7bb8a-istiod-ca-cert" pod "a6c79800-45c4-4778-91ef-027ceba7bb8a" (UID: "a6c79800-45c4-4778-91ef-027ceba7bb8a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:06:58.884007 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:58.883808 4376 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6c79800-45c4-4778-91ef-027ceba7bb8a-workload-socket\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:06:58.884007 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:58.883823 4376 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6c79800-45c4-4778-91ef-027ceba7bb8a-istio-data\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:06:58.884007 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:58.883831 4376 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6c79800-45c4-4778-91ef-027ceba7bb8a-istiod-ca-cert\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:06:58.884007 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:58.883844 4376 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6c79800-45c4-4778-91ef-027ceba7bb8a-workload-certs\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:06:58.884007 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:58.883853 4376 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6c79800-45c4-4778-91ef-027ceba7bb8a-credential-socket\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:06:58.885041 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:58.885014 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a6c79800-45c4-4778-91ef-027ceba7bb8a-istio-podinfo" pod "a6c79800-45c4-4778-91ef-027ceba7bb8a" (UID: "a6c79800-45c4-4778-91ef-027ceba7bb8a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 05:06:58.885159 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:58.885133 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6c79800-45c4-4778-91ef-027ceba7bb8a-kube-api-access-vvbtj" pod "a6c79800-45c4-4778-91ef-027ceba7bb8a" (UID: "a6c79800-45c4-4778-91ef-027ceba7bb8a"). InnerVolumeSpecName "kube-api-access-vvbtj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:06:58.885306 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:58.885291 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6c79800-45c4-4778-91ef-027ceba7bb8a-istio-envoy" pod "a6c79800-45c4-4778-91ef-027ceba7bb8a" (UID: "a6c79800-45c4-4778-91ef-027ceba7bb8a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:06:58.885515 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:58.885501 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6c79800-45c4-4778-91ef-027ceba7bb8a-istio-token" pod "a6c79800-45c4-4778-91ef-027ceba7bb8a" (UID: "a6c79800-45c4-4778-91ef-027ceba7bb8a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:06:58.937880 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:58.937842 4376 generic.go:356] "Generic (PLEG): container finished" podID="a6c79800-45c4-4778-91ef-027ceba7bb8a" containerID="883f4122c1c076a8437e00ec21b3a649c13dc38046b2ca718b01132dfb4b1418" exitCode=0 Oct 31 05:06:58.938126 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:58.937934 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-fb346b-openshift-default-744c56d4f9-mhkhz" Oct 31 05:06:58.938126 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:58.937976 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-fb346b-openshift-default-744c56d4f9-mhkhz" event={"ID":"a6c79800-45c4-4778-91ef-027ceba7bb8a","Type":"ContainerDied","Data":"883f4122c1c076a8437e00ec21b3a649c13dc38046b2ca718b01132dfb4b1418"} Oct 31 05:06:58.938126 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:58.938028 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-fb346b-openshift-default-744c56d4f9-mhkhz" event={"ID":"a6c79800-45c4-4778-91ef-027ceba7bb8a","Type":"ContainerDied","Data":"e611feb531d0f329989673e2313b19071545068c4a39ee057fcee8e299584896"} Oct 31 05:06:58.938126 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:58.938058 4376 scope.go:122] "RemoveContainer" containerID="883f4122c1c076a8437e00ec21b3a649c13dc38046b2ca718b01132dfb4b1418" Oct 31 05:06:58.952402 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:58.952379 4376 scope.go:122] "RemoveContainer" containerID="fb3ba42b1998b8d6c9d673a03e1c679051143e65e9356c84feacc830afa26e8b" Oct 31 05:06:58.962902 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:58.962881 4376 scope.go:122] "RemoveContainer" containerID="883f4122c1c076a8437e00ec21b3a649c13dc38046b2ca718b01132dfb4b1418" Oct 31 05:06:58.963190 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:06:58.963172 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"883f4122c1c076a8437e00ec21b3a649c13dc38046b2ca718b01132dfb4b1418\": container with ID starting with 883f4122c1c076a8437e00ec21b3a649c13dc38046b2ca718b01132dfb4b1418 not found: ID does not exist" containerID="883f4122c1c076a8437e00ec21b3a649c13dc38046b2ca718b01132dfb4b1418" Oct 31 05:06:58.963245 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:58.963200 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"883f4122c1c076a8437e00ec21b3a649c13dc38046b2ca718b01132dfb4b1418"} err="failed to get container status \"883f4122c1c076a8437e00ec21b3a649c13dc38046b2ca718b01132dfb4b1418\": rpc error: code = NotFound desc = could not find container \"883f4122c1c076a8437e00ec21b3a649c13dc38046b2ca718b01132dfb4b1418\": container with ID starting with 883f4122c1c076a8437e00ec21b3a649c13dc38046b2ca718b01132dfb4b1418 not found: ID does not exist" Oct 31 05:06:58.963245 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:58.963216 4376 scope.go:122] "RemoveContainer" containerID="fb3ba42b1998b8d6c9d673a03e1c679051143e65e9356c84feacc830afa26e8b" Oct 31 05:06:58.963532 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:06:58.963513 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb3ba42b1998b8d6c9d673a03e1c679051143e65e9356c84feacc830afa26e8b\": container with ID starting with fb3ba42b1998b8d6c9d673a03e1c679051143e65e9356c84feacc830afa26e8b not found: ID does not exist" containerID="fb3ba42b1998b8d6c9d673a03e1c679051143e65e9356c84feacc830afa26e8b" Oct 31 05:06:58.963588 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:58.963541 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb3ba42b1998b8d6c9d673a03e1c679051143e65e9356c84feacc830afa26e8b"} err="failed to get container status \"fb3ba42b1998b8d6c9d673a03e1c679051143e65e9356c84feacc830afa26e8b\": rpc error: code = NotFound desc = could not find container \"fb3ba42b1998b8d6c9d673a03e1c679051143e65e9356c84feacc830afa26e8b\": container with ID starting with fb3ba42b1998b8d6c9d673a03e1c679051143e65e9356c84feacc830afa26e8b not found: ID does not exist" Oct 31 05:06:58.984988 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:58.984966 4376 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6c79800-45c4-4778-91ef-027ceba7bb8a-istio-token\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:06:58.985077 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:58.984991 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvbtj\" (UniqueName: \"kubernetes.io/projected/a6c79800-45c4-4778-91ef-027ceba7bb8a-kube-api-access-vvbtj\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:06:58.985077 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:58.985001 4376 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6c79800-45c4-4778-91ef-027ceba7bb8a-istio-podinfo\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:06:58.985077 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:58.985009 4376 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6c79800-45c4-4778-91ef-027ceba7bb8a-istio-envoy\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:06:59.008281 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:59.008225 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-fb346b-openshift-default-744c56d4f9-mhkhz"] Oct 31 05:06:59.012481 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:06:59.012464 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-fb346b-openshift-default-744c56d4f9-mhkhz"] Oct 31 05:07:00.502110 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:07:00.502079 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6c79800-45c4-4778-91ef-027ceba7bb8a" path="/var/lib/kubelet/pods/a6c79800-45c4-4778-91ef-027ceba7bb8a/volumes" Oct 31 05:07:02.015050 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:07:02.015017 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-47e9af-74ddc46b6c-dbghm"] Oct 31 05:07:04.977007 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:07:04.976969 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-12c0bc-69fdd4b4ff-n6qfz" podUID="7b2dd0ee-880c-4538-b706-3412b77b7a53" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 05:07:06.002478 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:07:06.002449 4376 generic.go:356] "Generic (PLEG): container finished" podID="7b2dd0ee-880c-4538-b706-3412b77b7a53" containerID="9975fba59982c4c92ecf05c9f0e39b77d40553c133dc3bd3ba89b4d4b81e0ab9" exitCode=0 Oct 31 05:07:06.002861 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:07:06.002524 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-12c0bc-69fdd4b4ff-n6qfz" event={"ID":"7b2dd0ee-880c-4538-b706-3412b77b7a53","Type":"ContainerDied","Data":"9975fba59982c4c92ecf05c9f0e39b77d40553c133dc3bd3ba89b4d4b81e0ab9"} Oct 31 05:07:06.067934 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:07:06.067908 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-12c0bc-69fdd4b4ff-n6qfz" Oct 31 05:07:06.089122 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:07:06.089095 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b2dd0ee-880c-4538-b706-3412b77b7a53-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b2dd0ee-880c-4538-b706-3412b77b7a53-plugins-config-volume\") pod \"7b2dd0ee-880c-4538-b706-3412b77b7a53\" (UID: \"7b2dd0ee-880c-4538-b706-3412b77b7a53\") " Oct 31 05:07:06.089248 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:07:06.089171 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b2dd0ee-880c-4538-b706-3412b77b7a53-kube-api-access-g6brh\" (UniqueName: \"kubernetes.io/projected/7b2dd0ee-880c-4538-b706-3412b77b7a53-kube-api-access-g6brh\") pod \"7b2dd0ee-880c-4538-b706-3412b77b7a53\" (UID: \"7b2dd0ee-880c-4538-b706-3412b77b7a53\") " Oct 31 05:07:06.089498 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:07:06.089477 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b2dd0ee-880c-4538-b706-3412b77b7a53-plugins-config-volume" pod "7b2dd0ee-880c-4538-b706-3412b77b7a53" (UID: "7b2dd0ee-880c-4538-b706-3412b77b7a53"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:07:06.089770 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:07:06.089748 4376 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b2dd0ee-880c-4538-b706-3412b77b7a53-plugins-config-volume\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:07:06.091282 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:07:06.091262 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b2dd0ee-880c-4538-b706-3412b77b7a53-kube-api-access-g6brh" pod "7b2dd0ee-880c-4538-b706-3412b77b7a53" (UID: "7b2dd0ee-880c-4538-b706-3412b77b7a53"). InnerVolumeSpecName "kube-api-access-g6brh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:07:06.191407 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:07:06.191375 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g6brh\" (UniqueName: \"kubernetes.io/projected/7b2dd0ee-880c-4538-b706-3412b77b7a53-kube-api-access-g6brh\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:07:07.011661 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:07:07.011624 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-12c0bc-69fdd4b4ff-n6qfz" event={"ID":"7b2dd0ee-880c-4538-b706-3412b77b7a53","Type":"ContainerDied","Data":"66f0d2b537fb1b958eeca84f5330f6f062efe43cdd118075f7524a71f2ce89dd"} Oct 31 05:07:07.012128 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:07:07.011678 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-12c0bc-69fdd4b4ff-n6qfz" Oct 31 05:07:07.012128 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:07:07.011690 4376 scope.go:122] "RemoveContainer" containerID="9975fba59982c4c92ecf05c9f0e39b77d40553c133dc3bd3ba89b4d4b81e0ab9" Oct 31 05:07:07.015782 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:07:07.015756 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-47e9af-74ddc46b6c-dbghm" podUID="e1abe8c9-b4dc-4380-8495-a0ce56fb2734" containerName="payload-pre-processing" containerID="cri-o://4f7ca525d6dac0927166bce87df58750e7de777702ea37d257a2204892174dd3" gracePeriod=25 Oct 31 05:07:07.070351 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:07:07.070318 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-12c0bc-69fdd4b4ff-n6qfz"] Oct 31 05:07:07.084542 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:07:07.084521 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-12c0bc-69fdd4b4ff-n6qfz"] Oct 31 05:07:08.445678 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:07:08.445641 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b2dd0ee-880c-4538-b706-3412b77b7a53" path="/var/lib/kubelet/pods/7b2dd0ee-880c-4538-b706-3412b77b7a53/volumes" Oct 31 05:07:11.008567 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:07:11.008529 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 05:07:14.656495 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:07:14.656461 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-c1fe26-openshift-default-88665c78c-frz7j"] Oct 31 05:07:14.657030 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:07:14.656670 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-c1fe26-openshift-default-88665c78c-frz7j" podUID="babafa7c-43f1-4ca0-aee3-6b8b4bda5dd7" containerName="istio-proxy" containerID="cri-o://0a8065bdd935048ca341ded4bc921cb4e4f6f813e828f3ddf717c3b72e65ed87" gracePeriod=30 Oct 31 05:07:15.163374 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:07:15.163321 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-47e9af-74ddc46b6c-dbghm" podUID="e1abe8c9-b4dc-4380-8495-a0ce56fb2734" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 05:07:16.102295 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:07:16.102257 4376 patch_prober.go:29] interesting pod/e2e-inf-b-c1fe26-openshift-default-88665c78c-frz7j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 05:07:16.102739 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:07:16.102320 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-c1fe26-openshift-default-88665c78c-frz7j" podUID="babafa7c-43f1-4ca0-aee3-6b8b4bda5dd7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 05:07:19.948790 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:07:19.948755 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c1fe26-openshift-default-88665c78c-frz7j" Oct 31 05:07:20.031958 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:07:20.031924 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/babafa7c-43f1-4ca0-aee3-6b8b4bda5dd7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/babafa7c-43f1-4ca0-aee3-6b8b4bda5dd7-istio-data\") pod \"babafa7c-43f1-4ca0-aee3-6b8b4bda5dd7\" (UID: \"babafa7c-43f1-4ca0-aee3-6b8b4bda5dd7\") " Oct 31 05:07:20.032148 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:07:20.032027 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/babafa7c-43f1-4ca0-aee3-6b8b4bda5dd7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/babafa7c-43f1-4ca0-aee3-6b8b4bda5dd7-workload-socket\") pod \"babafa7c-43f1-4ca0-aee3-6b8b4bda5dd7\" (UID: \"babafa7c-43f1-4ca0-aee3-6b8b4bda5dd7\") " Oct 31 05:07:20.032148 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:07:20.032061 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/babafa7c-43f1-4ca0-aee3-6b8b4bda5dd7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/babafa7c-43f1-4ca0-aee3-6b8b4bda5dd7-istiod-ca-cert\") pod \"babafa7c-43f1-4ca0-aee3-6b8b4bda5dd7\" (UID: \"babafa7c-43f1-4ca0-aee3-6b8b4bda5dd7\") " Oct 31 05:07:20.032148 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:07:20.032095 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/babafa7c-43f1-4ca0-aee3-6b8b4bda5dd7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/babafa7c-43f1-4ca0-aee3-6b8b4bda5dd7-workload-certs\") pod \"babafa7c-43f1-4ca0-aee3-6b8b4bda5dd7\" (UID: \"babafa7c-43f1-4ca0-aee3-6b8b4bda5dd7\") " Oct 31 05:07:20.032148 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:07:20.032118 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/babafa7c-43f1-4ca0-aee3-6b8b4bda5dd7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/babafa7c-43f1-4ca0-aee3-6b8b4bda5dd7-istio-podinfo\") pod \"babafa7c-43f1-4ca0-aee3-6b8b4bda5dd7\" (UID: \"babafa7c-43f1-4ca0-aee3-6b8b4bda5dd7\") " Oct 31 05:07:20.032434 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:07:20.032158 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/babafa7c-43f1-4ca0-aee3-6b8b4bda5dd7-kube-api-access-sw6tj\" (UniqueName: \"kubernetes.io/projected/babafa7c-43f1-4ca0-aee3-6b8b4bda5dd7-kube-api-access-sw6tj\") pod \"babafa7c-43f1-4ca0-aee3-6b8b4bda5dd7\" (UID: \"babafa7c-43f1-4ca0-aee3-6b8b4bda5dd7\") " Oct 31 05:07:20.032434 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:07:20.032188 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/babafa7c-43f1-4ca0-aee3-6b8b4bda5dd7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/babafa7c-43f1-4ca0-aee3-6b8b4bda5dd7-istio-envoy\") pod \"babafa7c-43f1-4ca0-aee3-6b8b4bda5dd7\" (UID: \"babafa7c-43f1-4ca0-aee3-6b8b4bda5dd7\") " Oct 31 05:07:20.032434 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:07:20.032246 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/babafa7c-43f1-4ca0-aee3-6b8b4bda5dd7-istio-token\" (UniqueName: \"kubernetes.io/projected/babafa7c-43f1-4ca0-aee3-6b8b4bda5dd7-istio-token\") pod \"babafa7c-43f1-4ca0-aee3-6b8b4bda5dd7\" (UID: \"babafa7c-43f1-4ca0-aee3-6b8b4bda5dd7\") " Oct 31 05:07:20.032434 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:07:20.032318 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/babafa7c-43f1-4ca0-aee3-6b8b4bda5dd7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/babafa7c-43f1-4ca0-aee3-6b8b4bda5dd7-credential-socket\") pod \"babafa7c-43f1-4ca0-aee3-6b8b4bda5dd7\" (UID: \"babafa7c-43f1-4ca0-aee3-6b8b4bda5dd7\") " Oct 31 05:07:20.033634 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:07:20.033605 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/babafa7c-43f1-4ca0-aee3-6b8b4bda5dd7-istio-data" pod "babafa7c-43f1-4ca0-aee3-6b8b4bda5dd7" (UID: "babafa7c-43f1-4ca0-aee3-6b8b4bda5dd7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:07:20.034780 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:07:20.033625 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/babafa7c-43f1-4ca0-aee3-6b8b4bda5dd7-workload-socket" pod "babafa7c-43f1-4ca0-aee3-6b8b4bda5dd7" (UID: "babafa7c-43f1-4ca0-aee3-6b8b4bda5dd7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:07:20.034929 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:07:20.033887 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/babafa7c-43f1-4ca0-aee3-6b8b4bda5dd7-istiod-ca-cert" pod "babafa7c-43f1-4ca0-aee3-6b8b4bda5dd7" (UID: "babafa7c-43f1-4ca0-aee3-6b8b4bda5dd7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:07:20.035124 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:07:20.034004 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/babafa7c-43f1-4ca0-aee3-6b8b4bda5dd7-workload-certs" pod "babafa7c-43f1-4ca0-aee3-6b8b4bda5dd7" (UID: "babafa7c-43f1-4ca0-aee3-6b8b4bda5dd7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:07:20.035236 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:07:20.034053 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/babafa7c-43f1-4ca0-aee3-6b8b4bda5dd7-credential-socket" pod "babafa7c-43f1-4ca0-aee3-6b8b4bda5dd7" (UID: "babafa7c-43f1-4ca0-aee3-6b8b4bda5dd7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:07:20.045685 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:07:20.045651 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/babafa7c-43f1-4ca0-aee3-6b8b4bda5dd7-istio-envoy" pod "babafa7c-43f1-4ca0-aee3-6b8b4bda5dd7" (UID: "babafa7c-43f1-4ca0-aee3-6b8b4bda5dd7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:07:20.045860 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:07:20.045750 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/babafa7c-43f1-4ca0-aee3-6b8b4bda5dd7-istio-podinfo" pod "babafa7c-43f1-4ca0-aee3-6b8b4bda5dd7" (UID: "babafa7c-43f1-4ca0-aee3-6b8b4bda5dd7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 05:07:20.045860 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:07:20.045800 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/babafa7c-43f1-4ca0-aee3-6b8b4bda5dd7-istio-token" pod "babafa7c-43f1-4ca0-aee3-6b8b4bda5dd7" (UID: "babafa7c-43f1-4ca0-aee3-6b8b4bda5dd7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:07:20.046919 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:07:20.046895 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/babafa7c-43f1-4ca0-aee3-6b8b4bda5dd7-kube-api-access-sw6tj" pod "babafa7c-43f1-4ca0-aee3-6b8b4bda5dd7" (UID: "babafa7c-43f1-4ca0-aee3-6b8b4bda5dd7"). InnerVolumeSpecName "kube-api-access-sw6tj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:07:20.134247 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:07:20.134217 4376 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/babafa7c-43f1-4ca0-aee3-6b8b4bda5dd7-workload-socket\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:07:20.134247 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:07:20.134244 4376 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/babafa7c-43f1-4ca0-aee3-6b8b4bda5dd7-istiod-ca-cert\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:07:20.134434 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:07:20.134254 4376 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/babafa7c-43f1-4ca0-aee3-6b8b4bda5dd7-workload-certs\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:07:20.134434 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:07:20.134262 4376 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/babafa7c-43f1-4ca0-aee3-6b8b4bda5dd7-istio-podinfo\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:07:20.134434 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:07:20.134272 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sw6tj\" (UniqueName: \"kubernetes.io/projected/babafa7c-43f1-4ca0-aee3-6b8b4bda5dd7-kube-api-access-sw6tj\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:07:20.134434 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:07:20.134281 4376 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/babafa7c-43f1-4ca0-aee3-6b8b4bda5dd7-istio-envoy\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:07:20.134434 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:07:20.134288 4376 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/babafa7c-43f1-4ca0-aee3-6b8b4bda5dd7-istio-token\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:07:20.134434 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:07:20.134297 4376 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/babafa7c-43f1-4ca0-aee3-6b8b4bda5dd7-credential-socket\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:07:20.134434 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:07:20.134305 4376 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/babafa7c-43f1-4ca0-aee3-6b8b4bda5dd7-istio-data\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:07:20.152462 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:07:20.152435 4376 generic.go:356] "Generic (PLEG): container finished" podID="babafa7c-43f1-4ca0-aee3-6b8b4bda5dd7" containerID="0a8065bdd935048ca341ded4bc921cb4e4f6f813e828f3ddf717c3b72e65ed87" exitCode=0 Oct 31 05:07:20.152567 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:07:20.152485 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c1fe26-openshift-default-88665c78c-frz7j" Oct 31 05:07:20.152567 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:07:20.152517 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c1fe26-openshift-default-88665c78c-frz7j" event={"ID":"babafa7c-43f1-4ca0-aee3-6b8b4bda5dd7","Type":"ContainerDied","Data":"0a8065bdd935048ca341ded4bc921cb4e4f6f813e828f3ddf717c3b72e65ed87"} Oct 31 05:07:20.152567 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:07:20.152559 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c1fe26-openshift-default-88665c78c-frz7j" event={"ID":"babafa7c-43f1-4ca0-aee3-6b8b4bda5dd7","Type":"ContainerDied","Data":"64545fb67fec6056521fe84d017f9feed8e1cb21e595976a7595d01fde152f37"} Oct 31 05:07:20.152715 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:07:20.152578 4376 scope.go:122] "RemoveContainer" containerID="0a8065bdd935048ca341ded4bc921cb4e4f6f813e828f3ddf717c3b72e65ed87" Oct 31 05:07:20.168609 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:07:20.168589 4376 scope.go:122] "RemoveContainer" containerID="0a8065bdd935048ca341ded4bc921cb4e4f6f813e828f3ddf717c3b72e65ed87" Oct 31 05:07:20.168909 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:07:20.168886 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a8065bdd935048ca341ded4bc921cb4e4f6f813e828f3ddf717c3b72e65ed87\": container with ID starting with 0a8065bdd935048ca341ded4bc921cb4e4f6f813e828f3ddf717c3b72e65ed87 not found: ID does not exist" containerID="0a8065bdd935048ca341ded4bc921cb4e4f6f813e828f3ddf717c3b72e65ed87" Oct 31 05:07:20.169008 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:07:20.168914 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a8065bdd935048ca341ded4bc921cb4e4f6f813e828f3ddf717c3b72e65ed87"} err="failed to get container status \"0a8065bdd935048ca341ded4bc921cb4e4f6f813e828f3ddf717c3b72e65ed87\": rpc error: code = NotFound desc = could not find container \"0a8065bdd935048ca341ded4bc921cb4e4f6f813e828f3ddf717c3b72e65ed87\": container with ID starting with 0a8065bdd935048ca341ded4bc921cb4e4f6f813e828f3ddf717c3b72e65ed87 not found: ID does not exist" Oct 31 05:07:20.213643 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:07:20.213582 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-c1fe26-openshift-default-88665c78c-frz7j"] Oct 31 05:07:20.218075 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:07:20.218057 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-c1fe26-openshift-default-88665c78c-frz7j"] Oct 31 05:07:20.446765 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:07:20.446733 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="babafa7c-43f1-4ca0-aee3-6b8b4bda5dd7" path="/var/lib/kubelet/pods/babafa7c-43f1-4ca0-aee3-6b8b4bda5dd7/volumes" Oct 31 05:07:25.163552 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:07:25.163511 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-47e9af-74ddc46b6c-dbghm" podUID="e1abe8c9-b4dc-4380-8495-a0ce56fb2734" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 05:07:27.226088 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:07:27.226052 4376 generic.go:356] "Generic (PLEG): container finished" podID="e1abe8c9-b4dc-4380-8495-a0ce56fb2734" containerID="4f7ca525d6dac0927166bce87df58750e7de777702ea37d257a2204892174dd3" exitCode=0 Oct 31 05:07:27.226533 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:07:27.226119 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-47e9af-74ddc46b6c-dbghm" event={"ID":"e1abe8c9-b4dc-4380-8495-a0ce56fb2734","Type":"ContainerDied","Data":"4f7ca525d6dac0927166bce87df58750e7de777702ea37d257a2204892174dd3"} Oct 31 05:07:27.308014 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:07:27.307982 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-47e9af-74ddc46b6c-dbghm" Oct 31 05:07:27.444897 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:07:27.444861 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1abe8c9-b4dc-4380-8495-a0ce56fb2734-kube-api-access-mh4xq\" (UniqueName: \"kubernetes.io/projected/e1abe8c9-b4dc-4380-8495-a0ce56fb2734-kube-api-access-mh4xq\") pod \"e1abe8c9-b4dc-4380-8495-a0ce56fb2734\" (UID: \"e1abe8c9-b4dc-4380-8495-a0ce56fb2734\") " Oct 31 05:07:27.445127 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:07:27.445113 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1abe8c9-b4dc-4380-8495-a0ce56fb2734-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1abe8c9-b4dc-4380-8495-a0ce56fb2734-plugins-config-volume\") pod \"e1abe8c9-b4dc-4380-8495-a0ce56fb2734\" (UID: \"e1abe8c9-b4dc-4380-8495-a0ce56fb2734\") " Oct 31 05:07:27.445509 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:07:27.445481 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1abe8c9-b4dc-4380-8495-a0ce56fb2734-plugins-config-volume" pod "e1abe8c9-b4dc-4380-8495-a0ce56fb2734" (UID: "e1abe8c9-b4dc-4380-8495-a0ce56fb2734"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:07:27.446956 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:07:27.446927 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1abe8c9-b4dc-4380-8495-a0ce56fb2734-kube-api-access-mh4xq" pod "e1abe8c9-b4dc-4380-8495-a0ce56fb2734" (UID: "e1abe8c9-b4dc-4380-8495-a0ce56fb2734"). InnerVolumeSpecName "kube-api-access-mh4xq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:07:27.547023 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:07:27.546945 4376 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1abe8c9-b4dc-4380-8495-a0ce56fb2734-plugins-config-volume\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:07:27.547023 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:07:27.546970 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mh4xq\" (UniqueName: \"kubernetes.io/projected/e1abe8c9-b4dc-4380-8495-a0ce56fb2734-kube-api-access-mh4xq\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:07:28.235042 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:07:28.235015 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-47e9af-74ddc46b6c-dbghm" Oct 31 05:07:28.235042 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:07:28.235028 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-47e9af-74ddc46b6c-dbghm" event={"ID":"e1abe8c9-b4dc-4380-8495-a0ce56fb2734","Type":"ContainerDied","Data":"0697e71030fc14dd9a279e7ec2b89a4b319d7d302c6c133bfb5e4ea350e3a536"} Oct 31 05:07:28.235575 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:07:28.235109 4376 scope.go:122] "RemoveContainer" containerID="4f7ca525d6dac0927166bce87df58750e7de777702ea37d257a2204892174dd3" Oct 31 05:07:28.294698 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:07:28.294666 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-47e9af-74ddc46b6c-dbghm"] Oct 31 05:07:28.308666 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:07:28.308644 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-47e9af-74ddc46b6c-dbghm"] Oct 31 05:07:28.451684 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:07:28.451642 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1abe8c9-b4dc-4380-8495-a0ce56fb2734" path="/var/lib/kubelet/pods/e1abe8c9-b4dc-4380-8495-a0ce56fb2734/volumes" Oct 31 05:07:38.405282 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:07:38.405251 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-12c0bc-openshift-default-84bddf978c-ghhlp"] Oct 31 05:07:38.405831 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:07:38.405485 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-12c0bc-openshift-default-84bddf978c-ghhlp" podUID="7bde4a1c-a591-4864-8276-2c3a74b99e70" containerName="istio-proxy" containerID="cri-o://26280a9a05b01397f9ac031363d2028b594ef6e4fd2a7cd1456cbf43dbb55b18" gracePeriod=30 Oct 31 05:07:38.975806 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:07:38.975775 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 05:07:41.007692 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:07:41.007642 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 05:07:43.699740 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:07:43.699704 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-12c0bc-openshift-default-84bddf978c-ghhlp" Oct 31 05:07:43.748523 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:07:43.748490 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bde4a1c-a591-4864-8276-2c3a74b99e70-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bde4a1c-a591-4864-8276-2c3a74b99e70-workload-socket\") pod \"7bde4a1c-a591-4864-8276-2c3a74b99e70\" (UID: \"7bde4a1c-a591-4864-8276-2c3a74b99e70\") " Oct 31 05:07:43.748674 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:07:43.748545 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bde4a1c-a591-4864-8276-2c3a74b99e70-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bde4a1c-a591-4864-8276-2c3a74b99e70-credential-socket\") pod \"7bde4a1c-a591-4864-8276-2c3a74b99e70\" (UID: \"7bde4a1c-a591-4864-8276-2c3a74b99e70\") " Oct 31 05:07:43.748674 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:07:43.748623 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7bde4a1c-a591-4864-8276-2c3a74b99e70-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bde4a1c-a591-4864-8276-2c3a74b99e70-istiod-ca-cert\") pod \"7bde4a1c-a591-4864-8276-2c3a74b99e70\" (UID: \"7bde4a1c-a591-4864-8276-2c3a74b99e70\") " Oct 31 05:07:43.748825 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:07:43.748677 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bde4a1c-a591-4864-8276-2c3a74b99e70-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bde4a1c-a591-4864-8276-2c3a74b99e70-workload-certs\") pod \"7bde4a1c-a591-4864-8276-2c3a74b99e70\" (UID: \"7bde4a1c-a591-4864-8276-2c3a74b99e70\") " Oct 31 05:07:43.748825 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:07:43.748725 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bde4a1c-a591-4864-8276-2c3a74b99e70-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bde4a1c-a591-4864-8276-2c3a74b99e70-istio-envoy\") pod \"7bde4a1c-a591-4864-8276-2c3a74b99e70\" (UID: \"7bde4a1c-a591-4864-8276-2c3a74b99e70\") " Oct 31 05:07:43.748825 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:07:43.748753 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bde4a1c-a591-4864-8276-2c3a74b99e70-workload-socket" pod "7bde4a1c-a591-4864-8276-2c3a74b99e70" (UID: "7bde4a1c-a591-4864-8276-2c3a74b99e70"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:07:43.749009 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:07:43.748833 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bde4a1c-a591-4864-8276-2c3a74b99e70-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bde4a1c-a591-4864-8276-2c3a74b99e70-istio-data\") pod \"7bde4a1c-a591-4864-8276-2c3a74b99e70\" (UID: \"7bde4a1c-a591-4864-8276-2c3a74b99e70\") " Oct 31 05:07:43.749009 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:07:43.748910 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bde4a1c-a591-4864-8276-2c3a74b99e70-credential-socket" pod "7bde4a1c-a591-4864-8276-2c3a74b99e70" (UID: "7bde4a1c-a591-4864-8276-2c3a74b99e70"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:07:43.749009 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:07:43.748945 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7bde4a1c-a591-4864-8276-2c3a74b99e70-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bde4a1c-a591-4864-8276-2c3a74b99e70-istio-podinfo\") pod \"7bde4a1c-a591-4864-8276-2c3a74b99e70\" (UID: \"7bde4a1c-a591-4864-8276-2c3a74b99e70\") " Oct 31 05:07:43.749009 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:07:43.748965 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bde4a1c-a591-4864-8276-2c3a74b99e70-workload-certs" pod "7bde4a1c-a591-4864-8276-2c3a74b99e70" (UID: "7bde4a1c-a591-4864-8276-2c3a74b99e70"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:07:43.749009 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:07:43.748984 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bde4a1c-a591-4864-8276-2c3a74b99e70-istio-token\" (UniqueName: \"kubernetes.io/projected/7bde4a1c-a591-4864-8276-2c3a74b99e70-istio-token\") pod \"7bde4a1c-a591-4864-8276-2c3a74b99e70\" (UID: \"7bde4a1c-a591-4864-8276-2c3a74b99e70\") " Oct 31 05:07:43.749289 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:07:43.749027 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bde4a1c-a591-4864-8276-2c3a74b99e70-kube-api-access-7crvm\" (UniqueName: \"kubernetes.io/projected/7bde4a1c-a591-4864-8276-2c3a74b99e70-kube-api-access-7crvm\") pod \"7bde4a1c-a591-4864-8276-2c3a74b99e70\" (UID: \"7bde4a1c-a591-4864-8276-2c3a74b99e70\") " Oct 31 05:07:43.749289 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:07:43.749139 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bde4a1c-a591-4864-8276-2c3a74b99e70-istio-data" pod "7bde4a1c-a591-4864-8276-2c3a74b99e70" (UID: "7bde4a1c-a591-4864-8276-2c3a74b99e70"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:07:43.749289 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:07:43.749047 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bde4a1c-a591-4864-8276-2c3a74b99e70-istiod-ca-cert" pod "7bde4a1c-a591-4864-8276-2c3a74b99e70" (UID: "7bde4a1c-a591-4864-8276-2c3a74b99e70"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:07:43.749869 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:07:43.749846 4376 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bde4a1c-a591-4864-8276-2c3a74b99e70-istiod-ca-cert\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:07:43.749960 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:07:43.749875 4376 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bde4a1c-a591-4864-8276-2c3a74b99e70-workload-certs\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:07:43.749960 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:07:43.749891 4376 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bde4a1c-a591-4864-8276-2c3a74b99e70-istio-data\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:07:43.749960 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:07:43.749905 4376 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bde4a1c-a591-4864-8276-2c3a74b99e70-workload-socket\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:07:43.749960 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:07:43.749919 4376 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bde4a1c-a591-4864-8276-2c3a74b99e70-credential-socket\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:07:43.751093 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:07:43.751073 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7bde4a1c-a591-4864-8276-2c3a74b99e70-istio-podinfo" pod "7bde4a1c-a591-4864-8276-2c3a74b99e70" (UID: "7bde4a1c-a591-4864-8276-2c3a74b99e70"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 05:07:43.751242 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:07:43.751219 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bde4a1c-a591-4864-8276-2c3a74b99e70-istio-envoy" pod "7bde4a1c-a591-4864-8276-2c3a74b99e70" (UID: "7bde4a1c-a591-4864-8276-2c3a74b99e70"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:07:43.751545 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:07:43.751517 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bde4a1c-a591-4864-8276-2c3a74b99e70-kube-api-access-7crvm" pod "7bde4a1c-a591-4864-8276-2c3a74b99e70" (UID: "7bde4a1c-a591-4864-8276-2c3a74b99e70"). InnerVolumeSpecName "kube-api-access-7crvm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:07:43.751661 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:07:43.751542 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bde4a1c-a591-4864-8276-2c3a74b99e70-istio-token" pod "7bde4a1c-a591-4864-8276-2c3a74b99e70" (UID: "7bde4a1c-a591-4864-8276-2c3a74b99e70"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:07:43.851469 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:07:43.851437 4376 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bde4a1c-a591-4864-8276-2c3a74b99e70-istio-podinfo\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:07:43.851469 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:07:43.851465 4376 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bde4a1c-a591-4864-8276-2c3a74b99e70-istio-token\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:07:43.851469 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:07:43.851475 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7crvm\" (UniqueName: \"kubernetes.io/projected/7bde4a1c-a591-4864-8276-2c3a74b99e70-kube-api-access-7crvm\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:07:43.851717 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:07:43.851484 4376 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bde4a1c-a591-4864-8276-2c3a74b99e70-istio-envoy\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:07:44.274144 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:07:44.274108 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-47e9af-openshift-default-5b7fdd8749-hb7v2"] Oct 31 05:07:44.274672 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:07:44.274637 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-47e9af-openshift-default-5b7fdd8749-hb7v2" podUID="341d12d1-db55-4078-acb0-47169d00ac7e" containerName="istio-proxy" containerID="cri-o://dfa799656a62f2e3dcf83ed2c25668b201a65551e0b0a4f37972ceafcf588755" gracePeriod=30 Oct 31 05:07:44.543003 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:07:44.542917 4376 generic.go:356] "Generic (PLEG): container finished" podID="7bde4a1c-a591-4864-8276-2c3a74b99e70" containerID="26280a9a05b01397f9ac031363d2028b594ef6e4fd2a7cd1456cbf43dbb55b18" exitCode=0 Oct 31 05:07:44.543445 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:07:44.542981 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-12c0bc-openshift-default-84bddf978c-ghhlp" Oct 31 05:07:44.544262 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:07:44.542996 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-12c0bc-openshift-default-84bddf978c-ghhlp" event={"ID":"7bde4a1c-a591-4864-8276-2c3a74b99e70","Type":"ContainerDied","Data":"26280a9a05b01397f9ac031363d2028b594ef6e4fd2a7cd1456cbf43dbb55b18"} Oct 31 05:07:44.544494 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:07:44.544468 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-12c0bc-openshift-default-84bddf978c-ghhlp" event={"ID":"7bde4a1c-a591-4864-8276-2c3a74b99e70","Type":"ContainerDied","Data":"2b8f3231fb96c5a5495462598e123caa61adfc6999017d3139d30613273b3cec"} Oct 31 05:07:44.544623 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:07:44.544513 4376 scope.go:122] "RemoveContainer" containerID="26280a9a05b01397f9ac031363d2028b594ef6e4fd2a7cd1456cbf43dbb55b18" Oct 31 05:07:44.555787 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:07:44.555769 4376 scope.go:122] "RemoveContainer" containerID="24de4e330c1dfdcedac40896966a3533b0dcb513c23465a2cebc3c5507138eb8" Oct 31 05:07:44.566819 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:07:44.566792 4376 scope.go:122] "RemoveContainer" containerID="26280a9a05b01397f9ac031363d2028b594ef6e4fd2a7cd1456cbf43dbb55b18" Oct 31 05:07:44.567095 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:07:44.567074 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26280a9a05b01397f9ac031363d2028b594ef6e4fd2a7cd1456cbf43dbb55b18\": container with ID starting with 26280a9a05b01397f9ac031363d2028b594ef6e4fd2a7cd1456cbf43dbb55b18 not found: ID does not exist" containerID="26280a9a05b01397f9ac031363d2028b594ef6e4fd2a7cd1456cbf43dbb55b18" Oct 31 05:07:44.567178 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:07:44.567106 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26280a9a05b01397f9ac031363d2028b594ef6e4fd2a7cd1456cbf43dbb55b18"} err="failed to get container status \"26280a9a05b01397f9ac031363d2028b594ef6e4fd2a7cd1456cbf43dbb55b18\": rpc error: code = NotFound desc = could not find container \"26280a9a05b01397f9ac031363d2028b594ef6e4fd2a7cd1456cbf43dbb55b18\": container with ID starting with 26280a9a05b01397f9ac031363d2028b594ef6e4fd2a7cd1456cbf43dbb55b18 not found: ID does not exist" Oct 31 05:07:44.567178 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:07:44.567129 4376 scope.go:122] "RemoveContainer" containerID="24de4e330c1dfdcedac40896966a3533b0dcb513c23465a2cebc3c5507138eb8" Oct 31 05:07:44.567422 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:07:44.567403 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24de4e330c1dfdcedac40896966a3533b0dcb513c23465a2cebc3c5507138eb8\": container with ID starting with 24de4e330c1dfdcedac40896966a3533b0dcb513c23465a2cebc3c5507138eb8 not found: ID does not exist" containerID="24de4e330c1dfdcedac40896966a3533b0dcb513c23465a2cebc3c5507138eb8" Oct 31 05:07:44.567470 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:07:44.567430 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24de4e330c1dfdcedac40896966a3533b0dcb513c23465a2cebc3c5507138eb8"} err="failed to get container status \"24de4e330c1dfdcedac40896966a3533b0dcb513c23465a2cebc3c5507138eb8\": rpc error: code = NotFound desc = could not find container \"24de4e330c1dfdcedac40896966a3533b0dcb513c23465a2cebc3c5507138eb8\": container with ID starting with 24de4e330c1dfdcedac40896966a3533b0dcb513c23465a2cebc3c5507138eb8 not found: ID does not exist" Oct 31 05:07:44.612349 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:07:44.611476 4376 patch_prober.go:29] interesting pod/e2e-inf-b-47e9af-openshift-default-5b7fdd8749-hb7v2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 05:07:44.612349 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:07:44.611537 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-47e9af-openshift-default-5b7fdd8749-hb7v2" podUID="341d12d1-db55-4078-acb0-47169d00ac7e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 05:07:44.624570 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:07:44.624534 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-12c0bc-openshift-default-84bddf978c-ghhlp"] Oct 31 05:07:44.632352 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:07:44.630231 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-12c0bc-openshift-default-84bddf978c-ghhlp"] Oct 31 05:07:46.444818 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:07:46.444786 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7bde4a1c-a591-4864-8276-2c3a74b99e70" path="/var/lib/kubelet/pods/7bde4a1c-a591-4864-8276-2c3a74b99e70/volumes" Oct 31 05:07:49.567756 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:07:49.567726 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-47e9af-openshift-default-5b7fdd8749-hb7v2" Oct 31 05:07:49.596638 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:07:49.596607 4376 generic.go:356] "Generic (PLEG): container finished" podID="341d12d1-db55-4078-acb0-47169d00ac7e" containerID="dfa799656a62f2e3dcf83ed2c25668b201a65551e0b0a4f37972ceafcf588755" exitCode=0 Oct 31 05:07:49.596798 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:07:49.596655 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-47e9af-openshift-default-5b7fdd8749-hb7v2" Oct 31 05:07:49.596798 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:07:49.596690 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-47e9af-openshift-default-5b7fdd8749-hb7v2" event={"ID":"341d12d1-db55-4078-acb0-47169d00ac7e","Type":"ContainerDied","Data":"dfa799656a62f2e3dcf83ed2c25668b201a65551e0b0a4f37972ceafcf588755"} Oct 31 05:07:49.596798 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:07:49.596746 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-47e9af-openshift-default-5b7fdd8749-hb7v2" event={"ID":"341d12d1-db55-4078-acb0-47169d00ac7e","Type":"ContainerDied","Data":"e33dd1ebd203db805ee2e4d23b76fa616daeb43fc535f107e5c8baddc5cff4b7"} Oct 31 05:07:49.596798 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:07:49.596780 4376 scope.go:122] "RemoveContainer" containerID="dfa799656a62f2e3dcf83ed2c25668b201a65551e0b0a4f37972ceafcf588755" Oct 31 05:07:49.607131 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:07:49.607113 4376 scope.go:122] "RemoveContainer" containerID="dfa799656a62f2e3dcf83ed2c25668b201a65551e0b0a4f37972ceafcf588755" Oct 31 05:07:49.607393 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:07:49.607374 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dfa799656a62f2e3dcf83ed2c25668b201a65551e0b0a4f37972ceafcf588755\": container with ID starting with dfa799656a62f2e3dcf83ed2c25668b201a65551e0b0a4f37972ceafcf588755 not found: ID does not exist" containerID="dfa799656a62f2e3dcf83ed2c25668b201a65551e0b0a4f37972ceafcf588755" Oct 31 05:07:49.607468 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:07:49.607406 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfa799656a62f2e3dcf83ed2c25668b201a65551e0b0a4f37972ceafcf588755"} err="failed to get container status \"dfa799656a62f2e3dcf83ed2c25668b201a65551e0b0a4f37972ceafcf588755\": rpc error: code = NotFound desc = could not find container \"dfa799656a62f2e3dcf83ed2c25668b201a65551e0b0a4f37972ceafcf588755\": container with ID starting with dfa799656a62f2e3dcf83ed2c25668b201a65551e0b0a4f37972ceafcf588755 not found: ID does not exist" Oct 31 05:07:49.637676 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:07:49.637640 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/341d12d1-db55-4078-acb0-47169d00ac7e-kube-api-access-frwnx\" (UniqueName: \"kubernetes.io/projected/341d12d1-db55-4078-acb0-47169d00ac7e-kube-api-access-frwnx\") pod \"341d12d1-db55-4078-acb0-47169d00ac7e\" (UID: \"341d12d1-db55-4078-acb0-47169d00ac7e\") " Oct 31 05:07:49.637832 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:07:49.637725 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/341d12d1-db55-4078-acb0-47169d00ac7e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/341d12d1-db55-4078-acb0-47169d00ac7e-workload-certs\") pod \"341d12d1-db55-4078-acb0-47169d00ac7e\" (UID: \"341d12d1-db55-4078-acb0-47169d00ac7e\") " Oct 31 05:07:49.637832 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:07:49.637825 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/341d12d1-db55-4078-acb0-47169d00ac7e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/341d12d1-db55-4078-acb0-47169d00ac7e-istio-envoy\") pod \"341d12d1-db55-4078-acb0-47169d00ac7e\" (UID: \"341d12d1-db55-4078-acb0-47169d00ac7e\") " Oct 31 05:07:49.637949 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:07:49.637856 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/341d12d1-db55-4078-acb0-47169d00ac7e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/341d12d1-db55-4078-acb0-47169d00ac7e-credential-socket\") pod \"341d12d1-db55-4078-acb0-47169d00ac7e\" (UID: \"341d12d1-db55-4078-acb0-47169d00ac7e\") " Oct 31 05:07:49.637949 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:07:49.637901 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/341d12d1-db55-4078-acb0-47169d00ac7e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/341d12d1-db55-4078-acb0-47169d00ac7e-istio-data\") pod \"341d12d1-db55-4078-acb0-47169d00ac7e\" (UID: \"341d12d1-db55-4078-acb0-47169d00ac7e\") " Oct 31 05:07:49.637949 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:07:49.637936 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/341d12d1-db55-4078-acb0-47169d00ac7e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/341d12d1-db55-4078-acb0-47169d00ac7e-istio-podinfo\") pod \"341d12d1-db55-4078-acb0-47169d00ac7e\" (UID: \"341d12d1-db55-4078-acb0-47169d00ac7e\") " Oct 31 05:07:49.638101 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:07:49.638069 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/341d12d1-db55-4078-acb0-47169d00ac7e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/341d12d1-db55-4078-acb0-47169d00ac7e-workload-socket\") pod \"341d12d1-db55-4078-acb0-47169d00ac7e\" (UID: \"341d12d1-db55-4078-acb0-47169d00ac7e\") " Oct 31 05:07:49.638184 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:07:49.638167 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/341d12d1-db55-4078-acb0-47169d00ac7e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/341d12d1-db55-4078-acb0-47169d00ac7e-istiod-ca-cert\") pod \"341d12d1-db55-4078-acb0-47169d00ac7e\" (UID: \"341d12d1-db55-4078-acb0-47169d00ac7e\") " Oct 31 05:07:49.638292 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:07:49.638274 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/341d12d1-db55-4078-acb0-47169d00ac7e-istio-token\" (UniqueName: \"kubernetes.io/projected/341d12d1-db55-4078-acb0-47169d00ac7e-istio-token\") pod \"341d12d1-db55-4078-acb0-47169d00ac7e\" (UID: \"341d12d1-db55-4078-acb0-47169d00ac7e\") " Oct 31 05:07:49.638614 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:07:49.638583 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/341d12d1-db55-4078-acb0-47169d00ac7e-workload-certs" pod "341d12d1-db55-4078-acb0-47169d00ac7e" (UID: "341d12d1-db55-4078-acb0-47169d00ac7e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:07:49.639100 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:07:49.639073 4376 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/341d12d1-db55-4078-acb0-47169d00ac7e-workload-certs\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:07:49.639192 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:07:49.639115 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/341d12d1-db55-4078-acb0-47169d00ac7e-workload-socket" pod "341d12d1-db55-4078-acb0-47169d00ac7e" (UID: "341d12d1-db55-4078-acb0-47169d00ac7e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:07:49.639310 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:07:49.639285 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/341d12d1-db55-4078-acb0-47169d00ac7e-credential-socket" pod "341d12d1-db55-4078-acb0-47169d00ac7e" (UID: "341d12d1-db55-4078-acb0-47169d00ac7e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:07:49.639523 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:07:49.639498 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/341d12d1-db55-4078-acb0-47169d00ac7e-istiod-ca-cert" pod "341d12d1-db55-4078-acb0-47169d00ac7e" (UID: "341d12d1-db55-4078-acb0-47169d00ac7e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:07:49.639593 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:07:49.639520 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/341d12d1-db55-4078-acb0-47169d00ac7e-istio-data" pod "341d12d1-db55-4078-acb0-47169d00ac7e" (UID: "341d12d1-db55-4078-acb0-47169d00ac7e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:07:49.640041 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:07:49.640015 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/341d12d1-db55-4078-acb0-47169d00ac7e-kube-api-access-frwnx" pod "341d12d1-db55-4078-acb0-47169d00ac7e" (UID: "341d12d1-db55-4078-acb0-47169d00ac7e"). InnerVolumeSpecName "kube-api-access-frwnx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:07:49.640584 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:07:49.640559 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/341d12d1-db55-4078-acb0-47169d00ac7e-istio-envoy" pod "341d12d1-db55-4078-acb0-47169d00ac7e" (UID: "341d12d1-db55-4078-acb0-47169d00ac7e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:07:49.641586 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:07:49.641545 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/341d12d1-db55-4078-acb0-47169d00ac7e-istio-podinfo" pod "341d12d1-db55-4078-acb0-47169d00ac7e" (UID: "341d12d1-db55-4078-acb0-47169d00ac7e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 05:07:49.641842 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:07:49.641816 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/341d12d1-db55-4078-acb0-47169d00ac7e-istio-token" pod "341d12d1-db55-4078-acb0-47169d00ac7e" (UID: "341d12d1-db55-4078-acb0-47169d00ac7e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:07:49.740251 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:07:49.740222 4376 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/341d12d1-db55-4078-acb0-47169d00ac7e-istio-token\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:07:49.740251 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:07:49.740247 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frwnx\" (UniqueName: \"kubernetes.io/projected/341d12d1-db55-4078-acb0-47169d00ac7e-kube-api-access-frwnx\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:07:49.740251 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:07:49.740257 4376 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/341d12d1-db55-4078-acb0-47169d00ac7e-istio-envoy\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:07:49.740518 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:07:49.740268 4376 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/341d12d1-db55-4078-acb0-47169d00ac7e-credential-socket\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:07:49.740518 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:07:49.740278 4376 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/341d12d1-db55-4078-acb0-47169d00ac7e-istio-data\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:07:49.740518 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:07:49.740286 4376 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/341d12d1-db55-4078-acb0-47169d00ac7e-istio-podinfo\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:07:49.740518 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:07:49.740294 4376 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/341d12d1-db55-4078-acb0-47169d00ac7e-workload-socket\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:07:49.740518 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:07:49.740302 4376 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/341d12d1-db55-4078-acb0-47169d00ac7e-istiod-ca-cert\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:07:49.948190 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:07:49.948153 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-47e9af-openshift-default-5b7fdd8749-hb7v2"] Oct 31 05:07:49.953709 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:07:49.953685 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-47e9af-openshift-default-5b7fdd8749-hb7v2"] Oct 31 05:07:50.446112 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:07:50.446079 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="341d12d1-db55-4078-acb0-47169d00ac7e" path="/var/lib/kubelet/pods/341d12d1-db55-4078-acb0-47169d00ac7e/volumes" Oct 31 05:08:11.000399 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:08:11.000357 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 05:08:33.596926 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:08:33.596894 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 05:08:41.000912 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:08:41.000870 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 05:09:11.003174 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:09:11.003134 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 05:09:19.604046 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:09:19.604004 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 05:09:41.002111 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:09:41.002069 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 05:10:11.002503 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:10:11.002467 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 05:10:22.096447 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:10:22.096412 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 05:10:40.597784 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:10:40.597751 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-47c409-8649bf6df7-rbdch"] Oct 31 05:10:41.099158 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:10:41.099117 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 05:10:45.599077 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:10:45.599036 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-47c409-8649bf6df7-rbdch" podUID="f48d27d4-dc90-4bf7-aa79-2b14c97859a2" containerName="payload-processing" containerID="cri-o://0c6c5b9c319ec0aeb984e2f930be79e000795979f7e74f1a42dafcc2a8c8a1f5" gracePeriod=25 Oct 31 05:10:48.167695 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:10:48.167655 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-47c409-8649bf6df7-rbdch" podUID="f48d27d4-dc90-4bf7-aa79-2b14c97859a2" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 05:10:58.168560 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:10:58.168473 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-47c409-8649bf6df7-rbdch" podUID="f48d27d4-dc90-4bf7-aa79-2b14c97859a2" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 05:10:58.750462 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:10:58.750428 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-401761-6ddf8dc4bf-fbwrw"] Oct 31 05:10:58.786111 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:10:58.786075 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-401761-f8dbb88dc-9ks76"] Oct 31 05:11:03.751002 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:03.750956 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-401761-6ddf8dc4bf-fbwrw" podUID="0551b77c-f034-48e4-b643-3361b903df31" containerName="payload-processing" containerID="cri-o://070a0fdc33137ca68093bd7a4a18a9dc4e197d6cf0a9546d9f26c217536715c9" gracePeriod=25 Oct 31 05:11:03.787354 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:03.787276 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-401761-f8dbb88dc-9ks76" podUID="8ab573d4-4b87-40e0-9d16-ff7fdd3c3963" containerName="payload-pre-processing" containerID="cri-o://9ce8304ec062a9b80afc6829cb5375563596b4665dc66b7a68a7f4f27fc85192" gracePeriod=25 Oct 31 05:11:05.879852 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:05.879825 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-47c409-8649bf6df7-rbdch" Oct 31 05:11:05.969510 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:05.969467 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f48d27d4-dc90-4bf7-aa79-2b14c97859a2-kube-api-access-rb87b\" (UniqueName: \"kubernetes.io/projected/f48d27d4-dc90-4bf7-aa79-2b14c97859a2-kube-api-access-rb87b\") pod \"f48d27d4-dc90-4bf7-aa79-2b14c97859a2\" (UID: \"f48d27d4-dc90-4bf7-aa79-2b14c97859a2\") " Oct 31 05:11:05.969719 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:05.969522 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f48d27d4-dc90-4bf7-aa79-2b14c97859a2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f48d27d4-dc90-4bf7-aa79-2b14c97859a2-plugins-config-volume\") pod \"f48d27d4-dc90-4bf7-aa79-2b14c97859a2\" (UID: \"f48d27d4-dc90-4bf7-aa79-2b14c97859a2\") " Oct 31 05:11:05.969975 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:05.969948 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f48d27d4-dc90-4bf7-aa79-2b14c97859a2-plugins-config-volume" pod "f48d27d4-dc90-4bf7-aa79-2b14c97859a2" (UID: "f48d27d4-dc90-4bf7-aa79-2b14c97859a2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:11:05.970242 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:05.970223 4376 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f48d27d4-dc90-4bf7-aa79-2b14c97859a2-plugins-config-volume\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:11:05.972318 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:05.972284 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f48d27d4-dc90-4bf7-aa79-2b14c97859a2-kube-api-access-rb87b" pod "f48d27d4-dc90-4bf7-aa79-2b14c97859a2" (UID: "f48d27d4-dc90-4bf7-aa79-2b14c97859a2"). InnerVolumeSpecName "kube-api-access-rb87b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:11:06.071437 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:06.071401 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rb87b\" (UniqueName: \"kubernetes.io/projected/f48d27d4-dc90-4bf7-aa79-2b14c97859a2-kube-api-access-rb87b\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:11:06.551689 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:06.551652 4376 generic.go:356] "Generic (PLEG): container finished" podID="f48d27d4-dc90-4bf7-aa79-2b14c97859a2" containerID="0c6c5b9c319ec0aeb984e2f930be79e000795979f7e74f1a42dafcc2a8c8a1f5" exitCode=0 Oct 31 05:11:06.551896 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:06.551719 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-47c409-8649bf6df7-rbdch" Oct 31 05:11:06.551896 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:06.551731 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-47c409-8649bf6df7-rbdch" event={"ID":"f48d27d4-dc90-4bf7-aa79-2b14c97859a2","Type":"ContainerDied","Data":"0c6c5b9c319ec0aeb984e2f930be79e000795979f7e74f1a42dafcc2a8c8a1f5"} Oct 31 05:11:06.551896 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:06.551811 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-47c409-8649bf6df7-rbdch" event={"ID":"f48d27d4-dc90-4bf7-aa79-2b14c97859a2","Type":"ContainerDied","Data":"e4c3c1a1e804a9e62972838a33072da41f6d8b840fd064abfb71caded92048fa"} Oct 31 05:11:06.551896 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:06.551833 4376 scope.go:122] "RemoveContainer" containerID="0c6c5b9c319ec0aeb984e2f930be79e000795979f7e74f1a42dafcc2a8c8a1f5" Oct 31 05:11:06.562967 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:06.562949 4376 scope.go:122] "RemoveContainer" containerID="0c6c5b9c319ec0aeb984e2f930be79e000795979f7e74f1a42dafcc2a8c8a1f5" Oct 31 05:11:06.563236 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:11:06.563212 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c6c5b9c319ec0aeb984e2f930be79e000795979f7e74f1a42dafcc2a8c8a1f5\": container with ID starting with 0c6c5b9c319ec0aeb984e2f930be79e000795979f7e74f1a42dafcc2a8c8a1f5 not found: ID does not exist" containerID="0c6c5b9c319ec0aeb984e2f930be79e000795979f7e74f1a42dafcc2a8c8a1f5" Oct 31 05:11:06.563282 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:06.563245 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c6c5b9c319ec0aeb984e2f930be79e000795979f7e74f1a42dafcc2a8c8a1f5"} err="failed to get container status \"0c6c5b9c319ec0aeb984e2f930be79e000795979f7e74f1a42dafcc2a8c8a1f5\": rpc error: code = NotFound desc = could not find container \"0c6c5b9c319ec0aeb984e2f930be79e000795979f7e74f1a42dafcc2a8c8a1f5\": container with ID starting with 0c6c5b9c319ec0aeb984e2f930be79e000795979f7e74f1a42dafcc2a8c8a1f5 not found: ID does not exist" Oct 31 05:11:06.612321 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:06.612288 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-47c409-8649bf6df7-rbdch"] Oct 31 05:11:06.615821 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:06.615782 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-47c409-8649bf6df7-rbdch"] Oct 31 05:11:08.444594 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:08.444563 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f48d27d4-dc90-4bf7-aa79-2b14c97859a2" path="/var/lib/kubelet/pods/f48d27d4-dc90-4bf7-aa79-2b14c97859a2/volumes" Oct 31 05:11:11.000935 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:11:11.000898 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 05:11:12.656066 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:12.656025 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-401761-6ddf8dc4bf-fbwrw" podUID="0551b77c-f034-48e4-b643-3361b903df31" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 05:11:12.656654 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:12.656510 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-401761-f8dbb88dc-9ks76" podUID="8ab573d4-4b87-40e0-9d16-ff7fdd3c3963" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 05:11:19.043999 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:19.043961 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-401761-c5bb964f-h6p9q"] Oct 31 05:11:19.044524 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:19.044208 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-401761-c5bb964f-h6p9q" podUID="c0b21674-516a-452f-86c5-b27115eb72e3" containerName="maas-api" containerID="cri-o://67d6bd11e8fff289fab43efd61f5680cc67fb583a4b54de8a43f5269581e8079" gracePeriod=30 Oct 31 05:11:19.387656 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:19.387618 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-401761-c5bb964f-h6p9q" Oct 31 05:11:19.552120 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:19.552088 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0b21674-516a-452f-86c5-b27115eb72e3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0b21674-516a-452f-86c5-b27115eb72e3-maas-api-tls\") pod \"c0b21674-516a-452f-86c5-b27115eb72e3\" (UID: \"c0b21674-516a-452f-86c5-b27115eb72e3\") " Oct 31 05:11:19.552313 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:19.552130 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0b21674-516a-452f-86c5-b27115eb72e3-kube-api-access-bsg2n\" (UniqueName: \"kubernetes.io/projected/c0b21674-516a-452f-86c5-b27115eb72e3-kube-api-access-bsg2n\") pod \"c0b21674-516a-452f-86c5-b27115eb72e3\" (UID: \"c0b21674-516a-452f-86c5-b27115eb72e3\") " Oct 31 05:11:19.552405 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:19.552390 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0b21674-516a-452f-86c5-b27115eb72e3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0b21674-516a-452f-86c5-b27115eb72e3-openshift-service-ca\") pod \"c0b21674-516a-452f-86c5-b27115eb72e3\" (UID: \"c0b21674-516a-452f-86c5-b27115eb72e3\") " Oct 31 05:11:19.552532 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:19.552513 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0b21674-516a-452f-86c5-b27115eb72e3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0b21674-516a-452f-86c5-b27115eb72e3-metrics-certs\") pod \"c0b21674-516a-452f-86c5-b27115eb72e3\" (UID: \"c0b21674-516a-452f-86c5-b27115eb72e3\") " Oct 31 05:11:19.552830 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:19.552803 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0b21674-516a-452f-86c5-b27115eb72e3-openshift-service-ca" pod "c0b21674-516a-452f-86c5-b27115eb72e3" (UID: "c0b21674-516a-452f-86c5-b27115eb72e3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:11:19.553471 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:19.553440 4376 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0b21674-516a-452f-86c5-b27115eb72e3-openshift-service-ca\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:11:19.554467 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:19.554442 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0b21674-516a-452f-86c5-b27115eb72e3-maas-api-tls" pod "c0b21674-516a-452f-86c5-b27115eb72e3" (UID: "c0b21674-516a-452f-86c5-b27115eb72e3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 05:11:19.554746 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:19.554723 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0b21674-516a-452f-86c5-b27115eb72e3-kube-api-access-bsg2n" pod "c0b21674-516a-452f-86c5-b27115eb72e3" (UID: "c0b21674-516a-452f-86c5-b27115eb72e3"). InnerVolumeSpecName "kube-api-access-bsg2n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:11:19.554854 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:19.554762 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0b21674-516a-452f-86c5-b27115eb72e3-metrics-certs" pod "c0b21674-516a-452f-86c5-b27115eb72e3" (UID: "c0b21674-516a-452f-86c5-b27115eb72e3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 05:11:19.661351 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:19.658657 4376 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0b21674-516a-452f-86c5-b27115eb72e3-metrics-certs\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:11:19.661351 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:19.658696 4376 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0b21674-516a-452f-86c5-b27115eb72e3-maas-api-tls\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:11:19.661351 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:19.658711 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bsg2n\" (UniqueName: \"kubernetes.io/projected/c0b21674-516a-452f-86c5-b27115eb72e3-kube-api-access-bsg2n\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:11:19.701351 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:19.700456 4376 generic.go:356] "Generic (PLEG): container finished" podID="c0b21674-516a-452f-86c5-b27115eb72e3" containerID="67d6bd11e8fff289fab43efd61f5680cc67fb583a4b54de8a43f5269581e8079" exitCode=0 Oct 31 05:11:19.701351 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:19.700692 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-401761-c5bb964f-h6p9q" event={"ID":"c0b21674-516a-452f-86c5-b27115eb72e3","Type":"ContainerDied","Data":"67d6bd11e8fff289fab43efd61f5680cc67fb583a4b54de8a43f5269581e8079"} Oct 31 05:11:19.701351 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:19.700744 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-401761-c5bb964f-h6p9q" event={"ID":"c0b21674-516a-452f-86c5-b27115eb72e3","Type":"ContainerDied","Data":"0386e8e25e5f08051eb536c461f392578b4ab96c2d20a0781708cfebf6da2e7e"} Oct 31 05:11:19.701351 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:19.700780 4376 scope.go:122] "RemoveContainer" containerID="67d6bd11e8fff289fab43efd61f5680cc67fb583a4b54de8a43f5269581e8079" Oct 31 05:11:19.701351 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:19.700936 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-401761-c5bb964f-h6p9q" Oct 31 05:11:19.759112 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:19.756777 4376 scope.go:122] "RemoveContainer" containerID="67d6bd11e8fff289fab43efd61f5680cc67fb583a4b54de8a43f5269581e8079" Oct 31 05:11:19.759112 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:11:19.757246 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67d6bd11e8fff289fab43efd61f5680cc67fb583a4b54de8a43f5269581e8079\": container with ID starting with 67d6bd11e8fff289fab43efd61f5680cc67fb583a4b54de8a43f5269581e8079 not found: ID does not exist" containerID="67d6bd11e8fff289fab43efd61f5680cc67fb583a4b54de8a43f5269581e8079" Oct 31 05:11:19.759112 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:19.757288 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67d6bd11e8fff289fab43efd61f5680cc67fb583a4b54de8a43f5269581e8079"} err="failed to get container status \"67d6bd11e8fff289fab43efd61f5680cc67fb583a4b54de8a43f5269581e8079\": rpc error: code = NotFound desc = could not find container \"67d6bd11e8fff289fab43efd61f5680cc67fb583a4b54de8a43f5269581e8079\": container with ID starting with 67d6bd11e8fff289fab43efd61f5680cc67fb583a4b54de8a43f5269581e8079 not found: ID does not exist" Oct 31 05:11:19.781273 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:19.781234 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-401761-c5bb964f-h6p9q"] Oct 31 05:11:19.795249 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:19.795218 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-401761-c5bb964f-h6p9q"] Oct 31 05:11:20.445239 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:20.445201 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0b21674-516a-452f-86c5-b27115eb72e3" path="/var/lib/kubelet/pods/c0b21674-516a-452f-86c5-b27115eb72e3/volumes" Oct 31 05:11:22.656655 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:22.656622 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-401761-6ddf8dc4bf-fbwrw" podUID="0551b77c-f034-48e4-b643-3361b903df31" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 05:11:22.656655 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:22.656640 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-401761-f8dbb88dc-9ks76" podUID="8ab573d4-4b87-40e0-9d16-ff7fdd3c3963" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 05:11:24.152751 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:24.152721 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-401761-6ddf8dc4bf-fbwrw" Oct 31 05:11:24.203299 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:24.203272 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-401761-f8dbb88dc-9ks76" Oct 31 05:11:24.240201 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:24.240157 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0551b77c-f034-48e4-b643-3361b903df31-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0551b77c-f034-48e4-b643-3361b903df31-plugins-config-volume\") pod \"0551b77c-f034-48e4-b643-3361b903df31\" (UID: \"0551b77c-f034-48e4-b643-3361b903df31\") " Oct 31 05:11:24.240440 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:24.240278 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ab573d4-4b87-40e0-9d16-ff7fdd3c3963-kube-api-access-rx2k7\" (UniqueName: \"kubernetes.io/projected/8ab573d4-4b87-40e0-9d16-ff7fdd3c3963-kube-api-access-rx2k7\") pod \"8ab573d4-4b87-40e0-9d16-ff7fdd3c3963\" (UID: \"8ab573d4-4b87-40e0-9d16-ff7fdd3c3963\") " Oct 31 05:11:24.240440 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:24.240338 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ab573d4-4b87-40e0-9d16-ff7fdd3c3963-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ab573d4-4b87-40e0-9d16-ff7fdd3c3963-plugins-config-volume\") pod \"8ab573d4-4b87-40e0-9d16-ff7fdd3c3963\" (UID: \"8ab573d4-4b87-40e0-9d16-ff7fdd3c3963\") " Oct 31 05:11:24.240576 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:24.240494 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0551b77c-f034-48e4-b643-3361b903df31-kube-api-access-f9kkt\" (UniqueName: \"kubernetes.io/projected/0551b77c-f034-48e4-b643-3361b903df31-kube-api-access-f9kkt\") pod \"0551b77c-f034-48e4-b643-3361b903df31\" (UID: \"0551b77c-f034-48e4-b643-3361b903df31\") " Oct 31 05:11:24.240852 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:24.240732 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0551b77c-f034-48e4-b643-3361b903df31-plugins-config-volume" pod "0551b77c-f034-48e4-b643-3361b903df31" (UID: "0551b77c-f034-48e4-b643-3361b903df31"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:11:24.240852 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:24.240815 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ab573d4-4b87-40e0-9d16-ff7fdd3c3963-plugins-config-volume" pod "8ab573d4-4b87-40e0-9d16-ff7fdd3c3963" (UID: "8ab573d4-4b87-40e0-9d16-ff7fdd3c3963"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:11:24.241212 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:24.241181 4376 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ab573d4-4b87-40e0-9d16-ff7fdd3c3963-plugins-config-volume\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:11:24.241212 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:24.241209 4376 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0551b77c-f034-48e4-b643-3361b903df31-plugins-config-volume\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:11:24.242853 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:24.242822 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0551b77c-f034-48e4-b643-3361b903df31-kube-api-access-f9kkt" pod "0551b77c-f034-48e4-b643-3361b903df31" (UID: "0551b77c-f034-48e4-b643-3361b903df31"). InnerVolumeSpecName "kube-api-access-f9kkt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:11:24.243122 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:24.243101 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ab573d4-4b87-40e0-9d16-ff7fdd3c3963-kube-api-access-rx2k7" pod "8ab573d4-4b87-40e0-9d16-ff7fdd3c3963" (UID: "8ab573d4-4b87-40e0-9d16-ff7fdd3c3963"). InnerVolumeSpecName "kube-api-access-rx2k7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:11:24.342943 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:24.342840 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rx2k7\" (UniqueName: \"kubernetes.io/projected/8ab573d4-4b87-40e0-9d16-ff7fdd3c3963-kube-api-access-rx2k7\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:11:24.342943 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:24.342883 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9kkt\" (UniqueName: \"kubernetes.io/projected/0551b77c-f034-48e4-b643-3361b903df31-kube-api-access-f9kkt\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:11:24.740535 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:24.740451 4376 generic.go:356] "Generic (PLEG): container finished" podID="8ab573d4-4b87-40e0-9d16-ff7fdd3c3963" containerID="9ce8304ec062a9b80afc6829cb5375563596b4665dc66b7a68a7f4f27fc85192" exitCode=0 Oct 31 05:11:24.740535 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:24.740514 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-401761-f8dbb88dc-9ks76" Oct 31 05:11:24.740800 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:24.740537 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-401761-f8dbb88dc-9ks76" event={"ID":"8ab573d4-4b87-40e0-9d16-ff7fdd3c3963","Type":"ContainerDied","Data":"9ce8304ec062a9b80afc6829cb5375563596b4665dc66b7a68a7f4f27fc85192"} Oct 31 05:11:24.740800 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:24.740593 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-401761-f8dbb88dc-9ks76" event={"ID":"8ab573d4-4b87-40e0-9d16-ff7fdd3c3963","Type":"ContainerDied","Data":"6bcf65fcf5fc1220b7191798e72f69ec092f87335b59a3a0fbc20ad042d538f4"} Oct 31 05:11:24.740800 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:24.740619 4376 scope.go:122] "RemoveContainer" containerID="9ce8304ec062a9b80afc6829cb5375563596b4665dc66b7a68a7f4f27fc85192" Oct 31 05:11:24.742370 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:24.742341 4376 generic.go:356] "Generic (PLEG): container finished" podID="0551b77c-f034-48e4-b643-3361b903df31" containerID="070a0fdc33137ca68093bd7a4a18a9dc4e197d6cf0a9546d9f26c217536715c9" exitCode=0 Oct 31 05:11:24.742517 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:24.742404 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-401761-6ddf8dc4bf-fbwrw" Oct 31 05:11:24.742517 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:24.742425 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-401761-6ddf8dc4bf-fbwrw" event={"ID":"0551b77c-f034-48e4-b643-3361b903df31","Type":"ContainerDied","Data":"070a0fdc33137ca68093bd7a4a18a9dc4e197d6cf0a9546d9f26c217536715c9"} Oct 31 05:11:24.742517 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:24.742470 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-401761-6ddf8dc4bf-fbwrw" event={"ID":"0551b77c-f034-48e4-b643-3361b903df31","Type":"ContainerDied","Data":"f9fd237b0531d196a75cf07efa66ca19ce6ebbb7b3a6c5cc713806c47e9a47d0"} Oct 31 05:11:24.752196 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:24.752179 4376 scope.go:122] "RemoveContainer" containerID="9ce8304ec062a9b80afc6829cb5375563596b4665dc66b7a68a7f4f27fc85192" Oct 31 05:11:24.752498 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:11:24.752477 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ce8304ec062a9b80afc6829cb5375563596b4665dc66b7a68a7f4f27fc85192\": container with ID starting with 9ce8304ec062a9b80afc6829cb5375563596b4665dc66b7a68a7f4f27fc85192 not found: ID does not exist" containerID="9ce8304ec062a9b80afc6829cb5375563596b4665dc66b7a68a7f4f27fc85192" Oct 31 05:11:24.752578 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:24.752513 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ce8304ec062a9b80afc6829cb5375563596b4665dc66b7a68a7f4f27fc85192"} err="failed to get container status \"9ce8304ec062a9b80afc6829cb5375563596b4665dc66b7a68a7f4f27fc85192\": rpc error: code = NotFound desc = could not find container \"9ce8304ec062a9b80afc6829cb5375563596b4665dc66b7a68a7f4f27fc85192\": container with ID starting with 9ce8304ec062a9b80afc6829cb5375563596b4665dc66b7a68a7f4f27fc85192 not found: ID does not exist" Oct 31 05:11:24.752578 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:24.752532 4376 scope.go:122] "RemoveContainer" containerID="070a0fdc33137ca68093bd7a4a18a9dc4e197d6cf0a9546d9f26c217536715c9" Oct 31 05:11:24.762654 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:24.762636 4376 scope.go:122] "RemoveContainer" containerID="070a0fdc33137ca68093bd7a4a18a9dc4e197d6cf0a9546d9f26c217536715c9" Oct 31 05:11:24.762949 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:11:24.762930 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"070a0fdc33137ca68093bd7a4a18a9dc4e197d6cf0a9546d9f26c217536715c9\": container with ID starting with 070a0fdc33137ca68093bd7a4a18a9dc4e197d6cf0a9546d9f26c217536715c9 not found: ID does not exist" containerID="070a0fdc33137ca68093bd7a4a18a9dc4e197d6cf0a9546d9f26c217536715c9" Oct 31 05:11:24.763021 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:24.762955 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"070a0fdc33137ca68093bd7a4a18a9dc4e197d6cf0a9546d9f26c217536715c9"} err="failed to get container status \"070a0fdc33137ca68093bd7a4a18a9dc4e197d6cf0a9546d9f26c217536715c9\": rpc error: code = NotFound desc = could not find container \"070a0fdc33137ca68093bd7a4a18a9dc4e197d6cf0a9546d9f26c217536715c9\": container with ID starting with 070a0fdc33137ca68093bd7a4a18a9dc4e197d6cf0a9546d9f26c217536715c9 not found: ID does not exist" Oct 31 05:11:24.799890 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:24.799862 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-401761-f8dbb88dc-9ks76"] Oct 31 05:11:24.813851 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:24.813820 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-401761-f8dbb88dc-9ks76"] Oct 31 05:11:24.840205 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:24.840172 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-401761-6ddf8dc4bf-fbwrw"] Oct 31 05:11:24.846780 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:24.846749 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-401761-6ddf8dc4bf-fbwrw"] Oct 31 05:11:25.384469 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:25.380389 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-47c409-openshift-default-796cbb7889-drd46"] Oct 31 05:11:25.384469 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:25.380613 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-47c409-openshift-default-796cbb7889-drd46" podUID="d76a0ace-f020-430f-a26c-005231cb8be2" containerName="istio-proxy" containerID="cri-o://1c8feda615d68ba3a4acdd1e22634785025a378fe79bec24ea294b989852fadd" gracePeriod=30 Oct 31 05:11:26.444520 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:26.444487 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0551b77c-f034-48e4-b643-3361b903df31" path="/var/lib/kubelet/pods/0551b77c-f034-48e4-b643-3361b903df31/volumes" Oct 31 05:11:26.444996 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:26.444979 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ab573d4-4b87-40e0-9d16-ff7fdd3c3963" path="/var/lib/kubelet/pods/8ab573d4-4b87-40e0-9d16-ff7fdd3c3963/volumes" Oct 31 05:11:30.039312 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:30.039276 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-567ca6-86cc9d6977-pk7jd"] Oct 31 05:11:30.795198 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:30.795159 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-47c409-openshift-default-796cbb7889-drd46" Oct 31 05:11:30.805377 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:30.805348 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-47c409-openshift-default-796cbb7889-drd46" Oct 31 05:11:30.805572 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:30.805376 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-47c409-openshift-default-796cbb7889-drd46" event={"ID":"d76a0ace-f020-430f-a26c-005231cb8be2","Type":"ContainerDied","Data":"1c8feda615d68ba3a4acdd1e22634785025a378fe79bec24ea294b989852fadd"} Oct 31 05:11:30.805572 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:30.805279 4376 generic.go:356] "Generic (PLEG): container finished" podID="d76a0ace-f020-430f-a26c-005231cb8be2" containerID="1c8feda615d68ba3a4acdd1e22634785025a378fe79bec24ea294b989852fadd" exitCode=0 Oct 31 05:11:30.805572 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:30.805466 4376 scope.go:122] "RemoveContainer" containerID="1c8feda615d68ba3a4acdd1e22634785025a378fe79bec24ea294b989852fadd" Oct 31 05:11:30.805713 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:30.805568 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-47c409-openshift-default-796cbb7889-drd46" event={"ID":"d76a0ace-f020-430f-a26c-005231cb8be2","Type":"ContainerDied","Data":"fb19dbfbb11e68dd6f922602ead489bc79b40ff6540ee8cdeae5914558d4307a"} Oct 31 05:11:30.820687 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:30.820663 4376 scope.go:122] "RemoveContainer" containerID="1c8feda615d68ba3a4acdd1e22634785025a378fe79bec24ea294b989852fadd" Oct 31 05:11:30.821161 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:11:30.821130 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c8feda615d68ba3a4acdd1e22634785025a378fe79bec24ea294b989852fadd\": container with ID starting with 1c8feda615d68ba3a4acdd1e22634785025a378fe79bec24ea294b989852fadd not found: ID does not exist" containerID="1c8feda615d68ba3a4acdd1e22634785025a378fe79bec24ea294b989852fadd" Oct 31 05:11:30.821278 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:30.821170 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c8feda615d68ba3a4acdd1e22634785025a378fe79bec24ea294b989852fadd"} err="failed to get container status \"1c8feda615d68ba3a4acdd1e22634785025a378fe79bec24ea294b989852fadd\": rpc error: code = NotFound desc = could not find container \"1c8feda615d68ba3a4acdd1e22634785025a378fe79bec24ea294b989852fadd\": container with ID starting with 1c8feda615d68ba3a4acdd1e22634785025a378fe79bec24ea294b989852fadd not found: ID does not exist" Oct 31 05:11:30.860245 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:30.860191 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d76a0ace-f020-430f-a26c-005231cb8be2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d76a0ace-f020-430f-a26c-005231cb8be2-workload-socket\") pod \"d76a0ace-f020-430f-a26c-005231cb8be2\" (UID: \"d76a0ace-f020-430f-a26c-005231cb8be2\") " Oct 31 05:11:30.860245 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:30.860245 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d76a0ace-f020-430f-a26c-005231cb8be2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d76a0ace-f020-430f-a26c-005231cb8be2-istio-data\") pod \"d76a0ace-f020-430f-a26c-005231cb8be2\" (UID: \"d76a0ace-f020-430f-a26c-005231cb8be2\") " Oct 31 05:11:30.860508 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:30.860343 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d76a0ace-f020-430f-a26c-005231cb8be2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d76a0ace-f020-430f-a26c-005231cb8be2-credential-socket\") pod \"d76a0ace-f020-430f-a26c-005231cb8be2\" (UID: \"d76a0ace-f020-430f-a26c-005231cb8be2\") " Oct 31 05:11:30.860508 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:30.860496 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d76a0ace-f020-430f-a26c-005231cb8be2-kube-api-access-k6qv7\" (UniqueName: \"kubernetes.io/projected/d76a0ace-f020-430f-a26c-005231cb8be2-kube-api-access-k6qv7\") pod \"d76a0ace-f020-430f-a26c-005231cb8be2\" (UID: \"d76a0ace-f020-430f-a26c-005231cb8be2\") " Oct 31 05:11:30.860631 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:30.860557 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d76a0ace-f020-430f-a26c-005231cb8be2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d76a0ace-f020-430f-a26c-005231cb8be2-istio-envoy\") pod \"d76a0ace-f020-430f-a26c-005231cb8be2\" (UID: \"d76a0ace-f020-430f-a26c-005231cb8be2\") " Oct 31 05:11:30.860688 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:30.860654 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d76a0ace-f020-430f-a26c-005231cb8be2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d76a0ace-f020-430f-a26c-005231cb8be2-istiod-ca-cert\") pod \"d76a0ace-f020-430f-a26c-005231cb8be2\" (UID: \"d76a0ace-f020-430f-a26c-005231cb8be2\") " Oct 31 05:11:30.860744 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:30.860725 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d76a0ace-f020-430f-a26c-005231cb8be2-istio-token\" (UniqueName: \"kubernetes.io/projected/d76a0ace-f020-430f-a26c-005231cb8be2-istio-token\") pod \"d76a0ace-f020-430f-a26c-005231cb8be2\" (UID: \"d76a0ace-f020-430f-a26c-005231cb8be2\") " Oct 31 05:11:30.860811 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:30.860774 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d76a0ace-f020-430f-a26c-005231cb8be2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d76a0ace-f020-430f-a26c-005231cb8be2-istio-podinfo\") pod \"d76a0ace-f020-430f-a26c-005231cb8be2\" (UID: \"d76a0ace-f020-430f-a26c-005231cb8be2\") " Oct 31 05:11:30.860879 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:30.860863 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d76a0ace-f020-430f-a26c-005231cb8be2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d76a0ace-f020-430f-a26c-005231cb8be2-workload-certs\") pod \"d76a0ace-f020-430f-a26c-005231cb8be2\" (UID: \"d76a0ace-f020-430f-a26c-005231cb8be2\") " Oct 31 05:11:30.861677 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:30.861652 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d76a0ace-f020-430f-a26c-005231cb8be2-workload-certs" pod "d76a0ace-f020-430f-a26c-005231cb8be2" (UID: "d76a0ace-f020-430f-a26c-005231cb8be2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:11:30.862139 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:30.862115 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d76a0ace-f020-430f-a26c-005231cb8be2-credential-socket" pod "d76a0ace-f020-430f-a26c-005231cb8be2" (UID: "d76a0ace-f020-430f-a26c-005231cb8be2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:11:30.862378 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:30.862135 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d76a0ace-f020-430f-a26c-005231cb8be2-istio-data" pod "d76a0ace-f020-430f-a26c-005231cb8be2" (UID: "d76a0ace-f020-430f-a26c-005231cb8be2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:11:30.862529 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:30.862301 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d76a0ace-f020-430f-a26c-005231cb8be2-workload-socket" pod "d76a0ace-f020-430f-a26c-005231cb8be2" (UID: "d76a0ace-f020-430f-a26c-005231cb8be2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:11:30.862657 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:30.862632 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d76a0ace-f020-430f-a26c-005231cb8be2-istiod-ca-cert" pod "d76a0ace-f020-430f-a26c-005231cb8be2" (UID: "d76a0ace-f020-430f-a26c-005231cb8be2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:11:30.866693 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:30.866656 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d76a0ace-f020-430f-a26c-005231cb8be2-istio-podinfo" pod "d76a0ace-f020-430f-a26c-005231cb8be2" (UID: "d76a0ace-f020-430f-a26c-005231cb8be2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 05:11:30.866843 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:30.866822 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d76a0ace-f020-430f-a26c-005231cb8be2-istio-token" pod "d76a0ace-f020-430f-a26c-005231cb8be2" (UID: "d76a0ace-f020-430f-a26c-005231cb8be2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:11:30.866918 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:30.866907 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d76a0ace-f020-430f-a26c-005231cb8be2-istio-envoy" pod "d76a0ace-f020-430f-a26c-005231cb8be2" (UID: "d76a0ace-f020-430f-a26c-005231cb8be2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:11:30.869133 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:30.869104 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d76a0ace-f020-430f-a26c-005231cb8be2-kube-api-access-k6qv7" pod "d76a0ace-f020-430f-a26c-005231cb8be2" (UID: "d76a0ace-f020-430f-a26c-005231cb8be2"). InnerVolumeSpecName "kube-api-access-k6qv7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:11:30.962799 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:30.962763 4376 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d76a0ace-f020-430f-a26c-005231cb8be2-workload-socket\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:11:30.962799 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:30.962793 4376 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d76a0ace-f020-430f-a26c-005231cb8be2-istio-data\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:11:30.962799 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:30.962802 4376 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d76a0ace-f020-430f-a26c-005231cb8be2-credential-socket\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:11:30.963043 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:30.962811 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6qv7\" (UniqueName: \"kubernetes.io/projected/d76a0ace-f020-430f-a26c-005231cb8be2-kube-api-access-k6qv7\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:11:30.963043 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:30.962820 4376 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d76a0ace-f020-430f-a26c-005231cb8be2-istio-envoy\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:11:30.963043 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:30.962828 4376 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d76a0ace-f020-430f-a26c-005231cb8be2-istiod-ca-cert\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:11:30.963043 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:30.962835 4376 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d76a0ace-f020-430f-a26c-005231cb8be2-istio-token\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:11:30.963043 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:30.962843 4376 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d76a0ace-f020-430f-a26c-005231cb8be2-istio-podinfo\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:11:30.963043 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:30.962850 4376 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d76a0ace-f020-430f-a26c-005231cb8be2-workload-certs\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:11:31.158101 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:31.158065 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-47c409-openshift-default-796cbb7889-drd46"] Oct 31 05:11:31.172752 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:31.172724 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-47c409-openshift-default-796cbb7889-drd46"] Oct 31 05:11:31.505166 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:31.505127 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-567ca6-59b4b45645-98ghw"] Oct 31 05:11:31.505376 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:31.505340 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-567ca6-59b4b45645-98ghw" podUID="649b78d1-3eb4-4780-8462-f6c6a5cf7c6c" containerName="maas-api" containerID="cri-o://cbd13093204d28e5a920b81f077ede0be2004eb54d991427492f1b562d91b89b" gracePeriod=30 Oct 31 05:11:31.833354 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:31.831076 4376 generic.go:356] "Generic (PLEG): container finished" podID="649b78d1-3eb4-4780-8462-f6c6a5cf7c6c" containerID="cbd13093204d28e5a920b81f077ede0be2004eb54d991427492f1b562d91b89b" exitCode=0 Oct 31 05:11:31.833354 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:31.831171 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-567ca6-59b4b45645-98ghw" event={"ID":"649b78d1-3eb4-4780-8462-f6c6a5cf7c6c","Type":"ContainerDied","Data":"cbd13093204d28e5a920b81f077ede0be2004eb54d991427492f1b562d91b89b"} Oct 31 05:11:31.833354 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:31.831222 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-567ca6-59b4b45645-98ghw" event={"ID":"649b78d1-3eb4-4780-8462-f6c6a5cf7c6c","Type":"ContainerDied","Data":"7342d9b2b8ebf43fec2d23c49f0aece37e86bcbfbc610b1ee148b805efda80a0"} Oct 31 05:11:31.833354 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:31.831248 4376 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7342d9b2b8ebf43fec2d23c49f0aece37e86bcbfbc610b1ee148b805efda80a0" Oct 31 05:11:31.871351 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:31.865096 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-567ca6-59b4b45645-98ghw" Oct 31 05:11:32.001968 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:32.001933 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/649b78d1-3eb4-4780-8462-f6c6a5cf7c6c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/649b78d1-3eb4-4780-8462-f6c6a5cf7c6c-maas-api-tls\") pod \"649b78d1-3eb4-4780-8462-f6c6a5cf7c6c\" (UID: \"649b78d1-3eb4-4780-8462-f6c6a5cf7c6c\") " Oct 31 05:11:32.002274 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:32.002257 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/649b78d1-3eb4-4780-8462-f6c6a5cf7c6c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/649b78d1-3eb4-4780-8462-f6c6a5cf7c6c-metrics-certs\") pod \"649b78d1-3eb4-4780-8462-f6c6a5cf7c6c\" (UID: \"649b78d1-3eb4-4780-8462-f6c6a5cf7c6c\") " Oct 31 05:11:32.002369 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:32.002352 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/649b78d1-3eb4-4780-8462-f6c6a5cf7c6c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/649b78d1-3eb4-4780-8462-f6c6a5cf7c6c-openshift-service-ca\") pod \"649b78d1-3eb4-4780-8462-f6c6a5cf7c6c\" (UID: \"649b78d1-3eb4-4780-8462-f6c6a5cf7c6c\") " Oct 31 05:11:32.002429 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:32.002389 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/649b78d1-3eb4-4780-8462-f6c6a5cf7c6c-kube-api-access-hxsml\" (UniqueName: \"kubernetes.io/projected/649b78d1-3eb4-4780-8462-f6c6a5cf7c6c-kube-api-access-hxsml\") pod \"649b78d1-3eb4-4780-8462-f6c6a5cf7c6c\" (UID: \"649b78d1-3eb4-4780-8462-f6c6a5cf7c6c\") " Oct 31 05:11:32.002799 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:32.002764 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/649b78d1-3eb4-4780-8462-f6c6a5cf7c6c-openshift-service-ca" pod "649b78d1-3eb4-4780-8462-f6c6a5cf7c6c" (UID: "649b78d1-3eb4-4780-8462-f6c6a5cf7c6c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:11:32.003149 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:32.003128 4376 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/649b78d1-3eb4-4780-8462-f6c6a5cf7c6c-openshift-service-ca\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:11:32.004453 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:32.004418 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/649b78d1-3eb4-4780-8462-f6c6a5cf7c6c-maas-api-tls" pod "649b78d1-3eb4-4780-8462-f6c6a5cf7c6c" (UID: "649b78d1-3eb4-4780-8462-f6c6a5cf7c6c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 05:11:32.004570 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:32.004512 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/649b78d1-3eb4-4780-8462-f6c6a5cf7c6c-metrics-certs" pod "649b78d1-3eb4-4780-8462-f6c6a5cf7c6c" (UID: "649b78d1-3eb4-4780-8462-f6c6a5cf7c6c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 05:11:32.004877 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:32.004855 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/649b78d1-3eb4-4780-8462-f6c6a5cf7c6c-kube-api-access-hxsml" pod "649b78d1-3eb4-4780-8462-f6c6a5cf7c6c" (UID: "649b78d1-3eb4-4780-8462-f6c6a5cf7c6c"). InnerVolumeSpecName "kube-api-access-hxsml". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:11:32.117403 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:32.108366 4376 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/649b78d1-3eb4-4780-8462-f6c6a5cf7c6c-metrics-certs\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:11:32.117403 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:32.108404 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxsml\" (UniqueName: \"kubernetes.io/projected/649b78d1-3eb4-4780-8462-f6c6a5cf7c6c-kube-api-access-hxsml\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:11:32.117403 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:32.108420 4376 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/649b78d1-3eb4-4780-8462-f6c6a5cf7c6c-maas-api-tls\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:11:32.444988 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:32.444911 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d76a0ace-f020-430f-a26c-005231cb8be2" path="/var/lib/kubelet/pods/d76a0ace-f020-430f-a26c-005231cb8be2/volumes" Oct 31 05:11:32.841500 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:32.841413 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-567ca6-59b4b45645-98ghw" Oct 31 05:11:32.901243 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:32.901208 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-567ca6-59b4b45645-98ghw"] Oct 31 05:11:32.922372 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:32.918099 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-567ca6-59b4b45645-98ghw"] Oct 31 05:11:34.444373 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:34.444320 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="649b78d1-3eb4-4780-8462-f6c6a5cf7c6c" path="/var/lib/kubelet/pods/649b78d1-3eb4-4780-8462-f6c6a5cf7c6c/volumes" Oct 31 05:11:35.040632 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:35.040588 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-567ca6-86cc9d6977-pk7jd" podUID="681f54ec-b648-49cb-ae16-dc814e56bb71" containerName="payload-processing" containerID="cri-o://b77677df9903f6cb36d75a5ec150f5127979dd15bf231ea68a0f64de6ff9d994" gracePeriod=25 Oct 31 05:11:41.000534 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:11:41.000494 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 05:11:43.773041 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:43.773005 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 05:11:44.631649 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:44.631610 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-567ca6-86cc9d6977-pk7jd" podUID="681f54ec-b648-49cb-ae16-dc814e56bb71" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 05:11:44.821134 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:44.821098 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-c9707f-5d5859459c-tv5n9"] Oct 31 05:11:44.856715 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:44.856684 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c9707f-8598fffd47-fcvbn"] Oct 31 05:11:49.825367 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:49.823695 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c9707f-5d5859459c-tv5n9" podUID="fe905f47-822c-4bbb-b562-8bd04f2b35a1" containerName="payload-processing" containerID="cri-o://de1f895f9dd3e6d776886171b1813cd34b3987f58de5bbd7ffcfc00dc5bedc84" gracePeriod=25 Oct 31 05:11:49.862448 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:49.862398 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c9707f-8598fffd47-fcvbn" podUID="f31be489-e267-4087-9b94-86d7d622c841" containerName="payload-pre-processing" containerID="cri-o://245441bd30c6b7857276850e7acae00ac12f1bd56db09eb5521ead8cfc94b753" gracePeriod=25 Oct 31 05:11:54.631779 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:54.631741 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-567ca6-86cc9d6977-pk7jd" podUID="681f54ec-b648-49cb-ae16-dc814e56bb71" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 05:11:55.364076 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:55.364052 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-567ca6-86cc9d6977-pk7jd" Oct 31 05:11:55.425254 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:55.425224 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/681f54ec-b648-49cb-ae16-dc814e56bb71-kube-api-access-mjnn9\" (UniqueName: \"kubernetes.io/projected/681f54ec-b648-49cb-ae16-dc814e56bb71-kube-api-access-mjnn9\") pod \"681f54ec-b648-49cb-ae16-dc814e56bb71\" (UID: \"681f54ec-b648-49cb-ae16-dc814e56bb71\") " Oct 31 05:11:55.425439 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:55.425411 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/681f54ec-b648-49cb-ae16-dc814e56bb71-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/681f54ec-b648-49cb-ae16-dc814e56bb71-plugins-config-volume\") pod \"681f54ec-b648-49cb-ae16-dc814e56bb71\" (UID: \"681f54ec-b648-49cb-ae16-dc814e56bb71\") " Oct 31 05:11:55.425844 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:55.425814 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/681f54ec-b648-49cb-ae16-dc814e56bb71-plugins-config-volume" pod "681f54ec-b648-49cb-ae16-dc814e56bb71" (UID: "681f54ec-b648-49cb-ae16-dc814e56bb71"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:11:55.425992 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:55.425931 4376 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/681f54ec-b648-49cb-ae16-dc814e56bb71-plugins-config-volume\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:11:55.427743 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:55.427716 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/681f54ec-b648-49cb-ae16-dc814e56bb71-kube-api-access-mjnn9" pod "681f54ec-b648-49cb-ae16-dc814e56bb71" (UID: "681f54ec-b648-49cb-ae16-dc814e56bb71"). InnerVolumeSpecName "kube-api-access-mjnn9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:11:55.527622 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:55.527588 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mjnn9\" (UniqueName: \"kubernetes.io/projected/681f54ec-b648-49cb-ae16-dc814e56bb71-kube-api-access-mjnn9\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:11:56.073820 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:56.073788 4376 generic.go:356] "Generic (PLEG): container finished" podID="681f54ec-b648-49cb-ae16-dc814e56bb71" containerID="b77677df9903f6cb36d75a5ec150f5127979dd15bf231ea68a0f64de6ff9d994" exitCode=0 Oct 31 05:11:56.074298 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:56.073834 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-567ca6-86cc9d6977-pk7jd" Oct 31 05:11:56.074298 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:56.073872 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-567ca6-86cc9d6977-pk7jd" event={"ID":"681f54ec-b648-49cb-ae16-dc814e56bb71","Type":"ContainerDied","Data":"b77677df9903f6cb36d75a5ec150f5127979dd15bf231ea68a0f64de6ff9d994"} Oct 31 05:11:56.074298 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:56.073934 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-567ca6-86cc9d6977-pk7jd" event={"ID":"681f54ec-b648-49cb-ae16-dc814e56bb71","Type":"ContainerDied","Data":"66d9fbfbfa6ada854135eda468e9c36bf9a8560968d272b2cfe27710df2ad7a9"} Oct 31 05:11:56.074298 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:56.073968 4376 scope.go:122] "RemoveContainer" containerID="b77677df9903f6cb36d75a5ec150f5127979dd15bf231ea68a0f64de6ff9d994" Oct 31 05:11:56.085409 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:56.085355 4376 scope.go:122] "RemoveContainer" containerID="b77677df9903f6cb36d75a5ec150f5127979dd15bf231ea68a0f64de6ff9d994" Oct 31 05:11:56.085686 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:11:56.085666 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b77677df9903f6cb36d75a5ec150f5127979dd15bf231ea68a0f64de6ff9d994\": container with ID starting with b77677df9903f6cb36d75a5ec150f5127979dd15bf231ea68a0f64de6ff9d994 not found: ID does not exist" containerID="b77677df9903f6cb36d75a5ec150f5127979dd15bf231ea68a0f64de6ff9d994" Oct 31 05:11:56.085764 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:56.085697 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b77677df9903f6cb36d75a5ec150f5127979dd15bf231ea68a0f64de6ff9d994"} err="failed to get container status \"b77677df9903f6cb36d75a5ec150f5127979dd15bf231ea68a0f64de6ff9d994\": rpc error: code = NotFound desc = could not find container \"b77677df9903f6cb36d75a5ec150f5127979dd15bf231ea68a0f64de6ff9d994\": container with ID starting with b77677df9903f6cb36d75a5ec150f5127979dd15bf231ea68a0f64de6ff9d994 not found: ID does not exist" Oct 31 05:11:56.134323 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:56.134295 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-567ca6-86cc9d6977-pk7jd"] Oct 31 05:11:56.147008 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:56.146982 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-567ca6-86cc9d6977-pk7jd"] Oct 31 05:11:56.444976 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:56.444946 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="681f54ec-b648-49cb-ae16-dc814e56bb71" path="/var/lib/kubelet/pods/681f54ec-b648-49cb-ae16-dc814e56bb71/volumes" Oct 31 05:11:58.345273 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:58.345229 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c9707f-5d5859459c-tv5n9" podUID="fe905f47-822c-4bbb-b562-8bd04f2b35a1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 05:11:58.345740 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:11:58.345232 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c9707f-8598fffd47-fcvbn" podUID="f31be489-e267-4087-9b94-86d7d622c841" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 05:12:05.691099 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:12:05.691061 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c9707f-d9cc59bdd-xz2sb"] Oct 31 05:12:05.691653 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:12:05.691318 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c9707f-d9cc59bdd-xz2sb" podUID="65edcc7f-0600-4161-9ba1-89e26f064dd4" containerName="maas-api" containerID="cri-o://5d47d193f75c01dcae11a35926621b1a0b5ec6d6e740a33beb6610b14d5dff9f" gracePeriod=30 Oct 31 05:12:06.048512 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:12:06.046719 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c9707f-d9cc59bdd-xz2sb" Oct 31 05:12:06.146916 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:12:06.146881 4376 generic.go:356] "Generic (PLEG): container finished" podID="65edcc7f-0600-4161-9ba1-89e26f064dd4" containerID="5d47d193f75c01dcae11a35926621b1a0b5ec6d6e740a33beb6610b14d5dff9f" exitCode=0 Oct 31 05:12:06.147100 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:12:06.146939 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c9707f-d9cc59bdd-xz2sb" Oct 31 05:12:06.147100 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:12:06.146965 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c9707f-d9cc59bdd-xz2sb" event={"ID":"65edcc7f-0600-4161-9ba1-89e26f064dd4","Type":"ContainerDied","Data":"5d47d193f75c01dcae11a35926621b1a0b5ec6d6e740a33beb6610b14d5dff9f"} Oct 31 05:12:06.147100 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:12:06.147023 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c9707f-d9cc59bdd-xz2sb" event={"ID":"65edcc7f-0600-4161-9ba1-89e26f064dd4","Type":"ContainerDied","Data":"921ef8723867ec83929d8aa33f13c013e8f526bed60f1bda406e7ba575ea1711"} Oct 31 05:12:06.147100 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:12:06.147050 4376 scope.go:122] "RemoveContainer" containerID="5d47d193f75c01dcae11a35926621b1a0b5ec6d6e740a33beb6610b14d5dff9f" Oct 31 05:12:06.158666 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:12:06.158642 4376 scope.go:122] "RemoveContainer" containerID="5d47d193f75c01dcae11a35926621b1a0b5ec6d6e740a33beb6610b14d5dff9f" Oct 31 05:12:06.158989 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:12:06.158968 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d47d193f75c01dcae11a35926621b1a0b5ec6d6e740a33beb6610b14d5dff9f\": container with ID starting with 5d47d193f75c01dcae11a35926621b1a0b5ec6d6e740a33beb6610b14d5dff9f not found: ID does not exist" containerID="5d47d193f75c01dcae11a35926621b1a0b5ec6d6e740a33beb6610b14d5dff9f" Oct 31 05:12:06.159061 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:12:06.159002 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d47d193f75c01dcae11a35926621b1a0b5ec6d6e740a33beb6610b14d5dff9f"} err="failed to get container status \"5d47d193f75c01dcae11a35926621b1a0b5ec6d6e740a33beb6610b14d5dff9f\": rpc error: code = NotFound desc = could not find container \"5d47d193f75c01dcae11a35926621b1a0b5ec6d6e740a33beb6610b14d5dff9f\": container with ID starting with 5d47d193f75c01dcae11a35926621b1a0b5ec6d6e740a33beb6610b14d5dff9f not found: ID does not exist" Oct 31 05:12:06.193020 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:12:06.192989 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/65edcc7f-0600-4161-9ba1-89e26f064dd4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/65edcc7f-0600-4161-9ba1-89e26f064dd4-maas-api-tls\") pod \"65edcc7f-0600-4161-9ba1-89e26f064dd4\" (UID: \"65edcc7f-0600-4161-9ba1-89e26f064dd4\") " Oct 31 05:12:06.193185 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:12:06.193055 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65edcc7f-0600-4161-9ba1-89e26f064dd4-kube-api-access-jtzq6\" (UniqueName: \"kubernetes.io/projected/65edcc7f-0600-4161-9ba1-89e26f064dd4-kube-api-access-jtzq6\") pod \"65edcc7f-0600-4161-9ba1-89e26f064dd4\" (UID: \"65edcc7f-0600-4161-9ba1-89e26f064dd4\") " Oct 31 05:12:06.193185 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:12:06.193154 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/65edcc7f-0600-4161-9ba1-89e26f064dd4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/65edcc7f-0600-4161-9ba1-89e26f064dd4-metrics-certs\") pod \"65edcc7f-0600-4161-9ba1-89e26f064dd4\" (UID: \"65edcc7f-0600-4161-9ba1-89e26f064dd4\") " Oct 31 05:12:06.193308 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:12:06.193282 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65edcc7f-0600-4161-9ba1-89e26f064dd4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/65edcc7f-0600-4161-9ba1-89e26f064dd4-openshift-service-ca\") pod \"65edcc7f-0600-4161-9ba1-89e26f064dd4\" (UID: \"65edcc7f-0600-4161-9ba1-89e26f064dd4\") " Oct 31 05:12:06.193676 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:12:06.193621 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65edcc7f-0600-4161-9ba1-89e26f064dd4-openshift-service-ca" pod "65edcc7f-0600-4161-9ba1-89e26f064dd4" (UID: "65edcc7f-0600-4161-9ba1-89e26f064dd4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:12:06.193860 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:12:06.193837 4376 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/65edcc7f-0600-4161-9ba1-89e26f064dd4-openshift-service-ca\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:12:06.195189 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:12:06.195162 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65edcc7f-0600-4161-9ba1-89e26f064dd4-maas-api-tls" pod "65edcc7f-0600-4161-9ba1-89e26f064dd4" (UID: "65edcc7f-0600-4161-9ba1-89e26f064dd4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 05:12:06.195277 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:12:06.195186 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65edcc7f-0600-4161-9ba1-89e26f064dd4-metrics-certs" pod "65edcc7f-0600-4161-9ba1-89e26f064dd4" (UID: "65edcc7f-0600-4161-9ba1-89e26f064dd4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 05:12:06.195318 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:12:06.195270 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65edcc7f-0600-4161-9ba1-89e26f064dd4-kube-api-access-jtzq6" pod "65edcc7f-0600-4161-9ba1-89e26f064dd4" (UID: "65edcc7f-0600-4161-9ba1-89e26f064dd4"). InnerVolumeSpecName "kube-api-access-jtzq6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:12:06.297551 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:12:06.297460 4376 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/65edcc7f-0600-4161-9ba1-89e26f064dd4-maas-api-tls\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:12:06.297551 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:12:06.297498 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jtzq6\" (UniqueName: \"kubernetes.io/projected/65edcc7f-0600-4161-9ba1-89e26f064dd4-kube-api-access-jtzq6\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:12:06.297551 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:12:06.297514 4376 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/65edcc7f-0600-4161-9ba1-89e26f064dd4-metrics-certs\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:12:06.466532 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:12:06.466504 4376 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65edcc7f_0600_4161_9ba1_89e26f064dd4.slice\": RecentStats: unable to find data in memory cache]" Oct 31 05:12:06.466679 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:12:06.466508 4376 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65edcc7f_0600_4161_9ba1_89e26f064dd4.slice/crio-921ef8723867ec83929d8aa33f13c013e8f526bed60f1bda406e7ba575ea1711\": RecentStats: unable to find data in memory cache]" Oct 31 05:12:06.501688 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:12:06.501655 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c9707f-d9cc59bdd-xz2sb"] Oct 31 05:12:06.515682 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:12:06.515656 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c9707f-d9cc59bdd-xz2sb"] Oct 31 05:12:08.345678 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:12:08.345638 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c9707f-8598fffd47-fcvbn" podUID="f31be489-e267-4087-9b94-86d7d622c841" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 05:12:08.345678 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:12:08.345666 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c9707f-5d5859459c-tv5n9" podUID="fe905f47-822c-4bbb-b562-8bd04f2b35a1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 05:12:08.445277 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:12:08.445243 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65edcc7f-0600-4161-9ba1-89e26f064dd4" path="/var/lib/kubelet/pods/65edcc7f-0600-4161-9ba1-89e26f064dd4/volumes" Oct 31 05:12:10.150341 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:12:10.150303 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c9707f-5d5859459c-tv5n9" Oct 31 05:12:10.178847 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:12:10.178817 4376 generic.go:356] "Generic (PLEG): container finished" podID="f31be489-e267-4087-9b94-86d7d622c841" containerID="245441bd30c6b7857276850e7acae00ac12f1bd56db09eb5521ead8cfc94b753" exitCode=0 Oct 31 05:12:10.178981 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:12:10.178885 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c9707f-8598fffd47-fcvbn" event={"ID":"f31be489-e267-4087-9b94-86d7d622c841","Type":"ContainerDied","Data":"245441bd30c6b7857276850e7acae00ac12f1bd56db09eb5521ead8cfc94b753"} Oct 31 05:12:10.178981 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:12:10.178963 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c9707f-8598fffd47-fcvbn" event={"ID":"f31be489-e267-4087-9b94-86d7d622c841","Type":"ContainerDied","Data":"b1ef1d5aaf7e15bdd66ee3f73bb8c1ad3a6fd1f4ad79c4d16c34d98f949b13b3"} Oct 31 05:12:10.178981 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:12:10.178980 4376 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b1ef1d5aaf7e15bdd66ee3f73bb8c1ad3a6fd1f4ad79c4d16c34d98f949b13b3" Oct 31 05:12:10.180467 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:12:10.180443 4376 generic.go:356] "Generic (PLEG): container finished" podID="fe905f47-822c-4bbb-b562-8bd04f2b35a1" containerID="de1f895f9dd3e6d776886171b1813cd34b3987f58de5bbd7ffcfc00dc5bedc84" exitCode=0 Oct 31 05:12:10.180578 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:12:10.180497 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c9707f-5d5859459c-tv5n9" Oct 31 05:12:10.180578 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:12:10.180525 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c9707f-5d5859459c-tv5n9" event={"ID":"fe905f47-822c-4bbb-b562-8bd04f2b35a1","Type":"ContainerDied","Data":"de1f895f9dd3e6d776886171b1813cd34b3987f58de5bbd7ffcfc00dc5bedc84"} Oct 31 05:12:10.180578 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:12:10.180573 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c9707f-5d5859459c-tv5n9" event={"ID":"fe905f47-822c-4bbb-b562-8bd04f2b35a1","Type":"ContainerDied","Data":"ff28eab2535966dced62a2831668816b2517e7f5fd1a7f02f4f8f6e0e7e2f2eb"} Oct 31 05:12:10.180684 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:12:10.180603 4376 scope.go:122] "RemoveContainer" containerID="de1f895f9dd3e6d776886171b1813cd34b3987f58de5bbd7ffcfc00dc5bedc84" Oct 31 05:12:10.186696 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:12:10.186678 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c9707f-8598fffd47-fcvbn" Oct 31 05:12:10.190283 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:12:10.190266 4376 scope.go:122] "RemoveContainer" containerID="de1f895f9dd3e6d776886171b1813cd34b3987f58de5bbd7ffcfc00dc5bedc84" Oct 31 05:12:10.190596 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:12:10.190574 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de1f895f9dd3e6d776886171b1813cd34b3987f58de5bbd7ffcfc00dc5bedc84\": container with ID starting with de1f895f9dd3e6d776886171b1813cd34b3987f58de5bbd7ffcfc00dc5bedc84 not found: ID does not exist" containerID="de1f895f9dd3e6d776886171b1813cd34b3987f58de5bbd7ffcfc00dc5bedc84" Oct 31 05:12:10.190659 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:12:10.190608 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de1f895f9dd3e6d776886171b1813cd34b3987f58de5bbd7ffcfc00dc5bedc84"} err="failed to get container status \"de1f895f9dd3e6d776886171b1813cd34b3987f58de5bbd7ffcfc00dc5bedc84\": rpc error: code = NotFound desc = could not find container \"de1f895f9dd3e6d776886171b1813cd34b3987f58de5bbd7ffcfc00dc5bedc84\": container with ID starting with de1f895f9dd3e6d776886171b1813cd34b3987f58de5bbd7ffcfc00dc5bedc84 not found: ID does not exist" Oct 31 05:12:10.253386 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:12:10.253323 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe905f47-822c-4bbb-b562-8bd04f2b35a1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe905f47-822c-4bbb-b562-8bd04f2b35a1-plugins-config-volume\") pod \"fe905f47-822c-4bbb-b562-8bd04f2b35a1\" (UID: \"fe905f47-822c-4bbb-b562-8bd04f2b35a1\") " Oct 31 05:12:10.253790 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:12:10.253763 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe905f47-822c-4bbb-b562-8bd04f2b35a1-plugins-config-volume" pod "fe905f47-822c-4bbb-b562-8bd04f2b35a1" (UID: "fe905f47-822c-4bbb-b562-8bd04f2b35a1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:12:10.254473 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:12:10.254446 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe905f47-822c-4bbb-b562-8bd04f2b35a1-kube-api-access-jpwrb\" (UniqueName: \"kubernetes.io/projected/fe905f47-822c-4bbb-b562-8bd04f2b35a1-kube-api-access-jpwrb\") pod \"fe905f47-822c-4bbb-b562-8bd04f2b35a1\" (UID: \"fe905f47-822c-4bbb-b562-8bd04f2b35a1\") " Oct 31 05:12:10.255245 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:12:10.255068 4376 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe905f47-822c-4bbb-b562-8bd04f2b35a1-plugins-config-volume\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:12:10.257403 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:12:10.257379 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe905f47-822c-4bbb-b562-8bd04f2b35a1-kube-api-access-jpwrb" pod "fe905f47-822c-4bbb-b562-8bd04f2b35a1" (UID: "fe905f47-822c-4bbb-b562-8bd04f2b35a1"). InnerVolumeSpecName "kube-api-access-jpwrb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:12:10.370870 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:12:10.361919 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f31be489-e267-4087-9b94-86d7d622c841-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f31be489-e267-4087-9b94-86d7d622c841-plugins-config-volume\") pod \"f31be489-e267-4087-9b94-86d7d622c841\" (UID: \"f31be489-e267-4087-9b94-86d7d622c841\") " Oct 31 05:12:10.370870 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:12:10.362122 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f31be489-e267-4087-9b94-86d7d622c841-kube-api-access-fl7gh\" (UniqueName: \"kubernetes.io/projected/f31be489-e267-4087-9b94-86d7d622c841-kube-api-access-fl7gh\") pod \"f31be489-e267-4087-9b94-86d7d622c841\" (UID: \"f31be489-e267-4087-9b94-86d7d622c841\") " Oct 31 05:12:10.370870 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:12:10.362777 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jpwrb\" (UniqueName: \"kubernetes.io/projected/fe905f47-822c-4bbb-b562-8bd04f2b35a1-kube-api-access-jpwrb\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:12:10.398181 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:12:10.390971 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f31be489-e267-4087-9b94-86d7d622c841-kube-api-access-fl7gh" pod "f31be489-e267-4087-9b94-86d7d622c841" (UID: "f31be489-e267-4087-9b94-86d7d622c841"). InnerVolumeSpecName "kube-api-access-fl7gh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:12:10.398181 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:12:10.397468 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f31be489-e267-4087-9b94-86d7d622c841-plugins-config-volume" pod "f31be489-e267-4087-9b94-86d7d622c841" (UID: "f31be489-e267-4087-9b94-86d7d622c841"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:12:10.464292 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:12:10.464263 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fl7gh\" (UniqueName: \"kubernetes.io/projected/f31be489-e267-4087-9b94-86d7d622c841-kube-api-access-fl7gh\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:12:10.464292 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:12:10.464293 4376 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f31be489-e267-4087-9b94-86d7d622c841-plugins-config-volume\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:12:10.525825 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:12:10.525792 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-c9707f-5d5859459c-tv5n9"] Oct 31 05:12:10.531195 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:12:10.531167 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-c9707f-5d5859459c-tv5n9"] Oct 31 05:12:11.001437 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:12:11.001394 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 05:12:11.188632 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:12:11.188605 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c9707f-8598fffd47-fcvbn" Oct 31 05:12:11.248011 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:12:11.247972 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c9707f-8598fffd47-fcvbn"] Oct 31 05:12:11.261109 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:12:11.261078 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c9707f-8598fffd47-fcvbn"] Oct 31 05:12:12.445014 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:12:12.444982 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f31be489-e267-4087-9b94-86d7d622c841" path="/var/lib/kubelet/pods/f31be489-e267-4087-9b94-86d7d622c841/volumes" Oct 31 05:12:12.445539 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:12:12.445521 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe905f47-822c-4bbb-b562-8bd04f2b35a1" path="/var/lib/kubelet/pods/fe905f47-822c-4bbb-b562-8bd04f2b35a1/volumes" Oct 31 05:12:15.982172 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:12:15.982131 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-c9707f-openshift-default-65b9677b4c-b57kk"] Oct 31 05:12:15.982823 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:12:15.982452 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-c9707f-openshift-default-65b9677b4c-b57kk" podUID="932966ca-c565-4bab-a007-7f918c7f0c15" containerName="istio-proxy" containerID="cri-o://d19db211d103111c049291a91f9841f5f2e0da1f67602e3285ce2101fed0d941" gracePeriod=30 Oct 31 05:12:17.166002 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:12:17.165971 4376 scope.go:122] "RemoveContainer" containerID="245441bd30c6b7857276850e7acae00ac12f1bd56db09eb5521ead8cfc94b753" Oct 31 05:12:17.175816 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:12:17.175794 4376 scope.go:122] "RemoveContainer" containerID="cbd13093204d28e5a920b81f077ede0be2004eb54d991427492f1b562d91b89b" Oct 31 05:12:17.344393 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:12:17.344357 4376 patch_prober.go:29] interesting pod/e2e-shared-a-w2-c9707f-openshift-default-65b9677b4c-b57kk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 05:12:17.344586 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:12:17.344434 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-c9707f-openshift-default-65b9677b4c-b57kk" podUID="932966ca-c565-4bab-a007-7f918c7f0c15" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 05:12:21.255322 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:12:21.255260 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-c9707f-openshift-default-65b9677b4c-b57kk" Oct 31 05:12:21.275811 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:12:21.275781 4376 generic.go:356] "Generic (PLEG): container finished" podID="932966ca-c565-4bab-a007-7f918c7f0c15" containerID="d19db211d103111c049291a91f9841f5f2e0da1f67602e3285ce2101fed0d941" exitCode=0 Oct 31 05:12:21.275943 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:12:21.275826 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-c9707f-openshift-default-65b9677b4c-b57kk" Oct 31 05:12:21.275943 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:12:21.275871 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-c9707f-openshift-default-65b9677b4c-b57kk" event={"ID":"932966ca-c565-4bab-a007-7f918c7f0c15","Type":"ContainerDied","Data":"d19db211d103111c049291a91f9841f5f2e0da1f67602e3285ce2101fed0d941"} Oct 31 05:12:21.275943 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:12:21.275920 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-c9707f-openshift-default-65b9677b4c-b57kk" event={"ID":"932966ca-c565-4bab-a007-7f918c7f0c15","Type":"ContainerDied","Data":"97d19c8b418af56540f26c71d79c70cf2e7ed7a9feed3e8c30aa4d5385f38c08"} Oct 31 05:12:21.275943 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:12:21.275943 4376 scope.go:122] "RemoveContainer" containerID="d19db211d103111c049291a91f9841f5f2e0da1f67602e3285ce2101fed0d941" Oct 31 05:12:21.285692 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:12:21.285667 4376 scope.go:122] "RemoveContainer" containerID="0f66c5f0d835c6a13b765cb395beaa14cefce2a0b4d49bf6e756db60d2b08bfb" Oct 31 05:12:21.295274 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:12:21.295250 4376 scope.go:122] "RemoveContainer" containerID="d19db211d103111c049291a91f9841f5f2e0da1f67602e3285ce2101fed0d941" Oct 31 05:12:21.295564 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:12:21.295543 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d19db211d103111c049291a91f9841f5f2e0da1f67602e3285ce2101fed0d941\": container with ID starting with d19db211d103111c049291a91f9841f5f2e0da1f67602e3285ce2101fed0d941 not found: ID does not exist" containerID="d19db211d103111c049291a91f9841f5f2e0da1f67602e3285ce2101fed0d941" Oct 31 05:12:21.295626 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:12:21.295573 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d19db211d103111c049291a91f9841f5f2e0da1f67602e3285ce2101fed0d941"} err="failed to get container status \"d19db211d103111c049291a91f9841f5f2e0da1f67602e3285ce2101fed0d941\": rpc error: code = NotFound desc = could not find container \"d19db211d103111c049291a91f9841f5f2e0da1f67602e3285ce2101fed0d941\": container with ID starting with d19db211d103111c049291a91f9841f5f2e0da1f67602e3285ce2101fed0d941 not found: ID does not exist" Oct 31 05:12:21.295626 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:12:21.295592 4376 scope.go:122] "RemoveContainer" containerID="0f66c5f0d835c6a13b765cb395beaa14cefce2a0b4d49bf6e756db60d2b08bfb" Oct 31 05:12:21.295884 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:12:21.295866 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f66c5f0d835c6a13b765cb395beaa14cefce2a0b4d49bf6e756db60d2b08bfb\": container with ID starting with 0f66c5f0d835c6a13b765cb395beaa14cefce2a0b4d49bf6e756db60d2b08bfb not found: ID does not exist" containerID="0f66c5f0d835c6a13b765cb395beaa14cefce2a0b4d49bf6e756db60d2b08bfb" Oct 31 05:12:21.295939 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:12:21.295889 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f66c5f0d835c6a13b765cb395beaa14cefce2a0b4d49bf6e756db60d2b08bfb"} err="failed to get container status \"0f66c5f0d835c6a13b765cb395beaa14cefce2a0b4d49bf6e756db60d2b08bfb\": rpc error: code = NotFound desc = could not find container \"0f66c5f0d835c6a13b765cb395beaa14cefce2a0b4d49bf6e756db60d2b08bfb\": container with ID starting with 0f66c5f0d835c6a13b765cb395beaa14cefce2a0b4d49bf6e756db60d2b08bfb not found: ID does not exist" Oct 31 05:12:21.342964 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:12:21.342934 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/932966ca-c565-4bab-a007-7f918c7f0c15-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/932966ca-c565-4bab-a007-7f918c7f0c15-workload-certs\") pod \"932966ca-c565-4bab-a007-7f918c7f0c15\" (UID: \"932966ca-c565-4bab-a007-7f918c7f0c15\") " Oct 31 05:12:21.343140 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:12:21.343089 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/932966ca-c565-4bab-a007-7f918c7f0c15-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/932966ca-c565-4bab-a007-7f918c7f0c15-istiod-ca-cert\") pod \"932966ca-c565-4bab-a007-7f918c7f0c15\" (UID: \"932966ca-c565-4bab-a007-7f918c7f0c15\") " Oct 31 05:12:21.343219 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:12:21.343203 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/932966ca-c565-4bab-a007-7f918c7f0c15-workload-certs" pod "932966ca-c565-4bab-a007-7f918c7f0c15" (UID: "932966ca-c565-4bab-a007-7f918c7f0c15"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:12:21.343374 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:12:21.343248 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/932966ca-c565-4bab-a007-7f918c7f0c15-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/932966ca-c565-4bab-a007-7f918c7f0c15-workload-socket\") pod \"932966ca-c565-4bab-a007-7f918c7f0c15\" (UID: \"932966ca-c565-4bab-a007-7f918c7f0c15\") " Oct 31 05:12:21.343374 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:12:21.343288 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/932966ca-c565-4bab-a007-7f918c7f0c15-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/932966ca-c565-4bab-a007-7f918c7f0c15-istio-podinfo\") pod \"932966ca-c565-4bab-a007-7f918c7f0c15\" (UID: \"932966ca-c565-4bab-a007-7f918c7f0c15\") " Oct 31 05:12:21.343374 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:12:21.343312 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/932966ca-c565-4bab-a007-7f918c7f0c15-kube-api-access-kl5lh\" (UniqueName: \"kubernetes.io/projected/932966ca-c565-4bab-a007-7f918c7f0c15-kube-api-access-kl5lh\") pod \"932966ca-c565-4bab-a007-7f918c7f0c15\" (UID: \"932966ca-c565-4bab-a007-7f918c7f0c15\") " Oct 31 05:12:21.343571 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:12:21.343398 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/932966ca-c565-4bab-a007-7f918c7f0c15-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/932966ca-c565-4bab-a007-7f918c7f0c15-credential-socket\") pod \"932966ca-c565-4bab-a007-7f918c7f0c15\" (UID: \"932966ca-c565-4bab-a007-7f918c7f0c15\") " Oct 31 05:12:21.343571 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:12:21.343499 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/932966ca-c565-4bab-a007-7f918c7f0c15-workload-socket" pod "932966ca-c565-4bab-a007-7f918c7f0c15" (UID: "932966ca-c565-4bab-a007-7f918c7f0c15"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:12:21.343571 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:12:21.343517 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/932966ca-c565-4bab-a007-7f918c7f0c15-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/932966ca-c565-4bab-a007-7f918c7f0c15-istio-data\") pod \"932966ca-c565-4bab-a007-7f918c7f0c15\" (UID: \"932966ca-c565-4bab-a007-7f918c7f0c15\") " Oct 31 05:12:21.343571 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:12:21.343554 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/932966ca-c565-4bab-a007-7f918c7f0c15-istiod-ca-cert" pod "932966ca-c565-4bab-a007-7f918c7f0c15" (UID: "932966ca-c565-4bab-a007-7f918c7f0c15"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:12:21.343789 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:12:21.343616 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/932966ca-c565-4bab-a007-7f918c7f0c15-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/932966ca-c565-4bab-a007-7f918c7f0c15-istio-envoy\") pod \"932966ca-c565-4bab-a007-7f918c7f0c15\" (UID: \"932966ca-c565-4bab-a007-7f918c7f0c15\") " Oct 31 05:12:21.343789 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:12:21.343645 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/932966ca-c565-4bab-a007-7f918c7f0c15-istio-token\" (UniqueName: \"kubernetes.io/projected/932966ca-c565-4bab-a007-7f918c7f0c15-istio-token\") pod \"932966ca-c565-4bab-a007-7f918c7f0c15\" (UID: \"932966ca-c565-4bab-a007-7f918c7f0c15\") " Oct 31 05:12:21.343789 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:12:21.343714 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/932966ca-c565-4bab-a007-7f918c7f0c15-credential-socket" pod "932966ca-c565-4bab-a007-7f918c7f0c15" (UID: "932966ca-c565-4bab-a007-7f918c7f0c15"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:12:21.343967 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:12:21.343912 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/932966ca-c565-4bab-a007-7f918c7f0c15-istio-data" pod "932966ca-c565-4bab-a007-7f918c7f0c15" (UID: "932966ca-c565-4bab-a007-7f918c7f0c15"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:12:21.344382 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:12:21.344272 4376 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/932966ca-c565-4bab-a007-7f918c7f0c15-workload-certs\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:12:21.344382 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:12:21.344302 4376 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/932966ca-c565-4bab-a007-7f918c7f0c15-istiod-ca-cert\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:12:21.344382 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:12:21.344316 4376 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/932966ca-c565-4bab-a007-7f918c7f0c15-workload-socket\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:12:21.344382 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:12:21.344345 4376 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/932966ca-c565-4bab-a007-7f918c7f0c15-credential-socket\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:12:21.344382 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:12:21.344360 4376 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/932966ca-c565-4bab-a007-7f918c7f0c15-istio-data\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:12:21.345926 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:12:21.345890 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/932966ca-c565-4bab-a007-7f918c7f0c15-istio-envoy" pod "932966ca-c565-4bab-a007-7f918c7f0c15" (UID: "932966ca-c565-4bab-a007-7f918c7f0c15"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:12:21.346224 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:12:21.346195 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/932966ca-c565-4bab-a007-7f918c7f0c15-istio-podinfo" pod "932966ca-c565-4bab-a007-7f918c7f0c15" (UID: "932966ca-c565-4bab-a007-7f918c7f0c15"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 05:12:21.346467 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:12:21.346448 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/932966ca-c565-4bab-a007-7f918c7f0c15-kube-api-access-kl5lh" pod "932966ca-c565-4bab-a007-7f918c7f0c15" (UID: "932966ca-c565-4bab-a007-7f918c7f0c15"). InnerVolumeSpecName "kube-api-access-kl5lh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:12:21.346548 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:12:21.346479 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/932966ca-c565-4bab-a007-7f918c7f0c15-istio-token" pod "932966ca-c565-4bab-a007-7f918c7f0c15" (UID: "932966ca-c565-4bab-a007-7f918c7f0c15"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:12:21.445702 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:12:21.445670 4376 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/932966ca-c565-4bab-a007-7f918c7f0c15-istio-podinfo\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:12:21.445702 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:12:21.445700 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kl5lh\" (UniqueName: \"kubernetes.io/projected/932966ca-c565-4bab-a007-7f918c7f0c15-kube-api-access-kl5lh\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:12:21.445702 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:12:21.445712 4376 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/932966ca-c565-4bab-a007-7f918c7f0c15-istio-envoy\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:12:21.445943 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:12:21.445720 4376 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/932966ca-c565-4bab-a007-7f918c7f0c15-istio-token\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:12:21.628142 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:12:21.628108 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-c9707f-openshift-default-65b9677b4c-b57kk"] Oct 31 05:12:21.642357 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:12:21.642315 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-c9707f-openshift-default-65b9677b4c-b57kk"] Oct 31 05:12:22.446292 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:12:22.446259 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="932966ca-c565-4bab-a007-7f918c7f0c15" path="/var/lib/kubelet/pods/932966ca-c565-4bab-a007-7f918c7f0c15/volumes" Oct 31 05:12:36.775131 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:12:36.775069 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-77a153-openshift-default-7778f45d97-pzlhr"] Oct 31 05:12:36.775876 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:12:36.775356 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-77a153-openshift-default-7778f45d97-pzlhr" podUID="faaa68df-a884-400f-be9a-84bbb75ad520" containerName="istio-proxy" containerID="cri-o://2cbacd986d060bd433c53bf878c3d80a4fc6242f8e7f248db63a2b2a26633f17" gracePeriod=30 Oct 31 05:12:41.001260 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:12:41.001221 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 05:12:42.080440 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:12:42.080411 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-77a153-openshift-default-7778f45d97-pzlhr" Oct 31 05:12:42.207376 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:12:42.207294 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/faaa68df-a884-400f-be9a-84bbb75ad520-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/faaa68df-a884-400f-be9a-84bbb75ad520-istio-podinfo\") pod \"faaa68df-a884-400f-be9a-84bbb75ad520\" (UID: \"faaa68df-a884-400f-be9a-84bbb75ad520\") " Oct 31 05:12:42.207576 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:12:42.207391 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/faaa68df-a884-400f-be9a-84bbb75ad520-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/faaa68df-a884-400f-be9a-84bbb75ad520-credential-socket\") pod \"faaa68df-a884-400f-be9a-84bbb75ad520\" (UID: \"faaa68df-a884-400f-be9a-84bbb75ad520\") " Oct 31 05:12:42.207576 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:12:42.207420 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/faaa68df-a884-400f-be9a-84bbb75ad520-istio-token\" (UniqueName: \"kubernetes.io/projected/faaa68df-a884-400f-be9a-84bbb75ad520-istio-token\") pod \"faaa68df-a884-400f-be9a-84bbb75ad520\" (UID: \"faaa68df-a884-400f-be9a-84bbb75ad520\") " Oct 31 05:12:42.207576 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:12:42.207491 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/faaa68df-a884-400f-be9a-84bbb75ad520-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/faaa68df-a884-400f-be9a-84bbb75ad520-istio-data\") pod \"faaa68df-a884-400f-be9a-84bbb75ad520\" (UID: \"faaa68df-a884-400f-be9a-84bbb75ad520\") " Oct 31 05:12:42.207760 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:12:42.207607 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/faaa68df-a884-400f-be9a-84bbb75ad520-kube-api-access-2rgkp\" (UniqueName: \"kubernetes.io/projected/faaa68df-a884-400f-be9a-84bbb75ad520-kube-api-access-2rgkp\") pod \"faaa68df-a884-400f-be9a-84bbb75ad520\" (UID: \"faaa68df-a884-400f-be9a-84bbb75ad520\") " Oct 31 05:12:42.207760 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:12:42.207695 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/faaa68df-a884-400f-be9a-84bbb75ad520-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/faaa68df-a884-400f-be9a-84bbb75ad520-workload-certs\") pod \"faaa68df-a884-400f-be9a-84bbb75ad520\" (UID: \"faaa68df-a884-400f-be9a-84bbb75ad520\") " Oct 31 05:12:42.207866 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:12:42.207778 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/faaa68df-a884-400f-be9a-84bbb75ad520-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/faaa68df-a884-400f-be9a-84bbb75ad520-istio-envoy\") pod \"faaa68df-a884-400f-be9a-84bbb75ad520\" (UID: \"faaa68df-a884-400f-be9a-84bbb75ad520\") " Oct 31 05:12:42.207866 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:12:42.207703 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/faaa68df-a884-400f-be9a-84bbb75ad520-credential-socket" pod "faaa68df-a884-400f-be9a-84bbb75ad520" (UID: "faaa68df-a884-400f-be9a-84bbb75ad520"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:12:42.207866 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:12:42.207841 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/faaa68df-a884-400f-be9a-84bbb75ad520-istio-data" pod "faaa68df-a884-400f-be9a-84bbb75ad520" (UID: "faaa68df-a884-400f-be9a-84bbb75ad520"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:12:42.208037 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:12:42.207830 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/faaa68df-a884-400f-be9a-84bbb75ad520-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/faaa68df-a884-400f-be9a-84bbb75ad520-workload-socket\") pod \"faaa68df-a884-400f-be9a-84bbb75ad520\" (UID: \"faaa68df-a884-400f-be9a-84bbb75ad520\") " Oct 31 05:12:42.208037 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:12:42.207926 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/faaa68df-a884-400f-be9a-84bbb75ad520-workload-certs" pod "faaa68df-a884-400f-be9a-84bbb75ad520" (UID: "faaa68df-a884-400f-be9a-84bbb75ad520"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:12:42.208037 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:12:42.207980 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/faaa68df-a884-400f-be9a-84bbb75ad520-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/faaa68df-a884-400f-be9a-84bbb75ad520-istiod-ca-cert\") pod \"faaa68df-a884-400f-be9a-84bbb75ad520\" (UID: \"faaa68df-a884-400f-be9a-84bbb75ad520\") " Oct 31 05:12:42.208037 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:12:42.208028 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/faaa68df-a884-400f-be9a-84bbb75ad520-workload-socket" pod "faaa68df-a884-400f-be9a-84bbb75ad520" (UID: "faaa68df-a884-400f-be9a-84bbb75ad520"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:12:42.208442 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:12:42.208420 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/faaa68df-a884-400f-be9a-84bbb75ad520-istiod-ca-cert" pod "faaa68df-a884-400f-be9a-84bbb75ad520" (UID: "faaa68df-a884-400f-be9a-84bbb75ad520"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:12:42.208667 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:12:42.208650 4376 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/faaa68df-a884-400f-be9a-84bbb75ad520-workload-socket\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:12:42.208740 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:12:42.208673 4376 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/faaa68df-a884-400f-be9a-84bbb75ad520-istiod-ca-cert\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:12:42.208740 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:12:42.208690 4376 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/faaa68df-a884-400f-be9a-84bbb75ad520-credential-socket\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:12:42.208740 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:12:42.208703 4376 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/faaa68df-a884-400f-be9a-84bbb75ad520-istio-data\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:12:42.208740 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:12:42.208718 4376 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/faaa68df-a884-400f-be9a-84bbb75ad520-workload-certs\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:12:42.209626 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:12:42.209597 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/faaa68df-a884-400f-be9a-84bbb75ad520-istio-podinfo" pod "faaa68df-a884-400f-be9a-84bbb75ad520" (UID: "faaa68df-a884-400f-be9a-84bbb75ad520"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 05:12:42.209747 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:12:42.209728 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/faaa68df-a884-400f-be9a-84bbb75ad520-kube-api-access-2rgkp" pod "faaa68df-a884-400f-be9a-84bbb75ad520" (UID: "faaa68df-a884-400f-be9a-84bbb75ad520"). InnerVolumeSpecName "kube-api-access-2rgkp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:12:42.210023 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:12:42.210002 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/faaa68df-a884-400f-be9a-84bbb75ad520-istio-token" pod "faaa68df-a884-400f-be9a-84bbb75ad520" (UID: "faaa68df-a884-400f-be9a-84bbb75ad520"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:12:42.210064 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:12:42.210022 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/faaa68df-a884-400f-be9a-84bbb75ad520-istio-envoy" pod "faaa68df-a884-400f-be9a-84bbb75ad520" (UID: "faaa68df-a884-400f-be9a-84bbb75ad520"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:12:42.309822 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:12:42.309729 4376 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/faaa68df-a884-400f-be9a-84bbb75ad520-istio-podinfo\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:12:42.309822 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:12:42.309759 4376 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/faaa68df-a884-400f-be9a-84bbb75ad520-istio-token\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:12:42.309822 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:12:42.309774 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rgkp\" (UniqueName: \"kubernetes.io/projected/faaa68df-a884-400f-be9a-84bbb75ad520-kube-api-access-2rgkp\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:12:42.309822 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:12:42.309787 4376 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/faaa68df-a884-400f-be9a-84bbb75ad520-istio-envoy\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:12:42.418699 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:12:42.418665 4376 generic.go:356] "Generic (PLEG): container finished" podID="faaa68df-a884-400f-be9a-84bbb75ad520" containerID="2cbacd986d060bd433c53bf878c3d80a4fc6242f8e7f248db63a2b2a26633f17" exitCode=0 Oct 31 05:12:42.418879 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:12:42.418712 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-77a153-openshift-default-7778f45d97-pzlhr" Oct 31 05:12:42.418879 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:12:42.418743 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-77a153-openshift-default-7778f45d97-pzlhr" event={"ID":"faaa68df-a884-400f-be9a-84bbb75ad520","Type":"ContainerDied","Data":"2cbacd986d060bd433c53bf878c3d80a4fc6242f8e7f248db63a2b2a26633f17"} Oct 31 05:12:42.418879 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:12:42.418810 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-77a153-openshift-default-7778f45d97-pzlhr" event={"ID":"faaa68df-a884-400f-be9a-84bbb75ad520","Type":"ContainerDied","Data":"7a439005c5a6e2ade2ef8d6bacf159b08cdb236b532162271b79b66b3e9434b1"} Oct 31 05:12:42.418879 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:12:42.418840 4376 scope.go:122] "RemoveContainer" containerID="2cbacd986d060bd433c53bf878c3d80a4fc6242f8e7f248db63a2b2a26633f17" Oct 31 05:12:42.430117 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:12:42.430095 4376 scope.go:122] "RemoveContainer" containerID="2cbacd986d060bd433c53bf878c3d80a4fc6242f8e7f248db63a2b2a26633f17" Oct 31 05:12:42.430474 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:12:42.430451 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2cbacd986d060bd433c53bf878c3d80a4fc6242f8e7f248db63a2b2a26633f17\": container with ID starting with 2cbacd986d060bd433c53bf878c3d80a4fc6242f8e7f248db63a2b2a26633f17 not found: ID does not exist" containerID="2cbacd986d060bd433c53bf878c3d80a4fc6242f8e7f248db63a2b2a26633f17" Oct 31 05:12:42.430551 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:12:42.430482 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2cbacd986d060bd433c53bf878c3d80a4fc6242f8e7f248db63a2b2a26633f17"} err="failed to get container status \"2cbacd986d060bd433c53bf878c3d80a4fc6242f8e7f248db63a2b2a26633f17\": rpc error: code = NotFound desc = could not find container \"2cbacd986d060bd433c53bf878c3d80a4fc6242f8e7f248db63a2b2a26633f17\": container with ID starting with 2cbacd986d060bd433c53bf878c3d80a4fc6242f8e7f248db63a2b2a26633f17 not found: ID does not exist" Oct 31 05:12:42.507141 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:12:42.507099 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-77a153-openshift-default-7778f45d97-pzlhr"] Oct 31 05:12:42.512775 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:12:42.512731 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-77a153-openshift-default-7778f45d97-pzlhr"] Oct 31 05:12:43.106567 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:12:43.106527 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-d6ac82-75d47769c-bb76f"] Oct 31 05:12:43.137205 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:12:43.137163 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-d6ac82-8687db9c8c-c4fl6"] Oct 31 05:12:44.540363 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:12:44.536054 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="faaa68df-a884-400f-be9a-84bbb75ad520" path="/var/lib/kubelet/pods/faaa68df-a884-400f-be9a-84bbb75ad520/volumes" Oct 31 05:12:48.107615 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:12:48.107563 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-d6ac82-75d47769c-bb76f" podUID="a0071e27-190b-4977-9c6f-c3458c714d64" containerName="payload-processing" containerID="cri-o://2c93fdbb4630ac291697246dd077e2008daf68851d44c57151ffe6b21ad4a622" gracePeriod=25 Oct 31 05:12:48.138317 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:12:48.138268 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-d6ac82-8687db9c8c-c4fl6" podUID="adda31ec-5cfd-473c-8edd-5e7d335aec12" containerName="payload-pre-processing" containerID="cri-o://2dbfcb68e1ad6f89a0159ea41616e9fc93671218625c0f197dd23c46cea2569f" gracePeriod=25 Oct 31 05:12:49.047601 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:12:49.047565 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-d6ac82-8687db9c8c-c4fl6" podUID="adda31ec-5cfd-473c-8edd-5e7d335aec12" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 05:12:49.048660 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:12:49.048631 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-d6ac82-75d47769c-bb76f" podUID="a0071e27-190b-4977-9c6f-c3458c714d64" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 05:12:57.642069 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:12:57.642036 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-439989/premium-w1-439989-kserve-f9fc54cdf-sn2pp"] Oct 31 05:12:59.047109 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:12:59.047070 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-d6ac82-8687db9c8c-c4fl6" podUID="adda31ec-5cfd-473c-8edd-5e7d335aec12" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 05:12:59.048688 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:12:59.048650 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-d6ac82-75d47769c-bb76f" podUID="a0071e27-190b-4977-9c6f-c3458c714d64" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 05:12:59.384918 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:12:59.384882 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 05:13:08.415065 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:08.415040 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-d6ac82-75d47769c-bb76f" Oct 31 05:13:08.473146 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:08.473114 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-d6ac82-8687db9c8c-c4fl6" Oct 31 05:13:08.553217 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:08.553188 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0071e27-190b-4977-9c6f-c3458c714d64-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0071e27-190b-4977-9c6f-c3458c714d64-plugins-config-volume\") pod \"a0071e27-190b-4977-9c6f-c3458c714d64\" (UID: \"a0071e27-190b-4977-9c6f-c3458c714d64\") " Oct 31 05:13:08.553453 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:08.553305 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/adda31ec-5cfd-473c-8edd-5e7d335aec12-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/adda31ec-5cfd-473c-8edd-5e7d335aec12-plugins-config-volume\") pod \"adda31ec-5cfd-473c-8edd-5e7d335aec12\" (UID: \"adda31ec-5cfd-473c-8edd-5e7d335aec12\") " Oct 31 05:13:08.553453 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:08.553361 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0071e27-190b-4977-9c6f-c3458c714d64-kube-api-access-xz99d\" (UniqueName: \"kubernetes.io/projected/a0071e27-190b-4977-9c6f-c3458c714d64-kube-api-access-xz99d\") pod \"a0071e27-190b-4977-9c6f-c3458c714d64\" (UID: \"a0071e27-190b-4977-9c6f-c3458c714d64\") " Oct 31 05:13:08.553659 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:08.553630 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/adda31ec-5cfd-473c-8edd-5e7d335aec12-kube-api-access-k7sbn\" (UniqueName: \"kubernetes.io/projected/adda31ec-5cfd-473c-8edd-5e7d335aec12-kube-api-access-k7sbn\") pod \"adda31ec-5cfd-473c-8edd-5e7d335aec12\" (UID: \"adda31ec-5cfd-473c-8edd-5e7d335aec12\") " Oct 31 05:13:08.554372 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:08.554310 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/adda31ec-5cfd-473c-8edd-5e7d335aec12-plugins-config-volume" pod "adda31ec-5cfd-473c-8edd-5e7d335aec12" (UID: "adda31ec-5cfd-473c-8edd-5e7d335aec12"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:13:08.554512 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:08.554406 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0071e27-190b-4977-9c6f-c3458c714d64-plugins-config-volume" pod "a0071e27-190b-4977-9c6f-c3458c714d64" (UID: "a0071e27-190b-4977-9c6f-c3458c714d64"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:13:08.557747 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:08.557720 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0071e27-190b-4977-9c6f-c3458c714d64-kube-api-access-xz99d" pod "a0071e27-190b-4977-9c6f-c3458c714d64" (UID: "a0071e27-190b-4977-9c6f-c3458c714d64"). InnerVolumeSpecName "kube-api-access-xz99d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:13:08.557868 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:08.557798 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adda31ec-5cfd-473c-8edd-5e7d335aec12-kube-api-access-k7sbn" pod "adda31ec-5cfd-473c-8edd-5e7d335aec12" (UID: "adda31ec-5cfd-473c-8edd-5e7d335aec12"). InnerVolumeSpecName "kube-api-access-k7sbn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:13:08.655370 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:08.655271 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7sbn\" (UniqueName: \"kubernetes.io/projected/adda31ec-5cfd-473c-8edd-5e7d335aec12-kube-api-access-k7sbn\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:13:08.655370 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:08.655299 4376 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0071e27-190b-4977-9c6f-c3458c714d64-plugins-config-volume\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:13:08.655370 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:08.655309 4376 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/adda31ec-5cfd-473c-8edd-5e7d335aec12-plugins-config-volume\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:13:08.655370 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:08.655320 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xz99d\" (UniqueName: \"kubernetes.io/projected/a0071e27-190b-4977-9c6f-c3458c714d64-kube-api-access-xz99d\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:13:08.745585 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:08.745550 4376 generic.go:356] "Generic (PLEG): container finished" podID="adda31ec-5cfd-473c-8edd-5e7d335aec12" containerID="2dbfcb68e1ad6f89a0159ea41616e9fc93671218625c0f197dd23c46cea2569f" exitCode=0 Oct 31 05:13:08.745780 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:08.745608 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-d6ac82-8687db9c8c-c4fl6" Oct 31 05:13:08.745780 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:08.745650 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-d6ac82-8687db9c8c-c4fl6" event={"ID":"adda31ec-5cfd-473c-8edd-5e7d335aec12","Type":"ContainerDied","Data":"2dbfcb68e1ad6f89a0159ea41616e9fc93671218625c0f197dd23c46cea2569f"} Oct 31 05:13:08.745780 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:08.745710 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-d6ac82-8687db9c8c-c4fl6" event={"ID":"adda31ec-5cfd-473c-8edd-5e7d335aec12","Type":"ContainerDied","Data":"23a64a7c21ca76331d22954118444061e6cfa973f3160c34d8947fefa1f08a61"} Oct 31 05:13:08.745780 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:08.745739 4376 scope.go:122] "RemoveContainer" containerID="2dbfcb68e1ad6f89a0159ea41616e9fc93671218625c0f197dd23c46cea2569f" Oct 31 05:13:08.747504 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:08.747480 4376 generic.go:356] "Generic (PLEG): container finished" podID="a0071e27-190b-4977-9c6f-c3458c714d64" containerID="2c93fdbb4630ac291697246dd077e2008daf68851d44c57151ffe6b21ad4a622" exitCode=0 Oct 31 05:13:08.747625 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:08.747532 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-d6ac82-75d47769c-bb76f" Oct 31 05:13:08.747625 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:08.747537 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-d6ac82-75d47769c-bb76f" event={"ID":"a0071e27-190b-4977-9c6f-c3458c714d64","Type":"ContainerDied","Data":"2c93fdbb4630ac291697246dd077e2008daf68851d44c57151ffe6b21ad4a622"} Oct 31 05:13:08.747625 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:08.747588 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-d6ac82-75d47769c-bb76f" event={"ID":"a0071e27-190b-4977-9c6f-c3458c714d64","Type":"ContainerDied","Data":"a5aadd551e81cc654994327612114143c1894e7c5207792b58063e5148ac1b6c"} Oct 31 05:13:08.758102 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:08.758084 4376 scope.go:122] "RemoveContainer" containerID="2dbfcb68e1ad6f89a0159ea41616e9fc93671218625c0f197dd23c46cea2569f" Oct 31 05:13:08.758390 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:13:08.758368 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2dbfcb68e1ad6f89a0159ea41616e9fc93671218625c0f197dd23c46cea2569f\": container with ID starting with 2dbfcb68e1ad6f89a0159ea41616e9fc93671218625c0f197dd23c46cea2569f not found: ID does not exist" containerID="2dbfcb68e1ad6f89a0159ea41616e9fc93671218625c0f197dd23c46cea2569f" Oct 31 05:13:08.758454 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:08.758398 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2dbfcb68e1ad6f89a0159ea41616e9fc93671218625c0f197dd23c46cea2569f"} err="failed to get container status \"2dbfcb68e1ad6f89a0159ea41616e9fc93671218625c0f197dd23c46cea2569f\": rpc error: code = NotFound desc = could not find container \"2dbfcb68e1ad6f89a0159ea41616e9fc93671218625c0f197dd23c46cea2569f\": container with ID starting with 2dbfcb68e1ad6f89a0159ea41616e9fc93671218625c0f197dd23c46cea2569f not found: ID does not exist" Oct 31 05:13:08.758454 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:08.758417 4376 scope.go:122] "RemoveContainer" containerID="2c93fdbb4630ac291697246dd077e2008daf68851d44c57151ffe6b21ad4a622" Oct 31 05:13:08.769263 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:08.769240 4376 scope.go:122] "RemoveContainer" containerID="2c93fdbb4630ac291697246dd077e2008daf68851d44c57151ffe6b21ad4a622" Oct 31 05:13:08.769581 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:13:08.769559 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c93fdbb4630ac291697246dd077e2008daf68851d44c57151ffe6b21ad4a622\": container with ID starting with 2c93fdbb4630ac291697246dd077e2008daf68851d44c57151ffe6b21ad4a622 not found: ID does not exist" containerID="2c93fdbb4630ac291697246dd077e2008daf68851d44c57151ffe6b21ad4a622" Oct 31 05:13:08.769671 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:08.769586 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c93fdbb4630ac291697246dd077e2008daf68851d44c57151ffe6b21ad4a622"} err="failed to get container status \"2c93fdbb4630ac291697246dd077e2008daf68851d44c57151ffe6b21ad4a622\": rpc error: code = NotFound desc = could not find container \"2c93fdbb4630ac291697246dd077e2008daf68851d44c57151ffe6b21ad4a622\": container with ID starting with 2c93fdbb4630ac291697246dd077e2008daf68851d44c57151ffe6b21ad4a622 not found: ID does not exist" Oct 31 05:13:08.807364 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:08.807322 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-d6ac82-75d47769c-bb76f"] Oct 31 05:13:08.811672 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:08.811649 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-d6ac82-75d47769c-bb76f"] Oct 31 05:13:08.850863 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:08.850832 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-d6ac82-8687db9c8c-c4fl6"] Oct 31 05:13:08.856731 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:08.856705 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-d6ac82-8687db9c8c-c4fl6"] Oct 31 05:13:10.037083 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:10.037051 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-439989-5d779f9865-lc48d"] Oct 31 05:13:10.406161 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:10.406125 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-09a20a/distinct-2-w2-09a20a-kserve-779d648d75-mshl7"] Oct 31 05:13:10.413825 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:10.413794 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-09a20a/distinct-w2-09a20a-kserve-76557f6448-l8d69"] Oct 31 05:13:10.422658 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:10.422631 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-09a20a/premium-w2-09a20a-kserve-57c45db74f-s8glf"] Oct 31 05:13:10.445307 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:10.445280 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0071e27-190b-4977-9c6f-c3458c714d64" path="/var/lib/kubelet/pods/a0071e27-190b-4977-9c6f-c3458c714d64/volumes" Oct 31 05:13:10.445743 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:10.445730 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="adda31ec-5cfd-473c-8edd-5e7d335aec12" path="/var/lib/kubelet/pods/adda31ec-5cfd-473c-8edd-5e7d335aec12/volumes" Oct 31 05:13:11.001514 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:13:11.001469 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 05:13:12.663470 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:12.663428 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-439989/premium-w1-439989-kserve-f9fc54cdf-sn2pp" podUID="17b2f181-2e66-4ce2-95b0-cb213f11a0e7" containerName="main" containerID="cri-o://dcc1b1ea0c718cc4cc60686b4e335294d9a3a0518243b36a4576ad0ce4c0b538" gracePeriod=45 Oct 31 05:13:12.777990 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:12.777960 4376 generic.go:356] "Generic (PLEG): container finished" podID="17b2f181-2e66-4ce2-95b0-cb213f11a0e7" containerID="dcc1b1ea0c718cc4cc60686b4e335294d9a3a0518243b36a4576ad0ce4c0b538" exitCode=0 Oct 31 05:13:12.778106 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:12.778033 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-439989/premium-w1-439989-kserve-f9fc54cdf-sn2pp" event={"ID":"17b2f181-2e66-4ce2-95b0-cb213f11a0e7","Type":"ContainerDied","Data":"dcc1b1ea0c718cc4cc60686b4e335294d9a3a0518243b36a4576ad0ce4c0b538"} Oct 31 05:13:12.934518 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:12.934487 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-439989/premium-w1-439989-kserve-f9fc54cdf-sn2pp" Oct 31 05:13:13.030395 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:13.030355 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17b2f181-2e66-4ce2-95b0-cb213f11a0e7-kube-api-access-ffw28\" (UniqueName: \"kubernetes.io/projected/17b2f181-2e66-4ce2-95b0-cb213f11a0e7-kube-api-access-ffw28\") pod \"17b2f181-2e66-4ce2-95b0-cb213f11a0e7\" (UID: \"17b2f181-2e66-4ce2-95b0-cb213f11a0e7\") " Oct 31 05:13:13.030586 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:13.030409 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17b2f181-2e66-4ce2-95b0-cb213f11a0e7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17b2f181-2e66-4ce2-95b0-cb213f11a0e7-tmp-dir\") pod \"17b2f181-2e66-4ce2-95b0-cb213f11a0e7\" (UID: \"17b2f181-2e66-4ce2-95b0-cb213f11a0e7\") " Oct 31 05:13:13.030586 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:13.030529 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/17b2f181-2e66-4ce2-95b0-cb213f11a0e7-tls-certs\" (UniqueName: \"kubernetes.io/secret/17b2f181-2e66-4ce2-95b0-cb213f11a0e7-tls-certs\") pod \"17b2f181-2e66-4ce2-95b0-cb213f11a0e7\" (UID: \"17b2f181-2e66-4ce2-95b0-cb213f11a0e7\") " Oct 31 05:13:13.030734 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:13.030630 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17b2f181-2e66-4ce2-95b0-cb213f11a0e7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/17b2f181-2e66-4ce2-95b0-cb213f11a0e7-dshm\") pod \"17b2f181-2e66-4ce2-95b0-cb213f11a0e7\" (UID: \"17b2f181-2e66-4ce2-95b0-cb213f11a0e7\") " Oct 31 05:13:13.030734 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:13.030656 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17b2f181-2e66-4ce2-95b0-cb213f11a0e7-home\" (UniqueName: \"kubernetes.io/empty-dir/17b2f181-2e66-4ce2-95b0-cb213f11a0e7-home\") pod \"17b2f181-2e66-4ce2-95b0-cb213f11a0e7\" (UID: \"17b2f181-2e66-4ce2-95b0-cb213f11a0e7\") " Oct 31 05:13:13.030734 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:13.030716 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17b2f181-2e66-4ce2-95b0-cb213f11a0e7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17b2f181-2e66-4ce2-95b0-cb213f11a0e7-model-cache\") pod \"17b2f181-2e66-4ce2-95b0-cb213f11a0e7\" (UID: \"17b2f181-2e66-4ce2-95b0-cb213f11a0e7\") " Oct 31 05:13:13.030874 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:13.030751 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17b2f181-2e66-4ce2-95b0-cb213f11a0e7-tmp-dir" pod "17b2f181-2e66-4ce2-95b0-cb213f11a0e7" (UID: "17b2f181-2e66-4ce2-95b0-cb213f11a0e7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:13:13.031470 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:13.031137 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17b2f181-2e66-4ce2-95b0-cb213f11a0e7-home" pod "17b2f181-2e66-4ce2-95b0-cb213f11a0e7" (UID: "17b2f181-2e66-4ce2-95b0-cb213f11a0e7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:13:13.031470 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:13.031369 4376 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17b2f181-2e66-4ce2-95b0-cb213f11a0e7-home\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:13:13.031470 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:13.031387 4376 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17b2f181-2e66-4ce2-95b0-cb213f11a0e7-tmp-dir\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:13:13.031470 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:13.031439 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17b2f181-2e66-4ce2-95b0-cb213f11a0e7-model-cache" pod "17b2f181-2e66-4ce2-95b0-cb213f11a0e7" (UID: "17b2f181-2e66-4ce2-95b0-cb213f11a0e7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:13:13.032858 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:13.032829 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17b2f181-2e66-4ce2-95b0-cb213f11a0e7-kube-api-access-ffw28" pod "17b2f181-2e66-4ce2-95b0-cb213f11a0e7" (UID: "17b2f181-2e66-4ce2-95b0-cb213f11a0e7"). InnerVolumeSpecName "kube-api-access-ffw28". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:13:13.033107 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:13.033078 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17b2f181-2e66-4ce2-95b0-cb213f11a0e7-tls-certs" pod "17b2f181-2e66-4ce2-95b0-cb213f11a0e7" (UID: "17b2f181-2e66-4ce2-95b0-cb213f11a0e7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 05:13:13.033252 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:13.033210 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17b2f181-2e66-4ce2-95b0-cb213f11a0e7-dshm" pod "17b2f181-2e66-4ce2-95b0-cb213f11a0e7" (UID: "17b2f181-2e66-4ce2-95b0-cb213f11a0e7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:13:13.132710 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:13.132678 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ffw28\" (UniqueName: \"kubernetes.io/projected/17b2f181-2e66-4ce2-95b0-cb213f11a0e7-kube-api-access-ffw28\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:13:13.132710 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:13.132709 4376 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17b2f181-2e66-4ce2-95b0-cb213f11a0e7-tls-certs\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:13:13.132710 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:13.132720 4376 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17b2f181-2e66-4ce2-95b0-cb213f11a0e7-dshm\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:13:13.132961 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:13.132733 4376 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17b2f181-2e66-4ce2-95b0-cb213f11a0e7-model-cache\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:13:13.785701 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:13.785669 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-439989/premium-w1-439989-kserve-f9fc54cdf-sn2pp" Oct 31 05:13:13.786194 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:13.785701 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-439989/premium-w1-439989-kserve-f9fc54cdf-sn2pp" event={"ID":"17b2f181-2e66-4ce2-95b0-cb213f11a0e7","Type":"ContainerDied","Data":"18ac60be655227c8c23cafcf62e75c9870a6bfab5ba8e5302dca2181928c9797"} Oct 31 05:13:13.786194 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:13.785761 4376 scope.go:122] "RemoveContainer" containerID="dcc1b1ea0c718cc4cc60686b4e335294d9a3a0518243b36a4576ad0ce4c0b538" Oct 31 05:13:13.843846 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:13.843810 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-439989/premium-w1-439989-kserve-f9fc54cdf-sn2pp"] Oct 31 05:13:13.857981 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:13.857954 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-439989/premium-w1-439989-kserve-f9fc54cdf-sn2pp"] Oct 31 05:13:14.444298 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:14.444266 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17b2f181-2e66-4ce2-95b0-cb213f11a0e7" path="/var/lib/kubelet/pods/17b2f181-2e66-4ce2-95b0-cb213f11a0e7/volumes" Oct 31 05:13:15.037890 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:15.037845 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-439989-5d779f9865-lc48d" podUID="0a252fc6-14bf-4524-907a-f8ee294acda5" containerName="payload-processing" containerID="cri-o://f676360a316ac36141aebd4e14dc1b541d41b2d44de27e267b4315a9b52ac0d7" gracePeriod=25 Oct 31 05:13:19.230181 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:19.230139 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-439989-5d779f9865-lc48d" podUID="0a252fc6-14bf-4524-907a-f8ee294acda5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 05:13:22.156687 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:22.156641 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-09a20a-b4f7d6d49-2lp58"] Oct 31 05:13:22.182072 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:22.182028 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-09a20a-5bf985cbc9-scqzr"] Oct 31 05:13:25.435311 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:25.435264 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-09a20a/distinct-2-w2-09a20a-kserve-779d648d75-mshl7" podUID="c81f5d58-c5c7-46c1-b832-c550b092f02b" containerName="main" containerID="cri-o://0b2f66ca1a9f304890933a12fda4dc76a29093853423773883f4866b86978ffd" gracePeriod=45 Oct 31 05:13:25.437298 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:25.437254 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-09a20a/distinct-w2-09a20a-kserve-76557f6448-l8d69" podUID="7d298000-b65b-45fd-a1ee-092e47a7eb24" containerName="main" containerID="cri-o://c6f1417f936622a50af2a4c91f291b31122a0958bfaa10f70676fdbb15c3e4b2" gracePeriod=45 Oct 31 05:13:25.446851 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:25.446816 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-09a20a/premium-w2-09a20a-kserve-57c45db74f-s8glf" podUID="a3f9d0a0-a106-48bb-b6f2-a980c3aff4e7" containerName="main" containerID="cri-o://0ecb5f1038cfbba39690602f96218fc175fbff09c119a5928389d7fd428ac72d" gracePeriod=45 Oct 31 05:13:25.644894 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:25.644608 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-09a20a/premium-w2-09a20a-kserve-57c45db74f-s8glf" podUID="a3f9d0a0-a106-48bb-b6f2-a980c3aff4e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": dial tcp 10.133.0.58:8000: connect: connection refused" Oct 31 05:13:25.768812 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:25.768785 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-09a20a/distinct-w2-09a20a-kserve-76557f6448-l8d69" Oct 31 05:13:25.772591 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:25.772567 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-09a20a/distinct-2-w2-09a20a-kserve-779d648d75-mshl7" Oct 31 05:13:25.778741 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:25.778701 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-09a20a/premium-w2-09a20a-kserve-57c45db74f-s8glf" Oct 31 05:13:25.828182 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:25.827899 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d298000-b65b-45fd-a1ee-092e47a7eb24-kube-api-access-bsxlc\" (UniqueName: \"kubernetes.io/projected/7d298000-b65b-45fd-a1ee-092e47a7eb24-kube-api-access-bsxlc\") pod \"7d298000-b65b-45fd-a1ee-092e47a7eb24\" (UID: \"7d298000-b65b-45fd-a1ee-092e47a7eb24\") " Oct 31 05:13:25.828182 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:25.827959 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d298000-b65b-45fd-a1ee-092e47a7eb24-home\" (UniqueName: \"kubernetes.io/empty-dir/7d298000-b65b-45fd-a1ee-092e47a7eb24-home\") pod \"7d298000-b65b-45fd-a1ee-092e47a7eb24\" (UID: \"7d298000-b65b-45fd-a1ee-092e47a7eb24\") " Oct 31 05:13:25.828182 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:25.827998 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c81f5d58-c5c7-46c1-b832-c550b092f02b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c81f5d58-c5c7-46c1-b832-c550b092f02b-model-cache\") pod \"c81f5d58-c5c7-46c1-b832-c550b092f02b\" (UID: \"c81f5d58-c5c7-46c1-b832-c550b092f02b\") " Oct 31 05:13:25.828182 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:25.828034 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3f9d0a0-a106-48bb-b6f2-a980c3aff4e7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3f9d0a0-a106-48bb-b6f2-a980c3aff4e7-dshm\") pod \"a3f9d0a0-a106-48bb-b6f2-a980c3aff4e7\" (UID: \"a3f9d0a0-a106-48bb-b6f2-a980c3aff4e7\") " Oct 31 05:13:25.828182 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:25.828061 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c81f5d58-c5c7-46c1-b832-c550b092f02b-kube-api-access-sswx2\" (UniqueName: \"kubernetes.io/projected/c81f5d58-c5c7-46c1-b832-c550b092f02b-kube-api-access-sswx2\") pod \"c81f5d58-c5c7-46c1-b832-c550b092f02b\" (UID: \"c81f5d58-c5c7-46c1-b832-c550b092f02b\") " Oct 31 05:13:25.828182 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:25.828110 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d298000-b65b-45fd-a1ee-092e47a7eb24-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d298000-b65b-45fd-a1ee-092e47a7eb24-tmp-dir\") pod \"7d298000-b65b-45fd-a1ee-092e47a7eb24\" (UID: \"7d298000-b65b-45fd-a1ee-092e47a7eb24\") " Oct 31 05:13:25.828921 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:25.828711 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3f9d0a0-a106-48bb-b6f2-a980c3aff4e7-home\" (UniqueName: \"kubernetes.io/empty-dir/a3f9d0a0-a106-48bb-b6f2-a980c3aff4e7-home\") pod \"a3f9d0a0-a106-48bb-b6f2-a980c3aff4e7\" (UID: \"a3f9d0a0-a106-48bb-b6f2-a980c3aff4e7\") " Oct 31 05:13:25.828921 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:25.828855 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3f9d0a0-a106-48bb-b6f2-a980c3aff4e7-kube-api-access-dvsj5\" (UniqueName: \"kubernetes.io/projected/a3f9d0a0-a106-48bb-b6f2-a980c3aff4e7-kube-api-access-dvsj5\") pod \"a3f9d0a0-a106-48bb-b6f2-a980c3aff4e7\" (UID: \"a3f9d0a0-a106-48bb-b6f2-a980c3aff4e7\") " Oct 31 05:13:25.829147 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:25.829110 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d298000-b65b-45fd-a1ee-092e47a7eb24-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d298000-b65b-45fd-a1ee-092e47a7eb24-dshm\") pod \"7d298000-b65b-45fd-a1ee-092e47a7eb24\" (UID: \"7d298000-b65b-45fd-a1ee-092e47a7eb24\") " Oct 31 05:13:25.829311 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:25.829179 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c81f5d58-c5c7-46c1-b832-c550b092f02b-home\" (UniqueName: \"kubernetes.io/empty-dir/c81f5d58-c5c7-46c1-b832-c550b092f02b-home\") pod \"c81f5d58-c5c7-46c1-b832-c550b092f02b\" (UID: \"c81f5d58-c5c7-46c1-b832-c550b092f02b\") " Oct 31 05:13:25.829311 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:25.829226 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3f9d0a0-a106-48bb-b6f2-a980c3aff4e7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3f9d0a0-a106-48bb-b6f2-a980c3aff4e7-tmp-dir\") pod \"a3f9d0a0-a106-48bb-b6f2-a980c3aff4e7\" (UID: \"a3f9d0a0-a106-48bb-b6f2-a980c3aff4e7\") " Oct 31 05:13:25.829311 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:25.829249 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c81f5d58-c5c7-46c1-b832-c550b092f02b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c81f5d58-c5c7-46c1-b832-c550b092f02b-dshm\") pod \"c81f5d58-c5c7-46c1-b832-c550b092f02b\" (UID: \"c81f5d58-c5c7-46c1-b832-c550b092f02b\") " Oct 31 05:13:25.829311 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:25.829281 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d298000-b65b-45fd-a1ee-092e47a7eb24-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d298000-b65b-45fd-a1ee-092e47a7eb24-model-cache\") pod \"7d298000-b65b-45fd-a1ee-092e47a7eb24\" (UID: \"7d298000-b65b-45fd-a1ee-092e47a7eb24\") " Oct 31 05:13:25.829702 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:25.829350 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d298000-b65b-45fd-a1ee-092e47a7eb24-tls-certs\" (UniqueName: \"kubernetes.io/secret/7d298000-b65b-45fd-a1ee-092e47a7eb24-tls-certs\") pod \"7d298000-b65b-45fd-a1ee-092e47a7eb24\" (UID: \"7d298000-b65b-45fd-a1ee-092e47a7eb24\") " Oct 31 05:13:25.829702 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:25.829402 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c81f5d58-c5c7-46c1-b832-c550b092f02b-tls-certs\" (UniqueName: \"kubernetes.io/secret/c81f5d58-c5c7-46c1-b832-c550b092f02b-tls-certs\") pod \"c81f5d58-c5c7-46c1-b832-c550b092f02b\" (UID: \"c81f5d58-c5c7-46c1-b832-c550b092f02b\") " Oct 31 05:13:25.829702 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:25.829426 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3f9d0a0-a106-48bb-b6f2-a980c3aff4e7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3f9d0a0-a106-48bb-b6f2-a980c3aff4e7-model-cache\") pod \"a3f9d0a0-a106-48bb-b6f2-a980c3aff4e7\" (UID: \"a3f9d0a0-a106-48bb-b6f2-a980c3aff4e7\") " Oct 31 05:13:25.829702 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:25.829517 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a3f9d0a0-a106-48bb-b6f2-a980c3aff4e7-tls-certs\" (UniqueName: \"kubernetes.io/secret/a3f9d0a0-a106-48bb-b6f2-a980c3aff4e7-tls-certs\") pod \"a3f9d0a0-a106-48bb-b6f2-a980c3aff4e7\" (UID: \"a3f9d0a0-a106-48bb-b6f2-a980c3aff4e7\") " Oct 31 05:13:25.829702 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:25.829593 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c81f5d58-c5c7-46c1-b832-c550b092f02b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c81f5d58-c5c7-46c1-b832-c550b092f02b-tmp-dir\") pod \"c81f5d58-c5c7-46c1-b832-c550b092f02b\" (UID: \"c81f5d58-c5c7-46c1-b832-c550b092f02b\") " Oct 31 05:13:25.831576 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:25.830634 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3f9d0a0-a106-48bb-b6f2-a980c3aff4e7-tmp-dir" pod "a3f9d0a0-a106-48bb-b6f2-a980c3aff4e7" (UID: "a3f9d0a0-a106-48bb-b6f2-a980c3aff4e7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:13:25.831576 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:25.830820 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c81f5d58-c5c7-46c1-b832-c550b092f02b-tmp-dir" pod "c81f5d58-c5c7-46c1-b832-c550b092f02b" (UID: "c81f5d58-c5c7-46c1-b832-c550b092f02b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:13:25.831576 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:25.831012 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3f9d0a0-a106-48bb-b6f2-a980c3aff4e7-model-cache" pod "a3f9d0a0-a106-48bb-b6f2-a980c3aff4e7" (UID: "a3f9d0a0-a106-48bb-b6f2-a980c3aff4e7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:13:25.831576 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:25.831061 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d298000-b65b-45fd-a1ee-092e47a7eb24-tmp-dir" pod "7d298000-b65b-45fd-a1ee-092e47a7eb24" (UID: "7d298000-b65b-45fd-a1ee-092e47a7eb24"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:13:25.832622 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:25.832360 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c81f5d58-c5c7-46c1-b832-c550b092f02b-home" pod "c81f5d58-c5c7-46c1-b832-c550b092f02b" (UID: "c81f5d58-c5c7-46c1-b832-c550b092f02b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:13:25.833377 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:25.833352 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d298000-b65b-45fd-a1ee-092e47a7eb24-model-cache" pod "7d298000-b65b-45fd-a1ee-092e47a7eb24" (UID: "7d298000-b65b-45fd-a1ee-092e47a7eb24"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:13:25.834546 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:25.834518 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d298000-b65b-45fd-a1ee-092e47a7eb24-home" pod "7d298000-b65b-45fd-a1ee-092e47a7eb24" (UID: "7d298000-b65b-45fd-a1ee-092e47a7eb24"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:13:25.835705 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:25.835554 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d298000-b65b-45fd-a1ee-092e47a7eb24-tls-certs" pod "7d298000-b65b-45fd-a1ee-092e47a7eb24" (UID: "7d298000-b65b-45fd-a1ee-092e47a7eb24"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 05:13:25.835834 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:25.835811 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c81f5d58-c5c7-46c1-b832-c550b092f02b-model-cache" pod "c81f5d58-c5c7-46c1-b832-c550b092f02b" (UID: "c81f5d58-c5c7-46c1-b832-c550b092f02b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:13:25.835999 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:25.835982 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3f9d0a0-a106-48bb-b6f2-a980c3aff4e7-home" pod "a3f9d0a0-a106-48bb-b6f2-a980c3aff4e7" (UID: "a3f9d0a0-a106-48bb-b6f2-a980c3aff4e7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:13:25.836620 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:25.836567 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c81f5d58-c5c7-46c1-b832-c550b092f02b-dshm" pod "c81f5d58-c5c7-46c1-b832-c550b092f02b" (UID: "c81f5d58-c5c7-46c1-b832-c550b092f02b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:13:25.837247 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:25.837222 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d298000-b65b-45fd-a1ee-092e47a7eb24-dshm" pod "7d298000-b65b-45fd-a1ee-092e47a7eb24" (UID: "7d298000-b65b-45fd-a1ee-092e47a7eb24"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:13:25.837388 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:25.837301 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d298000-b65b-45fd-a1ee-092e47a7eb24-kube-api-access-bsxlc" pod "7d298000-b65b-45fd-a1ee-092e47a7eb24" (UID: "7d298000-b65b-45fd-a1ee-092e47a7eb24"). InnerVolumeSpecName "kube-api-access-bsxlc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:13:25.837852 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:25.837828 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3f9d0a0-a106-48bb-b6f2-a980c3aff4e7-kube-api-access-dvsj5" pod "a3f9d0a0-a106-48bb-b6f2-a980c3aff4e7" (UID: "a3f9d0a0-a106-48bb-b6f2-a980c3aff4e7"). InnerVolumeSpecName "kube-api-access-dvsj5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:13:25.838677 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:25.838656 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3f9d0a0-a106-48bb-b6f2-a980c3aff4e7-tls-certs" pod "a3f9d0a0-a106-48bb-b6f2-a980c3aff4e7" (UID: "a3f9d0a0-a106-48bb-b6f2-a980c3aff4e7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 05:13:25.838834 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:25.838810 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3f9d0a0-a106-48bb-b6f2-a980c3aff4e7-dshm" pod "a3f9d0a0-a106-48bb-b6f2-a980c3aff4e7" (UID: "a3f9d0a0-a106-48bb-b6f2-a980c3aff4e7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:13:25.839123 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:25.839086 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c81f5d58-c5c7-46c1-b832-c550b092f02b-tls-certs" pod "c81f5d58-c5c7-46c1-b832-c550b092f02b" (UID: "c81f5d58-c5c7-46c1-b832-c550b092f02b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 05:13:25.840087 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:25.840066 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c81f5d58-c5c7-46c1-b832-c550b092f02b-kube-api-access-sswx2" pod "c81f5d58-c5c7-46c1-b832-c550b092f02b" (UID: "c81f5d58-c5c7-46c1-b832-c550b092f02b"). InnerVolumeSpecName "kube-api-access-sswx2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:13:25.876056 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:25.876014 4376 generic.go:356] "Generic (PLEG): container finished" podID="7d298000-b65b-45fd-a1ee-092e47a7eb24" containerID="c6f1417f936622a50af2a4c91f291b31122a0958bfaa10f70676fdbb15c3e4b2" exitCode=0 Oct 31 05:13:25.876237 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:25.876066 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-09a20a/distinct-w2-09a20a-kserve-76557f6448-l8d69" Oct 31 05:13:25.876237 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:25.876065 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-09a20a/distinct-w2-09a20a-kserve-76557f6448-l8d69" event={"ID":"7d298000-b65b-45fd-a1ee-092e47a7eb24","Type":"ContainerDied","Data":"c6f1417f936622a50af2a4c91f291b31122a0958bfaa10f70676fdbb15c3e4b2"} Oct 31 05:13:25.876237 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:25.876199 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-09a20a/distinct-w2-09a20a-kserve-76557f6448-l8d69" event={"ID":"7d298000-b65b-45fd-a1ee-092e47a7eb24","Type":"ContainerDied","Data":"3dac9adf2b9b205a6c67b610fbd8b11348dd2b08dd8632743c7a1d138db8a9ac"} Oct 31 05:13:25.876237 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:25.876226 4376 scope.go:122] "RemoveContainer" containerID="c6f1417f936622a50af2a4c91f291b31122a0958bfaa10f70676fdbb15c3e4b2" Oct 31 05:13:25.881688 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:25.877983 4376 generic.go:356] "Generic (PLEG): container finished" podID="c81f5d58-c5c7-46c1-b832-c550b092f02b" containerID="0b2f66ca1a9f304890933a12fda4dc76a29093853423773883f4866b86978ffd" exitCode=0 Oct 31 05:13:25.881688 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:25.878036 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-09a20a/distinct-2-w2-09a20a-kserve-779d648d75-mshl7" Oct 31 05:13:25.881688 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:25.878066 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-09a20a/distinct-2-w2-09a20a-kserve-779d648d75-mshl7" event={"ID":"c81f5d58-c5c7-46c1-b832-c550b092f02b","Type":"ContainerDied","Data":"0b2f66ca1a9f304890933a12fda4dc76a29093853423773883f4866b86978ffd"} Oct 31 05:13:25.881688 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:25.878115 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-09a20a/distinct-2-w2-09a20a-kserve-779d648d75-mshl7" event={"ID":"c81f5d58-c5c7-46c1-b832-c550b092f02b","Type":"ContainerDied","Data":"1a909083c986885280be5691dfc0f6a97c4ec21b654e1d93a5ab3a37ad209848"} Oct 31 05:13:25.889450 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:25.889420 4376 generic.go:356] "Generic (PLEG): container finished" podID="a3f9d0a0-a106-48bb-b6f2-a980c3aff4e7" containerID="0ecb5f1038cfbba39690602f96218fc175fbff09c119a5928389d7fd428ac72d" exitCode=0 Oct 31 05:13:25.889570 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:25.889466 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-09a20a/premium-w2-09a20a-kserve-57c45db74f-s8glf" Oct 31 05:13:25.889634 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:25.889565 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-09a20a/premium-w2-09a20a-kserve-57c45db74f-s8glf" event={"ID":"a3f9d0a0-a106-48bb-b6f2-a980c3aff4e7","Type":"ContainerDied","Data":"0ecb5f1038cfbba39690602f96218fc175fbff09c119a5928389d7fd428ac72d"} Oct 31 05:13:25.889634 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:25.889600 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-09a20a/premium-w2-09a20a-kserve-57c45db74f-s8glf" event={"ID":"a3f9d0a0-a106-48bb-b6f2-a980c3aff4e7","Type":"ContainerDied","Data":"0c11a845978de64293a2fcc20dabe52f0c11773ca6c699862471f7bf28c5d440"} Oct 31 05:13:25.892066 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:25.892048 4376 scope.go:122] "RemoveContainer" containerID="c6f1417f936622a50af2a4c91f291b31122a0958bfaa10f70676fdbb15c3e4b2" Oct 31 05:13:25.892598 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:13:25.892380 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6f1417f936622a50af2a4c91f291b31122a0958bfaa10f70676fdbb15c3e4b2\": container with ID starting with c6f1417f936622a50af2a4c91f291b31122a0958bfaa10f70676fdbb15c3e4b2 not found: ID does not exist" containerID="c6f1417f936622a50af2a4c91f291b31122a0958bfaa10f70676fdbb15c3e4b2" Oct 31 05:13:25.892598 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:25.892433 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6f1417f936622a50af2a4c91f291b31122a0958bfaa10f70676fdbb15c3e4b2"} err="failed to get container status \"c6f1417f936622a50af2a4c91f291b31122a0958bfaa10f70676fdbb15c3e4b2\": rpc error: code = NotFound desc = could not find container \"c6f1417f936622a50af2a4c91f291b31122a0958bfaa10f70676fdbb15c3e4b2\": container with ID starting with c6f1417f936622a50af2a4c91f291b31122a0958bfaa10f70676fdbb15c3e4b2 not found: ID does not exist" Oct 31 05:13:25.892598 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:25.892458 4376 scope.go:122] "RemoveContainer" containerID="0b2f66ca1a9f304890933a12fda4dc76a29093853423773883f4866b86978ffd" Oct 31 05:13:25.905901 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:25.905873 4376 scope.go:122] "RemoveContainer" containerID="0b2f66ca1a9f304890933a12fda4dc76a29093853423773883f4866b86978ffd" Oct 31 05:13:25.906835 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:13:25.906814 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b2f66ca1a9f304890933a12fda4dc76a29093853423773883f4866b86978ffd\": container with ID starting with 0b2f66ca1a9f304890933a12fda4dc76a29093853423773883f4866b86978ffd not found: ID does not exist" containerID="0b2f66ca1a9f304890933a12fda4dc76a29093853423773883f4866b86978ffd" Oct 31 05:13:25.906922 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:25.906847 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b2f66ca1a9f304890933a12fda4dc76a29093853423773883f4866b86978ffd"} err="failed to get container status \"0b2f66ca1a9f304890933a12fda4dc76a29093853423773883f4866b86978ffd\": rpc error: code = NotFound desc = could not find container \"0b2f66ca1a9f304890933a12fda4dc76a29093853423773883f4866b86978ffd\": container with ID starting with 0b2f66ca1a9f304890933a12fda4dc76a29093853423773883f4866b86978ffd not found: ID does not exist" Oct 31 05:13:25.906922 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:25.906873 4376 scope.go:122] "RemoveContainer" containerID="0ecb5f1038cfbba39690602f96218fc175fbff09c119a5928389d7fd428ac72d" Oct 31 05:13:25.922444 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:25.922417 4376 scope.go:122] "RemoveContainer" containerID="0ecb5f1038cfbba39690602f96218fc175fbff09c119a5928389d7fd428ac72d" Oct 31 05:13:25.924735 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:13:25.924705 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ecb5f1038cfbba39690602f96218fc175fbff09c119a5928389d7fd428ac72d\": container with ID starting with 0ecb5f1038cfbba39690602f96218fc175fbff09c119a5928389d7fd428ac72d not found: ID does not exist" containerID="0ecb5f1038cfbba39690602f96218fc175fbff09c119a5928389d7fd428ac72d" Oct 31 05:13:25.924892 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:25.924749 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ecb5f1038cfbba39690602f96218fc175fbff09c119a5928389d7fd428ac72d"} err="failed to get container status \"0ecb5f1038cfbba39690602f96218fc175fbff09c119a5928389d7fd428ac72d\": rpc error: code = NotFound desc = could not find container \"0ecb5f1038cfbba39690602f96218fc175fbff09c119a5928389d7fd428ac72d\": container with ID starting with 0ecb5f1038cfbba39690602f96218fc175fbff09c119a5928389d7fd428ac72d not found: ID does not exist" Oct 31 05:13:25.931762 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:25.931732 4376 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c81f5d58-c5c7-46c1-b832-c550b092f02b-tmp-dir\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:13:25.931904 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:25.931764 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bsxlc\" (UniqueName: \"kubernetes.io/projected/7d298000-b65b-45fd-a1ee-092e47a7eb24-kube-api-access-bsxlc\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:13:25.931904 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:25.931779 4376 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d298000-b65b-45fd-a1ee-092e47a7eb24-home\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:13:25.931904 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:25.931793 4376 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c81f5d58-c5c7-46c1-b832-c550b092f02b-model-cache\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:13:25.931904 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:25.931804 4376 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3f9d0a0-a106-48bb-b6f2-a980c3aff4e7-dshm\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:13:25.931904 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:25.931817 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sswx2\" (UniqueName: \"kubernetes.io/projected/c81f5d58-c5c7-46c1-b832-c550b092f02b-kube-api-access-sswx2\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:13:25.931904 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:25.931829 4376 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d298000-b65b-45fd-a1ee-092e47a7eb24-tmp-dir\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:13:25.931904 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:25.931842 4376 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3f9d0a0-a106-48bb-b6f2-a980c3aff4e7-home\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:13:25.931904 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:25.931856 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvsj5\" (UniqueName: \"kubernetes.io/projected/a3f9d0a0-a106-48bb-b6f2-a980c3aff4e7-kube-api-access-dvsj5\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:13:25.931904 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:25.931868 4376 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d298000-b65b-45fd-a1ee-092e47a7eb24-dshm\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:13:25.931904 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:25.931879 4376 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c81f5d58-c5c7-46c1-b832-c550b092f02b-home\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:13:25.931904 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:25.931890 4376 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3f9d0a0-a106-48bb-b6f2-a980c3aff4e7-tmp-dir\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:13:25.931904 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:25.931901 4376 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c81f5d58-c5c7-46c1-b832-c550b092f02b-dshm\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:13:25.932537 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:25.931913 4376 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d298000-b65b-45fd-a1ee-092e47a7eb24-model-cache\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:13:25.932537 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:25.931925 4376 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d298000-b65b-45fd-a1ee-092e47a7eb24-tls-certs\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:13:25.932537 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:25.931937 4376 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c81f5d58-c5c7-46c1-b832-c550b092f02b-tls-certs\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:13:25.932537 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:25.931950 4376 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3f9d0a0-a106-48bb-b6f2-a980c3aff4e7-model-cache\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:13:25.932537 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:25.931962 4376 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3f9d0a0-a106-48bb-b6f2-a980c3aff4e7-tls-certs\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:13:25.971439 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:25.971405 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-09a20a/premium-w2-09a20a-kserve-57c45db74f-s8glf"] Oct 31 05:13:25.972716 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:25.972313 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-09a20a/premium-w2-09a20a-kserve-57c45db74f-s8glf"] Oct 31 05:13:26.010387 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:26.010292 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-09a20a/distinct-w2-09a20a-kserve-76557f6448-l8d69"] Oct 31 05:13:26.013732 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:26.013708 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-09a20a/distinct-w2-09a20a-kserve-76557f6448-l8d69"] Oct 31 05:13:26.053452 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:26.053415 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-09a20a/distinct-2-w2-09a20a-kserve-779d648d75-mshl7"] Oct 31 05:13:26.057549 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:26.057522 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-09a20a/distinct-2-w2-09a20a-kserve-779d648d75-mshl7"] Oct 31 05:13:26.444592 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:26.444561 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d298000-b65b-45fd-a1ee-092e47a7eb24" path="/var/lib/kubelet/pods/7d298000-b65b-45fd-a1ee-092e47a7eb24/volumes" Oct 31 05:13:26.445031 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:26.445016 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3f9d0a0-a106-48bb-b6f2-a980c3aff4e7" path="/var/lib/kubelet/pods/a3f9d0a0-a106-48bb-b6f2-a980c3aff4e7/volumes" Oct 31 05:13:26.445418 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:26.445405 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c81f5d58-c5c7-46c1-b832-c550b092f02b" path="/var/lib/kubelet/pods/c81f5d58-c5c7-46c1-b832-c550b092f02b/volumes" Oct 31 05:13:27.156994 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:27.156931 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-09a20a-b4f7d6d49-2lp58" podUID="df1df0a3-2701-4f32-a6f7-3f7ce203cfee" containerName="payload-processing" containerID="cri-o://85e13b0c3148e086c5aa422ad780b541df3510fb08bfb98b9a3ba3db31fac305" gracePeriod=25 Oct 31 05:13:27.183697 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:27.183657 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-09a20a-5bf985cbc9-scqzr" podUID="cb75359a-8880-406f-955c-5a3fe0fb796c" containerName="payload-pre-processing" containerID="cri-o://ebedea044c8eaa87c68fd97ebb83131a5fe25f4f04c65881135cdf40f0894b32" gracePeriod=25 Oct 31 05:13:29.231145 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:29.231105 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-439989-5d779f9865-lc48d" podUID="0a252fc6-14bf-4524-907a-f8ee294acda5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 05:13:33.113072 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:33.113033 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-09a20a-b4f7d6d49-2lp58" podUID="df1df0a3-2701-4f32-a6f7-3f7ce203cfee" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 05:13:33.113489 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:33.113033 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-09a20a-5bf985cbc9-scqzr" podUID="cb75359a-8880-406f-955c-5a3fe0fb796c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 05:13:35.318675 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:35.318644 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-439989-5d779f9865-lc48d" Oct 31 05:13:35.420352 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:35.419838 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a252fc6-14bf-4524-907a-f8ee294acda5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a252fc6-14bf-4524-907a-f8ee294acda5-plugins-config-volume\") pod \"0a252fc6-14bf-4524-907a-f8ee294acda5\" (UID: \"0a252fc6-14bf-4524-907a-f8ee294acda5\") " Oct 31 05:13:35.420352 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:35.419956 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a252fc6-14bf-4524-907a-f8ee294acda5-kube-api-access-pzl69\" (UniqueName: \"kubernetes.io/projected/0a252fc6-14bf-4524-907a-f8ee294acda5-kube-api-access-pzl69\") pod \"0a252fc6-14bf-4524-907a-f8ee294acda5\" (UID: \"0a252fc6-14bf-4524-907a-f8ee294acda5\") " Oct 31 05:13:35.442404 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:35.422145 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a252fc6-14bf-4524-907a-f8ee294acda5-plugins-config-volume" pod "0a252fc6-14bf-4524-907a-f8ee294acda5" (UID: "0a252fc6-14bf-4524-907a-f8ee294acda5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:13:35.442769 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:35.442530 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a252fc6-14bf-4524-907a-f8ee294acda5-kube-api-access-pzl69" pod "0a252fc6-14bf-4524-907a-f8ee294acda5" (UID: "0a252fc6-14bf-4524-907a-f8ee294acda5"). InnerVolumeSpecName "kube-api-access-pzl69". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:13:35.521717 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:35.521686 4376 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a252fc6-14bf-4524-907a-f8ee294acda5-plugins-config-volume\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:13:35.521717 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:35.521713 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzl69\" (UniqueName: \"kubernetes.io/projected/0a252fc6-14bf-4524-907a-f8ee294acda5-kube-api-access-pzl69\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:13:35.958763 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:35.958725 4376 generic.go:356] "Generic (PLEG): container finished" podID="0a252fc6-14bf-4524-907a-f8ee294acda5" containerID="f676360a316ac36141aebd4e14dc1b541d41b2d44de27e267b4315a9b52ac0d7" exitCode=0 Oct 31 05:13:35.958963 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:35.958770 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-439989-5d779f9865-lc48d" Oct 31 05:13:35.958963 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:35.958808 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-439989-5d779f9865-lc48d" event={"ID":"0a252fc6-14bf-4524-907a-f8ee294acda5","Type":"ContainerDied","Data":"f676360a316ac36141aebd4e14dc1b541d41b2d44de27e267b4315a9b52ac0d7"} Oct 31 05:13:35.958963 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:35.958856 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-439989-5d779f9865-lc48d" event={"ID":"0a252fc6-14bf-4524-907a-f8ee294acda5","Type":"ContainerDied","Data":"309925dfc201351fd53a96db994451a368e43d2fbac67a41944c12ed9965413f"} Oct 31 05:13:35.958963 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:35.958878 4376 scope.go:122] "RemoveContainer" containerID="f676360a316ac36141aebd4e14dc1b541d41b2d44de27e267b4315a9b52ac0d7" Oct 31 05:13:35.969627 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:35.969510 4376 scope.go:122] "RemoveContainer" containerID="f676360a316ac36141aebd4e14dc1b541d41b2d44de27e267b4315a9b52ac0d7" Oct 31 05:13:35.969879 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:13:35.969851 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f676360a316ac36141aebd4e14dc1b541d41b2d44de27e267b4315a9b52ac0d7\": container with ID starting with f676360a316ac36141aebd4e14dc1b541d41b2d44de27e267b4315a9b52ac0d7 not found: ID does not exist" containerID="f676360a316ac36141aebd4e14dc1b541d41b2d44de27e267b4315a9b52ac0d7" Oct 31 05:13:35.969879 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:35.969887 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f676360a316ac36141aebd4e14dc1b541d41b2d44de27e267b4315a9b52ac0d7"} err="failed to get container status \"f676360a316ac36141aebd4e14dc1b541d41b2d44de27e267b4315a9b52ac0d7\": rpc error: code = NotFound desc = could not find container \"f676360a316ac36141aebd4e14dc1b541d41b2d44de27e267b4315a9b52ac0d7\": container with ID starting with f676360a316ac36141aebd4e14dc1b541d41b2d44de27e267b4315a9b52ac0d7 not found: ID does not exist" Oct 31 05:13:36.017778 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:36.017748 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-439989-5d779f9865-lc48d"] Oct 31 05:13:36.031560 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:36.031533 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-439989-5d779f9865-lc48d"] Oct 31 05:13:36.444467 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:36.444436 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a252fc6-14bf-4524-907a-f8ee294acda5" path="/var/lib/kubelet/pods/0a252fc6-14bf-4524-907a-f8ee294acda5/volumes" Oct 31 05:13:37.892273 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:37.892236 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 05:13:41.001997 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:13:41.001955 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 05:13:43.113527 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:43.113491 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-09a20a-b4f7d6d49-2lp58" podUID="df1df0a3-2701-4f32-a6f7-3f7ce203cfee" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 05:13:43.114038 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:43.113863 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-09a20a-5bf985cbc9-scqzr" podUID="cb75359a-8880-406f-955c-5a3fe0fb796c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 05:13:44.240012 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:44.239970 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-654741-785568f87c-8k8tt"] Oct 31 05:13:44.301418 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:44.301371 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-654741-5dbd945cf9-wrd4x"] Oct 31 05:13:47.455107 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:47.455077 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-09a20a-b4f7d6d49-2lp58" Oct 31 05:13:47.484823 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:47.484799 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-09a20a-5bf985cbc9-scqzr" Oct 31 05:13:47.525224 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:47.525198 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df1df0a3-2701-4f32-a6f7-3f7ce203cfee-kube-api-access-jddmw\" (UniqueName: \"kubernetes.io/projected/df1df0a3-2701-4f32-a6f7-3f7ce203cfee-kube-api-access-jddmw\") pod \"df1df0a3-2701-4f32-a6f7-3f7ce203cfee\" (UID: \"df1df0a3-2701-4f32-a6f7-3f7ce203cfee\") " Oct 31 05:13:47.525417 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:47.525313 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df1df0a3-2701-4f32-a6f7-3f7ce203cfee-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df1df0a3-2701-4f32-a6f7-3f7ce203cfee-plugins-config-volume\") pod \"df1df0a3-2701-4f32-a6f7-3f7ce203cfee\" (UID: \"df1df0a3-2701-4f32-a6f7-3f7ce203cfee\") " Oct 31 05:13:47.525678 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:47.525648 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df1df0a3-2701-4f32-a6f7-3f7ce203cfee-plugins-config-volume" pod "df1df0a3-2701-4f32-a6f7-3f7ce203cfee" (UID: "df1df0a3-2701-4f32-a6f7-3f7ce203cfee"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:13:47.526962 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:47.526712 4376 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df1df0a3-2701-4f32-a6f7-3f7ce203cfee-plugins-config-volume\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:13:47.527786 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:47.527765 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df1df0a3-2701-4f32-a6f7-3f7ce203cfee-kube-api-access-jddmw" pod "df1df0a3-2701-4f32-a6f7-3f7ce203cfee" (UID: "df1df0a3-2701-4f32-a6f7-3f7ce203cfee"). InnerVolumeSpecName "kube-api-access-jddmw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:13:47.627729 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:47.627687 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb75359a-8880-406f-955c-5a3fe0fb796c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb75359a-8880-406f-955c-5a3fe0fb796c-plugins-config-volume\") pod \"cb75359a-8880-406f-955c-5a3fe0fb796c\" (UID: \"cb75359a-8880-406f-955c-5a3fe0fb796c\") " Oct 31 05:13:47.627912 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:47.627757 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb75359a-8880-406f-955c-5a3fe0fb796c-kube-api-access-zmjhn\" (UniqueName: \"kubernetes.io/projected/cb75359a-8880-406f-955c-5a3fe0fb796c-kube-api-access-zmjhn\") pod \"cb75359a-8880-406f-955c-5a3fe0fb796c\" (UID: \"cb75359a-8880-406f-955c-5a3fe0fb796c\") " Oct 31 05:13:47.628096 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:47.628085 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jddmw\" (UniqueName: \"kubernetes.io/projected/df1df0a3-2701-4f32-a6f7-3f7ce203cfee-kube-api-access-jddmw\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:13:47.628142 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:47.628114 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb75359a-8880-406f-955c-5a3fe0fb796c-plugins-config-volume" pod "cb75359a-8880-406f-955c-5a3fe0fb796c" (UID: "cb75359a-8880-406f-955c-5a3fe0fb796c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:13:47.629830 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:47.629809 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb75359a-8880-406f-955c-5a3fe0fb796c-kube-api-access-zmjhn" pod "cb75359a-8880-406f-955c-5a3fe0fb796c" (UID: "cb75359a-8880-406f-955c-5a3fe0fb796c"). InnerVolumeSpecName "kube-api-access-zmjhn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:13:47.729265 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:47.729191 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmjhn\" (UniqueName: \"kubernetes.io/projected/cb75359a-8880-406f-955c-5a3fe0fb796c-kube-api-access-zmjhn\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:13:47.729265 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:47.729218 4376 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb75359a-8880-406f-955c-5a3fe0fb796c-plugins-config-volume\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:13:48.110661 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:48.110574 4376 generic.go:356] "Generic (PLEG): container finished" podID="cb75359a-8880-406f-955c-5a3fe0fb796c" containerID="ebedea044c8eaa87c68fd97ebb83131a5fe25f4f04c65881135cdf40f0894b32" exitCode=0 Oct 31 05:13:48.111035 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:48.111008 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-09a20a-5bf985cbc9-scqzr" event={"ID":"cb75359a-8880-406f-955c-5a3fe0fb796c","Type":"ContainerDied","Data":"ebedea044c8eaa87c68fd97ebb83131a5fe25f4f04c65881135cdf40f0894b32"} Oct 31 05:13:48.111219 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:48.111200 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-09a20a-5bf985cbc9-scqzr" event={"ID":"cb75359a-8880-406f-955c-5a3fe0fb796c","Type":"ContainerDied","Data":"e7776281bc2c028430ed8cb58519eb60f2b838ac0fa9062671dac7eb53fc5b1f"} Oct 31 05:13:48.111361 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:48.111346 4376 scope.go:122] "RemoveContainer" containerID="ebedea044c8eaa87c68fd97ebb83131a5fe25f4f04c65881135cdf40f0894b32" Oct 31 05:13:48.111783 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:48.111622 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-09a20a-5bf985cbc9-scqzr" Oct 31 05:13:48.120279 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:48.120241 4376 generic.go:356] "Generic (PLEG): container finished" podID="df1df0a3-2701-4f32-a6f7-3f7ce203cfee" containerID="85e13b0c3148e086c5aa422ad780b541df3510fb08bfb98b9a3ba3db31fac305" exitCode=0 Oct 31 05:13:48.120449 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:48.120307 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-09a20a-b4f7d6d49-2lp58" event={"ID":"df1df0a3-2701-4f32-a6f7-3f7ce203cfee","Type":"ContainerDied","Data":"85e13b0c3148e086c5aa422ad780b541df3510fb08bfb98b9a3ba3db31fac305"} Oct 31 05:13:48.120449 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:48.120372 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-09a20a-b4f7d6d49-2lp58" event={"ID":"df1df0a3-2701-4f32-a6f7-3f7ce203cfee","Type":"ContainerDied","Data":"b5589f6f6b2d9a7d696d51b7895a2713e777eac1fd617ea3d87049f9a7f7ec87"} Oct 31 05:13:48.120449 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:48.120448 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-09a20a-b4f7d6d49-2lp58" Oct 31 05:13:48.128596 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:48.128573 4376 scope.go:122] "RemoveContainer" containerID="ebedea044c8eaa87c68fd97ebb83131a5fe25f4f04c65881135cdf40f0894b32" Oct 31 05:13:48.129119 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:13:48.129060 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebedea044c8eaa87c68fd97ebb83131a5fe25f4f04c65881135cdf40f0894b32\": container with ID starting with ebedea044c8eaa87c68fd97ebb83131a5fe25f4f04c65881135cdf40f0894b32 not found: ID does not exist" containerID="ebedea044c8eaa87c68fd97ebb83131a5fe25f4f04c65881135cdf40f0894b32" Oct 31 05:13:48.129119 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:48.129100 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebedea044c8eaa87c68fd97ebb83131a5fe25f4f04c65881135cdf40f0894b32"} err="failed to get container status \"ebedea044c8eaa87c68fd97ebb83131a5fe25f4f04c65881135cdf40f0894b32\": rpc error: code = NotFound desc = could not find container \"ebedea044c8eaa87c68fd97ebb83131a5fe25f4f04c65881135cdf40f0894b32\": container with ID starting with ebedea044c8eaa87c68fd97ebb83131a5fe25f4f04c65881135cdf40f0894b32 not found: ID does not exist" Oct 31 05:13:48.129422 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:48.129123 4376 scope.go:122] "RemoveContainer" containerID="85e13b0c3148e086c5aa422ad780b541df3510fb08bfb98b9a3ba3db31fac305" Oct 31 05:13:48.139343 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:48.139311 4376 scope.go:122] "RemoveContainer" containerID="85e13b0c3148e086c5aa422ad780b541df3510fb08bfb98b9a3ba3db31fac305" Oct 31 05:13:48.139636 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:13:48.139604 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85e13b0c3148e086c5aa422ad780b541df3510fb08bfb98b9a3ba3db31fac305\": container with ID starting with 85e13b0c3148e086c5aa422ad780b541df3510fb08bfb98b9a3ba3db31fac305 not found: ID does not exist" containerID="85e13b0c3148e086c5aa422ad780b541df3510fb08bfb98b9a3ba3db31fac305" Oct 31 05:13:48.139755 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:48.139642 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85e13b0c3148e086c5aa422ad780b541df3510fb08bfb98b9a3ba3db31fac305"} err="failed to get container status \"85e13b0c3148e086c5aa422ad780b541df3510fb08bfb98b9a3ba3db31fac305\": rpc error: code = NotFound desc = could not find container \"85e13b0c3148e086c5aa422ad780b541df3510fb08bfb98b9a3ba3db31fac305\": container with ID starting with 85e13b0c3148e086c5aa422ad780b541df3510fb08bfb98b9a3ba3db31fac305 not found: ID does not exist" Oct 31 05:13:48.173895 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:48.173865 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-09a20a-b4f7d6d49-2lp58"] Oct 31 05:13:48.182374 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:48.182343 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-09a20a-b4f7d6d49-2lp58"] Oct 31 05:13:48.241533 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:48.241486 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-09a20a-5bf985cbc9-scqzr"] Oct 31 05:13:48.246628 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:48.246593 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-09a20a-5bf985cbc9-scqzr"] Oct 31 05:13:48.444962 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:48.444929 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb75359a-8880-406f-955c-5a3fe0fb796c" path="/var/lib/kubelet/pods/cb75359a-8880-406f-955c-5a3fe0fb796c/volumes" Oct 31 05:13:48.445339 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:48.445316 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df1df0a3-2701-4f32-a6f7-3f7ce203cfee" path="/var/lib/kubelet/pods/df1df0a3-2701-4f32-a6f7-3f7ce203cfee/volumes" Oct 31 05:13:49.241232 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:49.241160 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-654741-785568f87c-8k8tt" podUID="2755051f-49ce-4ffe-8c54-0ae241bd45c2" containerName="payload-processing" containerID="cri-o://ec39282578e488f88ddb62e156380467b18ca902f9262892d32aa84ec924ca21" gracePeriod=25 Oct 31 05:13:49.302140 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:49.302096 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-654741-5dbd945cf9-wrd4x" podUID="f3b9ba37-7625-40d7-be02-793a54367260" containerName="payload-pre-processing" containerID="cri-o://a9ba12a8c50a9828306739d887f380ca0a746cae93dca61421acb423f3ed4041" gracePeriod=25 Oct 31 05:13:50.234180 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:50.234139 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-654741-785568f87c-8k8tt" podUID="2755051f-49ce-4ffe-8c54-0ae241bd45c2" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 05:13:50.234374 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:50.234151 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-654741-5dbd945cf9-wrd4x" podUID="f3b9ba37-7625-40d7-be02-793a54367260" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 05:13:55.067448 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:55.067352 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-654741-6d9cc779bc-wxdnw"] Oct 31 05:13:55.074781 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:55.074732 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-654741-6d9cc779bc-wxdnw" podUID="84000b34-2ffa-4914-97f3-73bafc243ec5" containerName="maas-api" containerID="cri-o://594800ec68e4702c2aece86edab37f7e0fd8a860c89daccceac4d6a698b9acd2" gracePeriod=30 Oct 31 05:13:55.424424 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:55.424396 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-654741-6d9cc779bc-wxdnw" Oct 31 05:13:55.576352 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:55.576157 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84000b34-2ffa-4914-97f3-73bafc243ec5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84000b34-2ffa-4914-97f3-73bafc243ec5-maas-api-tls\") pod \"84000b34-2ffa-4914-97f3-73bafc243ec5\" (UID: \"84000b34-2ffa-4914-97f3-73bafc243ec5\") " Oct 31 05:13:55.576352 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:55.576230 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84000b34-2ffa-4914-97f3-73bafc243ec5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84000b34-2ffa-4914-97f3-73bafc243ec5-openshift-service-ca\") pod \"84000b34-2ffa-4914-97f3-73bafc243ec5\" (UID: \"84000b34-2ffa-4914-97f3-73bafc243ec5\") " Oct 31 05:13:55.576352 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:55.576293 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84000b34-2ffa-4914-97f3-73bafc243ec5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/84000b34-2ffa-4914-97f3-73bafc243ec5-metrics-certs\") pod \"84000b34-2ffa-4914-97f3-73bafc243ec5\" (UID: \"84000b34-2ffa-4914-97f3-73bafc243ec5\") " Oct 31 05:13:55.576352 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:55.576355 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84000b34-2ffa-4914-97f3-73bafc243ec5-kube-api-access-t6f4q\" (UniqueName: \"kubernetes.io/projected/84000b34-2ffa-4914-97f3-73bafc243ec5-kube-api-access-t6f4q\") pod \"84000b34-2ffa-4914-97f3-73bafc243ec5\" (UID: \"84000b34-2ffa-4914-97f3-73bafc243ec5\") " Oct 31 05:13:55.583475 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:55.583440 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84000b34-2ffa-4914-97f3-73bafc243ec5-openshift-service-ca" pod "84000b34-2ffa-4914-97f3-73bafc243ec5" (UID: "84000b34-2ffa-4914-97f3-73bafc243ec5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:13:55.603586 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:55.600684 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84000b34-2ffa-4914-97f3-73bafc243ec5-kube-api-access-t6f4q" pod "84000b34-2ffa-4914-97f3-73bafc243ec5" (UID: "84000b34-2ffa-4914-97f3-73bafc243ec5"). InnerVolumeSpecName "kube-api-access-t6f4q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:13:55.603586 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:55.601002 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84000b34-2ffa-4914-97f3-73bafc243ec5-metrics-certs" pod "84000b34-2ffa-4914-97f3-73bafc243ec5" (UID: "84000b34-2ffa-4914-97f3-73bafc243ec5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 05:13:55.603586 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:55.602253 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84000b34-2ffa-4914-97f3-73bafc243ec5-maas-api-tls" pod "84000b34-2ffa-4914-97f3-73bafc243ec5" (UID: "84000b34-2ffa-4914-97f3-73bafc243ec5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 05:13:55.679226 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:55.679146 4376 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84000b34-2ffa-4914-97f3-73bafc243ec5-maas-api-tls\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:13:55.679226 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:55.679174 4376 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84000b34-2ffa-4914-97f3-73bafc243ec5-openshift-service-ca\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:13:55.679226 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:55.679183 4376 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84000b34-2ffa-4914-97f3-73bafc243ec5-metrics-certs\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:13:55.679226 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:55.679192 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6f4q\" (UniqueName: \"kubernetes.io/projected/84000b34-2ffa-4914-97f3-73bafc243ec5-kube-api-access-t6f4q\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:13:56.173958 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:56.173925 4376 generic.go:356] "Generic (PLEG): container finished" podID="84000b34-2ffa-4914-97f3-73bafc243ec5" containerID="594800ec68e4702c2aece86edab37f7e0fd8a860c89daccceac4d6a698b9acd2" exitCode=0 Oct 31 05:13:56.174399 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:56.173977 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-654741-6d9cc779bc-wxdnw" Oct 31 05:13:56.174399 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:56.174008 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-654741-6d9cc779bc-wxdnw" event={"ID":"84000b34-2ffa-4914-97f3-73bafc243ec5","Type":"ContainerDied","Data":"594800ec68e4702c2aece86edab37f7e0fd8a860c89daccceac4d6a698b9acd2"} Oct 31 05:13:56.174399 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:56.174060 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-654741-6d9cc779bc-wxdnw" event={"ID":"84000b34-2ffa-4914-97f3-73bafc243ec5","Type":"ContainerDied","Data":"fb139508a7bfcfa79bc390a5099e941e0ea8733026950ffa331a38481b8bfd58"} Oct 31 05:13:56.174399 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:56.174081 4376 scope.go:122] "RemoveContainer" containerID="594800ec68e4702c2aece86edab37f7e0fd8a860c89daccceac4d6a698b9acd2" Oct 31 05:13:56.184440 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:56.184424 4376 scope.go:122] "RemoveContainer" containerID="594800ec68e4702c2aece86edab37f7e0fd8a860c89daccceac4d6a698b9acd2" Oct 31 05:13:56.184781 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:13:56.184752 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"594800ec68e4702c2aece86edab37f7e0fd8a860c89daccceac4d6a698b9acd2\": container with ID starting with 594800ec68e4702c2aece86edab37f7e0fd8a860c89daccceac4d6a698b9acd2 not found: ID does not exist" containerID="594800ec68e4702c2aece86edab37f7e0fd8a860c89daccceac4d6a698b9acd2" Oct 31 05:13:56.184862 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:56.184788 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"594800ec68e4702c2aece86edab37f7e0fd8a860c89daccceac4d6a698b9acd2"} err="failed to get container status \"594800ec68e4702c2aece86edab37f7e0fd8a860c89daccceac4d6a698b9acd2\": rpc error: code = NotFound desc = could not find container \"594800ec68e4702c2aece86edab37f7e0fd8a860c89daccceac4d6a698b9acd2\": container with ID starting with 594800ec68e4702c2aece86edab37f7e0fd8a860c89daccceac4d6a698b9acd2 not found: ID does not exist" Oct 31 05:13:56.233232 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:56.233201 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-654741-6d9cc779bc-wxdnw"] Oct 31 05:13:56.247009 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:56.246977 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-654741-6d9cc779bc-wxdnw"] Oct 31 05:13:56.444121 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:13:56.444045 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84000b34-2ffa-4914-97f3-73bafc243ec5" path="/var/lib/kubelet/pods/84000b34-2ffa-4914-97f3-73bafc243ec5/volumes" Oct 31 05:14:00.234308 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:14:00.234219 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-654741-5dbd945cf9-wrd4x" podUID="f3b9ba37-7625-40d7-be02-793a54367260" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 05:14:00.234717 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:14:00.234219 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-654741-785568f87c-8k8tt" podUID="2755051f-49ce-4ffe-8c54-0ae241bd45c2" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 05:14:01.032916 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:14:01.032872 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-654741-openshift-default-8449896d9c-cf4pt"] Oct 31 05:14:01.033196 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:14:01.033167 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-654741-openshift-default-8449896d9c-cf4pt" podUID="32954f51-d716-449a-ba69-ca46851a9ce2" containerName="istio-proxy" containerID="cri-o://1d8a428f018fe1e4348d1ece7c53b66f271eeee3ff675cdcfa0acf34491c573f" gracePeriod=30 Oct 31 05:14:06.249448 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:14:06.249410 4376 generic.go:356] "Generic (PLEG): container finished" podID="32954f51-d716-449a-ba69-ca46851a9ce2" containerID="1d8a428f018fe1e4348d1ece7c53b66f271eeee3ff675cdcfa0acf34491c573f" exitCode=0 Oct 31 05:14:06.249448 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:14:06.249436 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-654741-openshift-default-8449896d9c-cf4pt" event={"ID":"32954f51-d716-449a-ba69-ca46851a9ce2","Type":"ContainerDied","Data":"1d8a428f018fe1e4348d1ece7c53b66f271eeee3ff675cdcfa0acf34491c573f"} Oct 31 05:14:06.250067 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:14:06.249574 4376 scope.go:122] "RemoveContainer" containerID="1e419808a87c9def4ea0621cb542832602a03d85745a696cd6428c97d9fa8b62" Oct 31 05:14:06.308687 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:14:06.308633 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-654741-openshift-default-8449896d9c-cf4pt" Oct 31 05:14:06.348592 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:14:06.348550 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32954f51-d716-449a-ba69-ca46851a9ce2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32954f51-d716-449a-ba69-ca46851a9ce2-istio-envoy\") pod \"32954f51-d716-449a-ba69-ca46851a9ce2\" (UID: \"32954f51-d716-449a-ba69-ca46851a9ce2\") " Oct 31 05:14:06.348749 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:14:06.348618 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/32954f51-d716-449a-ba69-ca46851a9ce2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32954f51-d716-449a-ba69-ca46851a9ce2-istio-podinfo\") pod \"32954f51-d716-449a-ba69-ca46851a9ce2\" (UID: \"32954f51-d716-449a-ba69-ca46851a9ce2\") " Oct 31 05:14:06.348749 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:14:06.348678 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32954f51-d716-449a-ba69-ca46851a9ce2-istio-token\" (UniqueName: \"kubernetes.io/projected/32954f51-d716-449a-ba69-ca46851a9ce2-istio-token\") pod \"32954f51-d716-449a-ba69-ca46851a9ce2\" (UID: \"32954f51-d716-449a-ba69-ca46851a9ce2\") " Oct 31 05:14:06.348749 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:14:06.348727 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32954f51-d716-449a-ba69-ca46851a9ce2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32954f51-d716-449a-ba69-ca46851a9ce2-istio-data\") pod \"32954f51-d716-449a-ba69-ca46851a9ce2\" (UID: \"32954f51-d716-449a-ba69-ca46851a9ce2\") " Oct 31 05:14:06.348952 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:14:06.348752 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32954f51-d716-449a-ba69-ca46851a9ce2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32954f51-d716-449a-ba69-ca46851a9ce2-credential-socket\") pod \"32954f51-d716-449a-ba69-ca46851a9ce2\" (UID: \"32954f51-d716-449a-ba69-ca46851a9ce2\") " Oct 31 05:14:06.348952 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:14:06.348799 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32954f51-d716-449a-ba69-ca46851a9ce2-kube-api-access-jwp2l\" (UniqueName: \"kubernetes.io/projected/32954f51-d716-449a-ba69-ca46851a9ce2-kube-api-access-jwp2l\") pod \"32954f51-d716-449a-ba69-ca46851a9ce2\" (UID: \"32954f51-d716-449a-ba69-ca46851a9ce2\") " Oct 31 05:14:06.348952 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:14:06.348840 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32954f51-d716-449a-ba69-ca46851a9ce2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32954f51-d716-449a-ba69-ca46851a9ce2-workload-socket\") pod \"32954f51-d716-449a-ba69-ca46851a9ce2\" (UID: \"32954f51-d716-449a-ba69-ca46851a9ce2\") " Oct 31 05:14:06.348952 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:14:06.348940 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32954f51-d716-449a-ba69-ca46851a9ce2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32954f51-d716-449a-ba69-ca46851a9ce2-istiod-ca-cert\") pod \"32954f51-d716-449a-ba69-ca46851a9ce2\" (UID: \"32954f51-d716-449a-ba69-ca46851a9ce2\") " Oct 31 05:14:06.349185 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:14:06.348980 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32954f51-d716-449a-ba69-ca46851a9ce2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32954f51-d716-449a-ba69-ca46851a9ce2-workload-certs\") pod \"32954f51-d716-449a-ba69-ca46851a9ce2\" (UID: \"32954f51-d716-449a-ba69-ca46851a9ce2\") " Oct 31 05:14:06.349185 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:14:06.349053 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32954f51-d716-449a-ba69-ca46851a9ce2-istio-data" pod "32954f51-d716-449a-ba69-ca46851a9ce2" (UID: "32954f51-d716-449a-ba69-ca46851a9ce2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:14:06.349185 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:14:06.349079 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32954f51-d716-449a-ba69-ca46851a9ce2-credential-socket" pod "32954f51-d716-449a-ba69-ca46851a9ce2" (UID: "32954f51-d716-449a-ba69-ca46851a9ce2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:14:06.349185 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:14:06.349175 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32954f51-d716-449a-ba69-ca46851a9ce2-workload-socket" pod "32954f51-d716-449a-ba69-ca46851a9ce2" (UID: "32954f51-d716-449a-ba69-ca46851a9ce2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:14:06.349454 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:14:06.349374 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32954f51-d716-449a-ba69-ca46851a9ce2-workload-certs" pod "32954f51-d716-449a-ba69-ca46851a9ce2" (UID: "32954f51-d716-449a-ba69-ca46851a9ce2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:14:06.349454 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:14:06.349431 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32954f51-d716-449a-ba69-ca46851a9ce2-istiod-ca-cert" pod "32954f51-d716-449a-ba69-ca46851a9ce2" (UID: "32954f51-d716-449a-ba69-ca46851a9ce2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:14:06.349633 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:14:06.349617 4376 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32954f51-d716-449a-ba69-ca46851a9ce2-istio-data\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:14:06.349714 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:14:06.349635 4376 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32954f51-d716-449a-ba69-ca46851a9ce2-credential-socket\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:14:06.349714 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:14:06.349655 4376 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32954f51-d716-449a-ba69-ca46851a9ce2-workload-socket\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:14:06.349714 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:14:06.349664 4376 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32954f51-d716-449a-ba69-ca46851a9ce2-istiod-ca-cert\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:14:06.349714 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:14:06.349673 4376 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32954f51-d716-449a-ba69-ca46851a9ce2-workload-certs\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:14:06.351050 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:14:06.351023 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/32954f51-d716-449a-ba69-ca46851a9ce2-istio-podinfo" pod "32954f51-d716-449a-ba69-ca46851a9ce2" (UID: "32954f51-d716-449a-ba69-ca46851a9ce2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 05:14:06.351152 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:14:06.351046 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32954f51-d716-449a-ba69-ca46851a9ce2-istio-token" pod "32954f51-d716-449a-ba69-ca46851a9ce2" (UID: "32954f51-d716-449a-ba69-ca46851a9ce2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:14:06.351246 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:14:06.351229 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32954f51-d716-449a-ba69-ca46851a9ce2-istio-envoy" pod "32954f51-d716-449a-ba69-ca46851a9ce2" (UID: "32954f51-d716-449a-ba69-ca46851a9ce2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:14:06.351287 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:14:06.351270 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32954f51-d716-449a-ba69-ca46851a9ce2-kube-api-access-jwp2l" pod "32954f51-d716-449a-ba69-ca46851a9ce2" (UID: "32954f51-d716-449a-ba69-ca46851a9ce2"). InnerVolumeSpecName "kube-api-access-jwp2l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:14:06.450623 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:14:06.450586 4376 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32954f51-d716-449a-ba69-ca46851a9ce2-istio-envoy\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:14:06.450623 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:14:06.450623 4376 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32954f51-d716-449a-ba69-ca46851a9ce2-istio-podinfo\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:14:06.450906 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:14:06.450637 4376 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32954f51-d716-449a-ba69-ca46851a9ce2-istio-token\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:14:06.450906 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:14:06.450652 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwp2l\" (UniqueName: \"kubernetes.io/projected/32954f51-d716-449a-ba69-ca46851a9ce2-kube-api-access-jwp2l\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:14:07.259161 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:14:07.259122 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-654741-openshift-default-8449896d9c-cf4pt" event={"ID":"32954f51-d716-449a-ba69-ca46851a9ce2","Type":"ContainerDied","Data":"a09d1cb50a03a82f070b9fce636cadf84121549c9828689296b92a9c0a09d32a"} Oct 31 05:14:07.259670 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:14:07.259172 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-654741-openshift-default-8449896d9c-cf4pt" Oct 31 05:14:07.259670 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:14:07.259196 4376 scope.go:122] "RemoveContainer" containerID="1d8a428f018fe1e4348d1ece7c53b66f271eeee3ff675cdcfa0acf34491c573f" Oct 31 05:14:07.380220 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:14:07.380188 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-654741-openshift-default-8449896d9c-cf4pt"] Oct 31 05:14:07.385129 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:14:07.384984 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-654741-openshift-default-8449896d9c-cf4pt"] Oct 31 05:14:08.445058 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:14:08.445021 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32954f51-d716-449a-ba69-ca46851a9ce2" path="/var/lib/kubelet/pods/32954f51-d716-449a-ba69-ca46851a9ce2/volumes" Oct 31 05:14:09.616106 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:14:09.616078 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-654741-785568f87c-8k8tt" Oct 31 05:14:09.619728 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:14:09.619709 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-654741-5dbd945cf9-wrd4x" Oct 31 05:14:09.704572 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:14:09.704348 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2755051f-49ce-4ffe-8c54-0ae241bd45c2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2755051f-49ce-4ffe-8c54-0ae241bd45c2-plugins-config-volume\") pod \"2755051f-49ce-4ffe-8c54-0ae241bd45c2\" (UID: \"2755051f-49ce-4ffe-8c54-0ae241bd45c2\") " Oct 31 05:14:09.709379 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:14:09.705131 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2755051f-49ce-4ffe-8c54-0ae241bd45c2-kube-api-access-w4562\" (UniqueName: \"kubernetes.io/projected/2755051f-49ce-4ffe-8c54-0ae241bd45c2-kube-api-access-w4562\") pod \"2755051f-49ce-4ffe-8c54-0ae241bd45c2\" (UID: \"2755051f-49ce-4ffe-8c54-0ae241bd45c2\") " Oct 31 05:14:09.709379 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:14:09.705190 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2755051f-49ce-4ffe-8c54-0ae241bd45c2-plugins-config-volume" pod "2755051f-49ce-4ffe-8c54-0ae241bd45c2" (UID: "2755051f-49ce-4ffe-8c54-0ae241bd45c2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:14:09.709379 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:14:09.705254 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3b9ba37-7625-40d7-be02-793a54367260-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3b9ba37-7625-40d7-be02-793a54367260-plugins-config-volume\") pod \"f3b9ba37-7625-40d7-be02-793a54367260\" (UID: \"f3b9ba37-7625-40d7-be02-793a54367260\") " Oct 31 05:14:09.709379 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:14:09.705419 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3b9ba37-7625-40d7-be02-793a54367260-kube-api-access-txmq6\" (UniqueName: \"kubernetes.io/projected/f3b9ba37-7625-40d7-be02-793a54367260-kube-api-access-txmq6\") pod \"f3b9ba37-7625-40d7-be02-793a54367260\" (UID: \"f3b9ba37-7625-40d7-be02-793a54367260\") " Oct 31 05:14:09.709379 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:14:09.706010 4376 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2755051f-49ce-4ffe-8c54-0ae241bd45c2-plugins-config-volume\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:14:09.709379 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:14:09.706075 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3b9ba37-7625-40d7-be02-793a54367260-plugins-config-volume" pod "f3b9ba37-7625-40d7-be02-793a54367260" (UID: "f3b9ba37-7625-40d7-be02-793a54367260"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:14:09.709379 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:14:09.709025 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3b9ba37-7625-40d7-be02-793a54367260-kube-api-access-txmq6" pod "f3b9ba37-7625-40d7-be02-793a54367260" (UID: "f3b9ba37-7625-40d7-be02-793a54367260"). InnerVolumeSpecName "kube-api-access-txmq6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:14:09.711084 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:14:09.711062 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2755051f-49ce-4ffe-8c54-0ae241bd45c2-kube-api-access-w4562" pod "2755051f-49ce-4ffe-8c54-0ae241bd45c2" (UID: "2755051f-49ce-4ffe-8c54-0ae241bd45c2"). InnerVolumeSpecName "kube-api-access-w4562". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:14:09.807273 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:14:09.807242 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-txmq6\" (UniqueName: \"kubernetes.io/projected/f3b9ba37-7625-40d7-be02-793a54367260-kube-api-access-txmq6\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:14:09.807273 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:14:09.807273 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4562\" (UniqueName: \"kubernetes.io/projected/2755051f-49ce-4ffe-8c54-0ae241bd45c2-kube-api-access-w4562\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:14:09.807510 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:14:09.807284 4376 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3b9ba37-7625-40d7-be02-793a54367260-plugins-config-volume\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:14:10.280916 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:14:10.280879 4376 generic.go:356] "Generic (PLEG): container finished" podID="2755051f-49ce-4ffe-8c54-0ae241bd45c2" containerID="ec39282578e488f88ddb62e156380467b18ca902f9262892d32aa84ec924ca21" exitCode=0 Oct 31 05:14:10.281091 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:14:10.280928 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-654741-785568f87c-8k8tt" Oct 31 05:14:10.281091 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:14:10.280960 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-654741-785568f87c-8k8tt" event={"ID":"2755051f-49ce-4ffe-8c54-0ae241bd45c2","Type":"ContainerDied","Data":"ec39282578e488f88ddb62e156380467b18ca902f9262892d32aa84ec924ca21"} Oct 31 05:14:10.281091 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:14:10.281054 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-654741-785568f87c-8k8tt" event={"ID":"2755051f-49ce-4ffe-8c54-0ae241bd45c2","Type":"ContainerDied","Data":"0beafa35131a65a4398a5360463dd34e96658bbd4c3ffa529317fc41f5d4e111"} Oct 31 05:14:10.281091 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:14:10.281081 4376 scope.go:122] "RemoveContainer" containerID="ec39282578e488f88ddb62e156380467b18ca902f9262892d32aa84ec924ca21" Oct 31 05:14:10.282522 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:14:10.282499 4376 generic.go:356] "Generic (PLEG): container finished" podID="f3b9ba37-7625-40d7-be02-793a54367260" containerID="a9ba12a8c50a9828306739d887f380ca0a746cae93dca61421acb423f3ed4041" exitCode=0 Oct 31 05:14:10.282646 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:14:10.282545 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-654741-5dbd945cf9-wrd4x" Oct 31 05:14:10.282646 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:14:10.282605 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-654741-5dbd945cf9-wrd4x" event={"ID":"f3b9ba37-7625-40d7-be02-793a54367260","Type":"ContainerDied","Data":"a9ba12a8c50a9828306739d887f380ca0a746cae93dca61421acb423f3ed4041"} Oct 31 05:14:10.282739 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:14:10.282662 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-654741-5dbd945cf9-wrd4x" event={"ID":"f3b9ba37-7625-40d7-be02-793a54367260","Type":"ContainerDied","Data":"0c01a94ed567d681859d37e1578877064d6b1141c12b81ebea90670ce0d9c5bd"} Oct 31 05:14:10.292452 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:14:10.292435 4376 scope.go:122] "RemoveContainer" containerID="ec39282578e488f88ddb62e156380467b18ca902f9262892d32aa84ec924ca21" Oct 31 05:14:10.292782 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:14:10.292753 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec39282578e488f88ddb62e156380467b18ca902f9262892d32aa84ec924ca21\": container with ID starting with ec39282578e488f88ddb62e156380467b18ca902f9262892d32aa84ec924ca21 not found: ID does not exist" containerID="ec39282578e488f88ddb62e156380467b18ca902f9262892d32aa84ec924ca21" Oct 31 05:14:10.292868 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:14:10.292795 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec39282578e488f88ddb62e156380467b18ca902f9262892d32aa84ec924ca21"} err="failed to get container status \"ec39282578e488f88ddb62e156380467b18ca902f9262892d32aa84ec924ca21\": rpc error: code = NotFound desc = could not find container \"ec39282578e488f88ddb62e156380467b18ca902f9262892d32aa84ec924ca21\": container with ID starting with ec39282578e488f88ddb62e156380467b18ca902f9262892d32aa84ec924ca21 not found: ID does not exist" Oct 31 05:14:10.292868 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:14:10.292820 4376 scope.go:122] "RemoveContainer" containerID="a9ba12a8c50a9828306739d887f380ca0a746cae93dca61421acb423f3ed4041" Oct 31 05:14:10.302677 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:14:10.302656 4376 scope.go:122] "RemoveContainer" containerID="a9ba12a8c50a9828306739d887f380ca0a746cae93dca61421acb423f3ed4041" Oct 31 05:14:10.302946 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:14:10.302927 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9ba12a8c50a9828306739d887f380ca0a746cae93dca61421acb423f3ed4041\": container with ID starting with a9ba12a8c50a9828306739d887f380ca0a746cae93dca61421acb423f3ed4041 not found: ID does not exist" containerID="a9ba12a8c50a9828306739d887f380ca0a746cae93dca61421acb423f3ed4041" Oct 31 05:14:10.303014 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:14:10.302959 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9ba12a8c50a9828306739d887f380ca0a746cae93dca61421acb423f3ed4041"} err="failed to get container status \"a9ba12a8c50a9828306739d887f380ca0a746cae93dca61421acb423f3ed4041\": rpc error: code = NotFound desc = could not find container \"a9ba12a8c50a9828306739d887f380ca0a746cae93dca61421acb423f3ed4041\": container with ID starting with a9ba12a8c50a9828306739d887f380ca0a746cae93dca61421acb423f3ed4041 not found: ID does not exist" Oct 31 05:14:10.338730 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:14:10.338690 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-654741-5dbd945cf9-wrd4x"] Oct 31 05:14:10.353256 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:14:10.353225 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-654741-5dbd945cf9-wrd4x"] Oct 31 05:14:10.390628 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:14:10.390595 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-654741-785568f87c-8k8tt"] Oct 31 05:14:10.397091 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:14:10.397061 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-654741-785568f87c-8k8tt"] Oct 31 05:14:10.443786 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:14:10.443749 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2755051f-49ce-4ffe-8c54-0ae241bd45c2" path="/var/lib/kubelet/pods/2755051f-49ce-4ffe-8c54-0ae241bd45c2/volumes" Oct 31 05:14:10.444225 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:14:10.444210 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3b9ba37-7625-40d7-be02-793a54367260" path="/var/lib/kubelet/pods/f3b9ba37-7625-40d7-be02-793a54367260/volumes" Oct 31 05:14:11.001297 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:14:11.001261 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 05:14:14.592117 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:14:14.592064 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-scmn8"] Oct 31 05:14:14.594472 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:14:14.594411 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-scmn8" podUID="f95ddb3e-3a81-433a-aead-9a220d1ae61b" containerName="istio-proxy" containerID="cri-o://0ec9f3db650e8b422b9517c0fd438d67096a750bc3a1ab2ace78cc32cb43788c" gracePeriod=30 Oct 31 05:14:14.988720 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:14:14.988685 4376 patch_prober.go:29] interesting pod/e2e-worker-w2-09a20a-openshift-default-55d8647ff-scmn8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 05:14:14.988907 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:14:14.988746 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-scmn8" podUID="f95ddb3e-3a81-433a-aead-9a220d1ae61b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 05:14:19.855550 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:14:19.855518 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-scmn8" Oct 31 05:14:19.944395 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:14:19.944292 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f95ddb3e-3a81-433a-aead-9a220d1ae61b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f95ddb3e-3a81-433a-aead-9a220d1ae61b-credential-socket\") pod \"f95ddb3e-3a81-433a-aead-9a220d1ae61b\" (UID: \"f95ddb3e-3a81-433a-aead-9a220d1ae61b\") " Oct 31 05:14:19.944395 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:14:19.944374 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f95ddb3e-3a81-433a-aead-9a220d1ae61b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f95ddb3e-3a81-433a-aead-9a220d1ae61b-istio-envoy\") pod \"f95ddb3e-3a81-433a-aead-9a220d1ae61b\" (UID: \"f95ddb3e-3a81-433a-aead-9a220d1ae61b\") " Oct 31 05:14:19.944637 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:14:19.944500 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f95ddb3e-3a81-433a-aead-9a220d1ae61b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f95ddb3e-3a81-433a-aead-9a220d1ae61b-istiod-ca-cert\") pod \"f95ddb3e-3a81-433a-aead-9a220d1ae61b\" (UID: \"f95ddb3e-3a81-433a-aead-9a220d1ae61b\") " Oct 31 05:14:19.944637 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:14:19.944579 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f95ddb3e-3a81-433a-aead-9a220d1ae61b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f95ddb3e-3a81-433a-aead-9a220d1ae61b-workload-socket\") pod \"f95ddb3e-3a81-433a-aead-9a220d1ae61b\" (UID: \"f95ddb3e-3a81-433a-aead-9a220d1ae61b\") " Oct 31 05:14:19.944637 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:14:19.944602 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f95ddb3e-3a81-433a-aead-9a220d1ae61b-credential-socket" pod "f95ddb3e-3a81-433a-aead-9a220d1ae61b" (UID: "f95ddb3e-3a81-433a-aead-9a220d1ae61b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:14:19.944824 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:14:19.944660 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f95ddb3e-3a81-433a-aead-9a220d1ae61b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f95ddb3e-3a81-433a-aead-9a220d1ae61b-istio-podinfo\") pod \"f95ddb3e-3a81-433a-aead-9a220d1ae61b\" (UID: \"f95ddb3e-3a81-433a-aead-9a220d1ae61b\") " Oct 31 05:14:19.944824 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:14:19.944726 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f95ddb3e-3a81-433a-aead-9a220d1ae61b-istio-token\" (UniqueName: \"kubernetes.io/projected/f95ddb3e-3a81-433a-aead-9a220d1ae61b-istio-token\") pod \"f95ddb3e-3a81-433a-aead-9a220d1ae61b\" (UID: \"f95ddb3e-3a81-433a-aead-9a220d1ae61b\") " Oct 31 05:14:19.944942 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:14:19.944830 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f95ddb3e-3a81-433a-aead-9a220d1ae61b-kube-api-access-lxlcr\" (UniqueName: \"kubernetes.io/projected/f95ddb3e-3a81-433a-aead-9a220d1ae61b-kube-api-access-lxlcr\") pod \"f95ddb3e-3a81-433a-aead-9a220d1ae61b\" (UID: \"f95ddb3e-3a81-433a-aead-9a220d1ae61b\") " Oct 31 05:14:19.944942 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:14:19.944897 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f95ddb3e-3a81-433a-aead-9a220d1ae61b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f95ddb3e-3a81-433a-aead-9a220d1ae61b-istio-data\") pod \"f95ddb3e-3a81-433a-aead-9a220d1ae61b\" (UID: \"f95ddb3e-3a81-433a-aead-9a220d1ae61b\") " Oct 31 05:14:19.945107 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:14:19.944941 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f95ddb3e-3a81-433a-aead-9a220d1ae61b-workload-socket" pod "f95ddb3e-3a81-433a-aead-9a220d1ae61b" (UID: "f95ddb3e-3a81-433a-aead-9a220d1ae61b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:14:19.945107 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:14:19.944960 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f95ddb3e-3a81-433a-aead-9a220d1ae61b-istiod-ca-cert" pod "f95ddb3e-3a81-433a-aead-9a220d1ae61b" (UID: "f95ddb3e-3a81-433a-aead-9a220d1ae61b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:14:19.945107 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:14:19.945009 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f95ddb3e-3a81-433a-aead-9a220d1ae61b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f95ddb3e-3a81-433a-aead-9a220d1ae61b-workload-certs\") pod \"f95ddb3e-3a81-433a-aead-9a220d1ae61b\" (UID: \"f95ddb3e-3a81-433a-aead-9a220d1ae61b\") " Oct 31 05:14:19.945314 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:14:19.945286 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f95ddb3e-3a81-433a-aead-9a220d1ae61b-workload-certs" pod "f95ddb3e-3a81-433a-aead-9a220d1ae61b" (UID: "f95ddb3e-3a81-433a-aead-9a220d1ae61b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:14:19.945558 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:14:19.945535 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f95ddb3e-3a81-433a-aead-9a220d1ae61b-istio-data" pod "f95ddb3e-3a81-433a-aead-9a220d1ae61b" (UID: "f95ddb3e-3a81-433a-aead-9a220d1ae61b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:14:19.945722 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:14:19.945701 4376 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f95ddb3e-3a81-433a-aead-9a220d1ae61b-istiod-ca-cert\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:14:19.945806 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:14:19.945727 4376 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f95ddb3e-3a81-433a-aead-9a220d1ae61b-workload-socket\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:14:19.945806 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:14:19.945743 4376 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f95ddb3e-3a81-433a-aead-9a220d1ae61b-istio-data\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:14:19.945806 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:14:19.945758 4376 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f95ddb3e-3a81-433a-aead-9a220d1ae61b-workload-certs\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:14:19.945806 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:14:19.945772 4376 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f95ddb3e-3a81-433a-aead-9a220d1ae61b-credential-socket\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:14:19.947288 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:14:19.947263 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f95ddb3e-3a81-433a-aead-9a220d1ae61b-istio-envoy" pod "f95ddb3e-3a81-433a-aead-9a220d1ae61b" (UID: "f95ddb3e-3a81-433a-aead-9a220d1ae61b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:14:19.947601 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:14:19.947571 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f95ddb3e-3a81-433a-aead-9a220d1ae61b-istio-podinfo" pod "f95ddb3e-3a81-433a-aead-9a220d1ae61b" (UID: "f95ddb3e-3a81-433a-aead-9a220d1ae61b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 05:14:19.947741 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:14:19.947725 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f95ddb3e-3a81-433a-aead-9a220d1ae61b-istio-token" pod "f95ddb3e-3a81-433a-aead-9a220d1ae61b" (UID: "f95ddb3e-3a81-433a-aead-9a220d1ae61b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:14:19.947815 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:14:19.947788 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f95ddb3e-3a81-433a-aead-9a220d1ae61b-kube-api-access-lxlcr" pod "f95ddb3e-3a81-433a-aead-9a220d1ae61b" (UID: "f95ddb3e-3a81-433a-aead-9a220d1ae61b"). InnerVolumeSpecName "kube-api-access-lxlcr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:14:20.046945 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:14:20.046911 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxlcr\" (UniqueName: \"kubernetes.io/projected/f95ddb3e-3a81-433a-aead-9a220d1ae61b-kube-api-access-lxlcr\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:14:20.046945 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:14:20.046945 4376 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f95ddb3e-3a81-433a-aead-9a220d1ae61b-istio-envoy\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:14:20.047159 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:14:20.046958 4376 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f95ddb3e-3a81-433a-aead-9a220d1ae61b-istio-podinfo\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:14:20.047159 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:14:20.046972 4376 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f95ddb3e-3a81-433a-aead-9a220d1ae61b-istio-token\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:14:20.383369 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:14:20.383339 4376 generic.go:356] "Generic (PLEG): container finished" podID="f95ddb3e-3a81-433a-aead-9a220d1ae61b" containerID="0ec9f3db650e8b422b9517c0fd438d67096a750bc3a1ab2ace78cc32cb43788c" exitCode=0 Oct 31 05:14:20.383560 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:14:20.383383 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-scmn8" Oct 31 05:14:20.383560 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:14:20.383419 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-scmn8" event={"ID":"f95ddb3e-3a81-433a-aead-9a220d1ae61b","Type":"ContainerDied","Data":"0ec9f3db650e8b422b9517c0fd438d67096a750bc3a1ab2ace78cc32cb43788c"} Oct 31 05:14:20.383560 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:14:20.383494 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-scmn8" event={"ID":"f95ddb3e-3a81-433a-aead-9a220d1ae61b","Type":"ContainerDied","Data":"fccc9b34eb8ff7e03c26aadbd1da0cc62c5060540a9de0f6bba436048098848b"} Oct 31 05:14:20.383560 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:14:20.383519 4376 scope.go:122] "RemoveContainer" containerID="0ec9f3db650e8b422b9517c0fd438d67096a750bc3a1ab2ace78cc32cb43788c" Oct 31 05:14:20.394149 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:14:20.394133 4376 scope.go:122] "RemoveContainer" containerID="0ec9f3db650e8b422b9517c0fd438d67096a750bc3a1ab2ace78cc32cb43788c" Oct 31 05:14:20.394453 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:14:20.394434 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ec9f3db650e8b422b9517c0fd438d67096a750bc3a1ab2ace78cc32cb43788c\": container with ID starting with 0ec9f3db650e8b422b9517c0fd438d67096a750bc3a1ab2ace78cc32cb43788c not found: ID does not exist" containerID="0ec9f3db650e8b422b9517c0fd438d67096a750bc3a1ab2ace78cc32cb43788c" Oct 31 05:14:20.394504 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:14:20.394462 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ec9f3db650e8b422b9517c0fd438d67096a750bc3a1ab2ace78cc32cb43788c"} err="failed to get container status \"0ec9f3db650e8b422b9517c0fd438d67096a750bc3a1ab2ace78cc32cb43788c\": rpc error: code = NotFound desc = could not find container \"0ec9f3db650e8b422b9517c0fd438d67096a750bc3a1ab2ace78cc32cb43788c\": container with ID starting with 0ec9f3db650e8b422b9517c0fd438d67096a750bc3a1ab2ace78cc32cb43788c not found: ID does not exist" Oct 31 05:14:20.458062 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:14:20.458031 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-scmn8"] Oct 31 05:14:20.472376 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:14:20.472346 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-scmn8"] Oct 31 05:14:22.444730 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:14:22.444692 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f95ddb3e-3a81-433a-aead-9a220d1ae61b" path="/var/lib/kubelet/pods/f95ddb3e-3a81-433a-aead-9a220d1ae61b/volumes" Oct 31 05:14:41.002795 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:14:41.002754 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 05:14:42.115034 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:14:42.114996 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 05:14:46.392435 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:14:46.392399 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-57dc6d-5675c77f6c-8t8zt"] Oct 31 05:14:46.428683 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:14:46.428646 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-57dc6d-66998bfb9f-tdltw"] Oct 31 05:14:51.393050 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:14:51.392994 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-57dc6d-5675c77f6c-8t8zt" podUID="a4beb0fb-a6b6-4864-a65a-38e9e3a9c05f" containerName="payload-processing" containerID="cri-o://1758d081570b8116946a169c4a5db9527b35b5cecea86452c37517f52ca46260" gracePeriod=25 Oct 31 05:14:51.429504 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:14:51.429458 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-57dc6d-66998bfb9f-tdltw" podUID="07764cf3-3c73-4f06-82c6-7df4d5e9955a" containerName="payload-pre-processing" containerID="cri-o://cf5938a4da3a2d3c07bd214995e57d887657c35f23802f7eeaa0d0cd98b8be1c" gracePeriod=25 Oct 31 05:14:51.619337 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:14:51.619297 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-57dc6d-5675c77f6c-8t8zt" podUID="a4beb0fb-a6b6-4864-a65a-38e9e3a9c05f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 05:14:52.639145 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:14:52.638786 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-57dc6d-66998bfb9f-tdltw" podUID="07764cf3-3c73-4f06-82c6-7df4d5e9955a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 05:15:00.207832 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:00.207757 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a99f2a-29890395-cmqz6"] Oct 31 05:15:00.208511 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:00.208490 4376 cpu_manager.go:414] "containerMap: removing container" podUID="df1df0a3-2701-4f32-a6f7-3f7ce203cfee" containerName="payload-processing" Oct 31 05:15:00.208615 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:00.208514 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df1df0a3-2701-4f32-a6f7-3f7ce203cfee" containerName="payload-processing" Oct 31 05:15:00.208615 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:00.208536 4376 cpu_manager.go:414] "containerMap: removing container" podUID="932966ca-c565-4bab-a007-7f918c7f0c15" containerName="istio-proxy" Oct 31 05:15:00.208615 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:00.208545 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="932966ca-c565-4bab-a007-7f918c7f0c15" containerName="istio-proxy" Oct 31 05:15:00.208615 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:00.208556 4376 cpu_manager.go:414] "containerMap: removing container" podUID="a3f9d0a0-a106-48bb-b6f2-a980c3aff4e7" containerName="main" Oct 31 05:15:00.208615 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:00.208561 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3f9d0a0-a106-48bb-b6f2-a980c3aff4e7" containerName="main" Oct 31 05:15:00.208615 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:00.208569 4376 cpu_manager.go:414] "containerMap: removing container" podUID="649b78d1-3eb4-4780-8462-f6c6a5cf7c6c" containerName="maas-api" Oct 31 05:15:00.208615 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:00.208574 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="649b78d1-3eb4-4780-8462-f6c6a5cf7c6c" containerName="maas-api" Oct 31 05:15:00.208615 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:00.208582 4376 cpu_manager.go:414] "containerMap: removing container" podUID="f31be489-e267-4087-9b94-86d7d622c841" containerName="payload-pre-processing" Oct 31 05:15:00.208615 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:00.208587 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f31be489-e267-4087-9b94-86d7d622c841" containerName="payload-pre-processing" Oct 31 05:15:00.208615 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:00.208595 4376 cpu_manager.go:414] "containerMap: removing container" podUID="f95ddb3e-3a81-433a-aead-9a220d1ae61b" containerName="istio-proxy" Oct 31 05:15:00.208615 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:00.208599 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f95ddb3e-3a81-433a-aead-9a220d1ae61b" containerName="istio-proxy" Oct 31 05:15:00.208615 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:00.208607 4376 cpu_manager.go:414] "containerMap: removing container" podUID="adda31ec-5cfd-473c-8edd-5e7d335aec12" containerName="payload-pre-processing" Oct 31 05:15:00.208615 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:00.208613 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="adda31ec-5cfd-473c-8edd-5e7d335aec12" containerName="payload-pre-processing" Oct 31 05:15:00.209368 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:00.208630 4376 cpu_manager.go:414] "containerMap: removing container" podUID="cb75359a-8880-406f-955c-5a3fe0fb796c" containerName="payload-pre-processing" Oct 31 05:15:00.209368 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:00.208635 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb75359a-8880-406f-955c-5a3fe0fb796c" containerName="payload-pre-processing" Oct 31 05:15:00.209368 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:00.208644 4376 cpu_manager.go:414] "containerMap: removing container" podUID="a6c79800-45c4-4778-91ef-027ceba7bb8a" containerName="istio-proxy" Oct 31 05:15:00.209368 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:00.208649 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6c79800-45c4-4778-91ef-027ceba7bb8a" containerName="istio-proxy" Oct 31 05:15:00.209368 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:00.208657 4376 cpu_manager.go:414] "containerMap: removing container" podUID="fe905f47-822c-4bbb-b562-8bd04f2b35a1" containerName="payload-processing" Oct 31 05:15:00.209368 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:00.208662 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe905f47-822c-4bbb-b562-8bd04f2b35a1" containerName="payload-processing" Oct 31 05:15:00.209368 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:00.208668 4376 cpu_manager.go:414] "containerMap: removing container" podUID="17b2f181-2e66-4ce2-95b0-cb213f11a0e7" containerName="main" Oct 31 05:15:00.209368 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:00.208675 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17b2f181-2e66-4ce2-95b0-cb213f11a0e7" containerName="main" Oct 31 05:15:00.209368 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:00.208684 4376 cpu_manager.go:414] "containerMap: removing container" podUID="c81f5d58-c5c7-46c1-b832-c550b092f02b" containerName="main" Oct 31 05:15:00.209368 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:00.208691 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c81f5d58-c5c7-46c1-b832-c550b092f02b" containerName="main" Oct 31 05:15:00.209368 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:00.208702 4376 cpu_manager.go:414] "containerMap: removing container" podUID="7b2dd0ee-880c-4538-b706-3412b77b7a53" containerName="payload-pre-processing" Oct 31 05:15:00.209368 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:00.208709 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b2dd0ee-880c-4538-b706-3412b77b7a53" containerName="payload-pre-processing" Oct 31 05:15:00.209368 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:00.208725 4376 cpu_manager.go:414] "containerMap: removing container" podUID="84000b34-2ffa-4914-97f3-73bafc243ec5" containerName="maas-api" Oct 31 05:15:00.209368 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:00.208731 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84000b34-2ffa-4914-97f3-73bafc243ec5" containerName="maas-api" Oct 31 05:15:00.209368 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:00.208742 4376 cpu_manager.go:414] "containerMap: removing container" podUID="65edcc7f-0600-4161-9ba1-89e26f064dd4" containerName="maas-api" Oct 31 05:15:00.209368 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:00.208749 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65edcc7f-0600-4161-9ba1-89e26f064dd4" containerName="maas-api" Oct 31 05:15:00.209368 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:00.208764 4376 cpu_manager.go:414] "containerMap: removing container" podUID="932966ca-c565-4bab-a007-7f918c7f0c15" containerName="istio-proxy" Oct 31 05:15:00.209368 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:00.208771 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="932966ca-c565-4bab-a007-7f918c7f0c15" containerName="istio-proxy" Oct 31 05:15:00.209368 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:00.208782 4376 cpu_manager.go:414] "containerMap: removing container" podUID="7d918c33-5fb2-4f29-a616-c29446e15ff2" containerName="payload-processing" Oct 31 05:15:00.209368 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:00.208790 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d918c33-5fb2-4f29-a616-c29446e15ff2" containerName="payload-processing" Oct 31 05:15:00.209368 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:00.208800 4376 cpu_manager.go:414] "containerMap: removing container" podUID="48d10907-ef2a-43b7-b481-746473b85ef6" containerName="payload-pre-processing" Oct 31 05:15:00.209368 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:00.208807 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48d10907-ef2a-43b7-b481-746473b85ef6" containerName="payload-pre-processing" Oct 31 05:15:00.209368 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:00.208821 4376 cpu_manager.go:414] "containerMap: removing container" podUID="0551b77c-f034-48e4-b643-3361b903df31" containerName="payload-processing" Oct 31 05:15:00.209368 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:00.208828 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0551b77c-f034-48e4-b643-3361b903df31" containerName="payload-processing" Oct 31 05:15:00.209368 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:00.208838 4376 cpu_manager.go:414] "containerMap: removing container" podUID="babafa7c-43f1-4ca0-aee3-6b8b4bda5dd7" containerName="istio-proxy" Oct 31 05:15:00.209368 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:00.208844 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="babafa7c-43f1-4ca0-aee3-6b8b4bda5dd7" containerName="istio-proxy" Oct 31 05:15:00.209368 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:00.208857 4376 cpu_manager.go:414] "containerMap: removing container" podUID="a0071e27-190b-4977-9c6f-c3458c714d64" containerName="payload-processing" Oct 31 05:15:00.209368 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:00.208862 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0071e27-190b-4977-9c6f-c3458c714d64" containerName="payload-processing" Oct 31 05:15:00.209368 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:00.208873 4376 cpu_manager.go:414] "containerMap: removing container" podUID="2755051f-49ce-4ffe-8c54-0ae241bd45c2" containerName="payload-processing" Oct 31 05:15:00.209368 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:00.208878 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2755051f-49ce-4ffe-8c54-0ae241bd45c2" containerName="payload-processing" Oct 31 05:15:00.209368 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:00.208887 4376 cpu_manager.go:414] "containerMap: removing container" podUID="0a252fc6-14bf-4524-907a-f8ee294acda5" containerName="payload-processing" Oct 31 05:15:00.209368 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:00.208891 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a252fc6-14bf-4524-907a-f8ee294acda5" containerName="payload-processing" Oct 31 05:15:00.209368 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:00.208900 4376 cpu_manager.go:414] "containerMap: removing container" podUID="f48d27d4-dc90-4bf7-aa79-2b14c97859a2" containerName="payload-processing" Oct 31 05:15:00.209368 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:00.208906 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f48d27d4-dc90-4bf7-aa79-2b14c97859a2" containerName="payload-processing" Oct 31 05:15:00.209368 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:00.208916 4376 cpu_manager.go:414] "containerMap: removing container" podUID="e1abe8c9-b4dc-4380-8495-a0ce56fb2734" containerName="payload-pre-processing" Oct 31 05:15:00.209368 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:00.208924 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1abe8c9-b4dc-4380-8495-a0ce56fb2734" containerName="payload-pre-processing" Oct 31 05:15:00.209368 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:00.208940 4376 cpu_manager.go:414] "containerMap: removing container" podUID="341d12d1-db55-4078-acb0-47169d00ac7e" containerName="istio-proxy" Oct 31 05:15:00.209368 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:00.208947 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="341d12d1-db55-4078-acb0-47169d00ac7e" containerName="istio-proxy" Oct 31 05:15:00.209368 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:00.208961 4376 cpu_manager.go:414] "containerMap: removing container" podUID="32954f51-d716-449a-ba69-ca46851a9ce2" containerName="istio-proxy" Oct 31 05:15:00.209368 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:00.208967 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32954f51-d716-449a-ba69-ca46851a9ce2" containerName="istio-proxy" Oct 31 05:15:00.209368 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:00.208986 4376 cpu_manager.go:414] "containerMap: removing container" podUID="32954f51-d716-449a-ba69-ca46851a9ce2" containerName="istio-proxy" Oct 31 05:15:00.209368 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:00.208993 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32954f51-d716-449a-ba69-ca46851a9ce2" containerName="istio-proxy" Oct 31 05:15:00.209368 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:00.209004 4376 cpu_manager.go:414] "containerMap: removing container" podUID="c0b21674-516a-452f-86c5-b27115eb72e3" containerName="maas-api" Oct 31 05:15:00.209368 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:00.209013 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0b21674-516a-452f-86c5-b27115eb72e3" containerName="maas-api" Oct 31 05:15:00.209368 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:00.209022 4376 cpu_manager.go:414] "containerMap: removing container" podUID="681f54ec-b648-49cb-ae16-dc814e56bb71" containerName="payload-processing" Oct 31 05:15:00.209368 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:00.209027 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="681f54ec-b648-49cb-ae16-dc814e56bb71" containerName="payload-processing" Oct 31 05:15:00.209368 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:00.209033 4376 cpu_manager.go:414] "containerMap: removing container" podUID="7bde4a1c-a591-4864-8276-2c3a74b99e70" containerName="istio-proxy" Oct 31 05:15:00.209368 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:00.209039 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7bde4a1c-a591-4864-8276-2c3a74b99e70" containerName="istio-proxy" Oct 31 05:15:00.209368 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:00.209048 4376 cpu_manager.go:414] "containerMap: removing container" podUID="7d298000-b65b-45fd-a1ee-092e47a7eb24" containerName="main" Oct 31 05:15:00.209368 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:00.209052 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d298000-b65b-45fd-a1ee-092e47a7eb24" containerName="main" Oct 31 05:15:00.209368 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:00.209059 4376 cpu_manager.go:414] "containerMap: removing container" podUID="f3b9ba37-7625-40d7-be02-793a54367260" containerName="payload-pre-processing" Oct 31 05:15:00.209368 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:00.209063 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3b9ba37-7625-40d7-be02-793a54367260" containerName="payload-pre-processing" Oct 31 05:15:00.209368 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:00.209069 4376 cpu_manager.go:414] "containerMap: removing container" podUID="7bde4a1c-a591-4864-8276-2c3a74b99e70" containerName="istio-proxy" Oct 31 05:15:00.209368 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:00.209074 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7bde4a1c-a591-4864-8276-2c3a74b99e70" containerName="istio-proxy" Oct 31 05:15:00.209368 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:00.209086 4376 cpu_manager.go:414] "containerMap: removing container" podUID="8ab573d4-4b87-40e0-9d16-ff7fdd3c3963" containerName="payload-pre-processing" Oct 31 05:15:00.209368 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:00.209093 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ab573d4-4b87-40e0-9d16-ff7fdd3c3963" containerName="payload-pre-processing" Oct 31 05:15:00.209368 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:00.209103 4376 cpu_manager.go:414] "containerMap: removing container" podUID="faaa68df-a884-400f-be9a-84bbb75ad520" containerName="istio-proxy" Oct 31 05:15:00.209368 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:00.209107 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="faaa68df-a884-400f-be9a-84bbb75ad520" containerName="istio-proxy" Oct 31 05:15:00.209368 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:00.209115 4376 cpu_manager.go:414] "containerMap: removing container" podUID="d76a0ace-f020-430f-a26c-005231cb8be2" containerName="istio-proxy" Oct 31 05:15:00.209368 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:00.209119 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d76a0ace-f020-430f-a26c-005231cb8be2" containerName="istio-proxy" Oct 31 05:15:00.209368 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:00.209212 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="f48d27d4-dc90-4bf7-aa79-2b14c97859a2" containerName="payload-processing" Oct 31 05:15:00.209368 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:00.209227 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1abe8c9-b4dc-4380-8495-a0ce56fb2734" containerName="payload-pre-processing" Oct 31 05:15:00.209368 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:00.209236 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="341d12d1-db55-4078-acb0-47169d00ac7e" containerName="istio-proxy" Oct 31 05:15:00.209368 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:00.209243 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="32954f51-d716-449a-ba69-ca46851a9ce2" containerName="istio-proxy" Oct 31 05:15:00.209368 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:00.209251 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="32954f51-d716-449a-ba69-ca46851a9ce2" containerName="istio-proxy" Oct 31 05:15:00.209368 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:00.209255 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0b21674-516a-452f-86c5-b27115eb72e3" containerName="maas-api" Oct 31 05:15:00.209368 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:00.209260 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="681f54ec-b648-49cb-ae16-dc814e56bb71" containerName="payload-processing" Oct 31 05:15:00.209368 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:00.209265 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="7bde4a1c-a591-4864-8276-2c3a74b99e70" containerName="istio-proxy" Oct 31 05:15:00.209368 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:00.209271 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d298000-b65b-45fd-a1ee-092e47a7eb24" containerName="main" Oct 31 05:15:00.209368 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:00.209277 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3b9ba37-7625-40d7-be02-793a54367260" containerName="payload-pre-processing" Oct 31 05:15:00.209368 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:00.209282 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="7bde4a1c-a591-4864-8276-2c3a74b99e70" containerName="istio-proxy" Oct 31 05:15:00.209368 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:00.209288 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6c79800-45c4-4778-91ef-027ceba7bb8a" containerName="istio-proxy" Oct 31 05:15:00.209368 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:00.209294 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ab573d4-4b87-40e0-9d16-ff7fdd3c3963" containerName="payload-pre-processing" Oct 31 05:15:00.209368 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:00.209304 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="faaa68df-a884-400f-be9a-84bbb75ad520" containerName="istio-proxy" Oct 31 05:15:00.209368 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:00.209314 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="d76a0ace-f020-430f-a26c-005231cb8be2" containerName="istio-proxy" Oct 31 05:15:00.209368 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:00.209321 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="df1df0a3-2701-4f32-a6f7-3f7ce203cfee" containerName="payload-processing" Oct 31 05:15:00.209368 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:00.209340 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="932966ca-c565-4bab-a007-7f918c7f0c15" containerName="istio-proxy" Oct 31 05:15:00.209368 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:00.209350 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3f9d0a0-a106-48bb-b6f2-a980c3aff4e7" containerName="main" Oct 31 05:15:00.209368 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:00.209359 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="649b78d1-3eb4-4780-8462-f6c6a5cf7c6c" containerName="maas-api" Oct 31 05:15:00.209368 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:00.209365 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="f31be489-e267-4087-9b94-86d7d622c841" containerName="payload-pre-processing" Oct 31 05:15:00.209368 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:00.209373 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="f95ddb3e-3a81-433a-aead-9a220d1ae61b" containerName="istio-proxy" Oct 31 05:15:00.209368 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:00.209379 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="adda31ec-5cfd-473c-8edd-5e7d335aec12" containerName="payload-pre-processing" Oct 31 05:15:00.209368 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:00.209385 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb75359a-8880-406f-955c-5a3fe0fb796c" containerName="payload-pre-processing" Oct 31 05:15:00.209368 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:00.209391 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6c79800-45c4-4778-91ef-027ceba7bb8a" containerName="istio-proxy" Oct 31 05:15:00.209368 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:00.209397 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe905f47-822c-4bbb-b562-8bd04f2b35a1" containerName="payload-processing" Oct 31 05:15:00.209368 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:00.209403 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="17b2f181-2e66-4ce2-95b0-cb213f11a0e7" containerName="main" Oct 31 05:15:00.209368 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:00.209409 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="c81f5d58-c5c7-46c1-b832-c550b092f02b" containerName="main" Oct 31 05:15:00.209368 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:00.209413 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b2dd0ee-880c-4538-b706-3412b77b7a53" containerName="payload-pre-processing" Oct 31 05:15:00.209368 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:00.209420 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="84000b34-2ffa-4914-97f3-73bafc243ec5" containerName="maas-api" Oct 31 05:15:00.209368 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:00.209424 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="65edcc7f-0600-4161-9ba1-89e26f064dd4" containerName="maas-api" Oct 31 05:15:00.209368 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:00.209430 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="932966ca-c565-4bab-a007-7f918c7f0c15" containerName="istio-proxy" Oct 31 05:15:00.209368 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:00.209436 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d918c33-5fb2-4f29-a616-c29446e15ff2" containerName="payload-processing" Oct 31 05:15:00.213088 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:00.209441 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="48d10907-ef2a-43b7-b481-746473b85ef6" containerName="payload-pre-processing" Oct 31 05:15:00.213088 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:00.209450 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="0551b77c-f034-48e4-b643-3361b903df31" containerName="payload-processing" Oct 31 05:15:00.213088 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:00.209458 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="babafa7c-43f1-4ca0-aee3-6b8b4bda5dd7" containerName="istio-proxy" Oct 31 05:15:00.213088 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:00.209464 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0071e27-190b-4977-9c6f-c3458c714d64" containerName="payload-processing" Oct 31 05:15:00.213088 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:00.209470 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="96a130da-eaed-4765-905b-261822ce8014" containerName="istio-proxy" Oct 31 05:15:00.213088 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:00.209476 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="2755051f-49ce-4ffe-8c54-0ae241bd45c2" containerName="payload-processing" Oct 31 05:15:00.213088 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:00.209481 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a252fc6-14bf-4524-907a-f8ee294acda5" containerName="payload-processing" Oct 31 05:15:00.213088 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:00.212924 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a99f2a-29890395-cmqz6" Oct 31 05:15:00.229494 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:00.229465 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a99f2a-29890395-cmqz6"] Oct 31 05:15:00.278779 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:00.278747 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzb8r\"" Oct 31 05:15:00.313415 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:00.313375 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8q2wt\" (UniqueName: \"kubernetes.io/projected/72af9c95-36f4-467b-ba78-ec84c1d1d51c-kube-api-access-8q2wt\") pod \"maas-api-key-cleanup-e2e-extmod-b-a99f2a-29890395-cmqz6\" (UID: \"72af9c95-36f4-467b-ba78-ec84c1d1d51c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a99f2a-29890395-cmqz6" Oct 31 05:15:00.313620 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:00.313595 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/72af9c95-36f4-467b-ba78-ec84c1d1d51c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-a99f2a-29890395-cmqz6\" (UID: \"72af9c95-36f4-467b-ba78-ec84c1d1d51c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a99f2a-29890395-cmqz6" Oct 31 05:15:00.414659 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:00.414623 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8q2wt\" (UniqueName: \"kubernetes.io/projected/72af9c95-36f4-467b-ba78-ec84c1d1d51c-kube-api-access-8q2wt\") pod \"maas-api-key-cleanup-e2e-extmod-b-a99f2a-29890395-cmqz6\" (UID: \"72af9c95-36f4-467b-ba78-ec84c1d1d51c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a99f2a-29890395-cmqz6" Oct 31 05:15:00.414832 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:00.414766 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/72af9c95-36f4-467b-ba78-ec84c1d1d51c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-a99f2a-29890395-cmqz6\" (UID: \"72af9c95-36f4-467b-ba78-ec84c1d1d51c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a99f2a-29890395-cmqz6" Oct 31 05:15:00.415448 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:00.415423 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/72af9c95-36f4-467b-ba78-ec84c1d1d51c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-a99f2a-29890395-cmqz6\" (UID: \"72af9c95-36f4-467b-ba78-ec84c1d1d51c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a99f2a-29890395-cmqz6" Oct 31 05:15:00.440488 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:00.440459 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8q2wt\" (UniqueName: \"kubernetes.io/projected/72af9c95-36f4-467b-ba78-ec84c1d1d51c-kube-api-access-8q2wt\") pod \"maas-api-key-cleanup-e2e-extmod-b-a99f2a-29890395-cmqz6\" (UID: \"72af9c95-36f4-467b-ba78-ec84c1d1d51c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a99f2a-29890395-cmqz6" Oct 31 05:15:00.567403 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:00.559557 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a99f2a-29890395-cmqz6" Oct 31 05:15:00.901838 ip-10-0-134-155 kubenswrapper[4376]: W1031 05:15:00.901805 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72af9c95_36f4_467b_ba78_ec84c1d1d51c.slice/crio-7ea9a2b2c576fcc7c58a7cddd731ae0af4efed449c0466e70233dca482576f11 WatchSource:0}: Error finding container 7ea9a2b2c576fcc7c58a7cddd731ae0af4efed449c0466e70233dca482576f11: Status 404 returned error can't find the container with id 7ea9a2b2c576fcc7c58a7cddd731ae0af4efed449c0466e70233dca482576f11 Oct 31 05:15:00.902480 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:00.902387 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a99f2a-29890395-cmqz6"] Oct 31 05:15:01.618954 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:01.618922 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-57dc6d-5675c77f6c-8t8zt" podUID="a4beb0fb-a6b6-4864-a65a-38e9e3a9c05f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 05:15:01.661085 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:01.661049 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a99f2a-29890395-cmqz6" event={"ID":"72af9c95-36f4-467b-ba78-ec84c1d1d51c","Type":"ContainerStarted","Data":"c6570ec8c6f55a4b48ad10fae6ee417d3491cef8c8e19fb9720fb0b6afe4930f"} Oct 31 05:15:01.661257 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:01.661104 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a99f2a-29890395-cmqz6" event={"ID":"72af9c95-36f4-467b-ba78-ec84c1d1d51c","Type":"ContainerStarted","Data":"7ea9a2b2c576fcc7c58a7cddd731ae0af4efed449c0466e70233dca482576f11"} Oct 31 05:15:01.707387 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:01.707314 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a99f2a-29890395-cmqz6" podStartSLOduration=1.707299782 podStartE2EDuration="1.707299782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 05:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 05:15:01.704295174 +0000 UTC m=+4249.418629791" watchObservedRunningTime="2026-10-31 05:15:01.707299782 +0000 UTC m=+4249.421634389" Oct 31 05:15:02.628546 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:02.628505 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-57dc6d-66998bfb9f-tdltw" podUID="07764cf3-3c73-4f06-82c6-7df4d5e9955a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 05:15:02.667955 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:02.667919 4376 generic.go:356] "Generic (PLEG): container finished" podID="72af9c95-36f4-467b-ba78-ec84c1d1d51c" containerID="c6570ec8c6f55a4b48ad10fae6ee417d3491cef8c8e19fb9720fb0b6afe4930f" exitCode=60 Oct 31 05:15:02.668128 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:02.668009 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a99f2a-29890395-cmqz6" event={"ID":"72af9c95-36f4-467b-ba78-ec84c1d1d51c","Type":"ContainerDied","Data":"c6570ec8c6f55a4b48ad10fae6ee417d3491cef8c8e19fb9720fb0b6afe4930f"} Oct 31 05:15:02.668284 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:02.668268 4376 scope.go:122] "RemoveContainer" containerID="c6570ec8c6f55a4b48ad10fae6ee417d3491cef8c8e19fb9720fb0b6afe4930f" Oct 31 05:15:03.701948 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:03.701843 4376 generic.go:356] "Generic (PLEG): container finished" podID="72af9c95-36f4-467b-ba78-ec84c1d1d51c" containerID="255928c4207b5d5275c492af97a79583fb953e3a06ad21210cc1d3d72463f7c9" exitCode=60 Oct 31 05:15:03.702705 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:03.702051 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a99f2a-29890395-cmqz6" event={"ID":"72af9c95-36f4-467b-ba78-ec84c1d1d51c","Type":"ContainerDied","Data":"255928c4207b5d5275c492af97a79583fb953e3a06ad21210cc1d3d72463f7c9"} Oct 31 05:15:03.702705 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:03.702146 4376 scope.go:122] "RemoveContainer" containerID="c6570ec8c6f55a4b48ad10fae6ee417d3491cef8c8e19fb9720fb0b6afe4930f" Oct 31 05:15:03.702857 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:03.702736 4376 scope.go:122] "RemoveContainer" containerID="255928c4207b5d5275c492af97a79583fb953e3a06ad21210cc1d3d72463f7c9" Oct 31 05:15:03.703155 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:15:03.703126 4376 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-a99f2a-29890395-cmqz6_odh-ai-gateway-infra(72af9c95-36f4-467b-ba78-ec84c1d1d51c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a99f2a-29890395-cmqz6" podUID="72af9c95-36f4-467b-ba78-ec84c1d1d51c" Oct 31 05:15:04.709690 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:04.709665 4376 scope.go:122] "RemoveContainer" containerID="255928c4207b5d5275c492af97a79583fb953e3a06ad21210cc1d3d72463f7c9" Oct 31 05:15:04.710091 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:15:04.709951 4376 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-a99f2a-29890395-cmqz6_odh-ai-gateway-infra(72af9c95-36f4-467b-ba78-ec84c1d1d51c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a99f2a-29890395-cmqz6" podUID="72af9c95-36f4-467b-ba78-ec84c1d1d51c" Oct 31 05:15:11.006718 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:15:11.006679 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 05:15:11.710731 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:11.710702 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-57dc6d-5675c77f6c-8t8zt" Oct 31 05:15:11.742448 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:11.742421 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-57dc6d-66998bfb9f-tdltw" Oct 31 05:15:11.770560 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:11.770524 4376 generic.go:356] "Generic (PLEG): container finished" podID="a4beb0fb-a6b6-4864-a65a-38e9e3a9c05f" containerID="1758d081570b8116946a169c4a5db9527b35b5cecea86452c37517f52ca46260" exitCode=0 Oct 31 05:15:11.770744 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:11.770583 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-57dc6d-5675c77f6c-8t8zt" Oct 31 05:15:11.770744 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:11.770606 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-57dc6d-5675c77f6c-8t8zt" event={"ID":"a4beb0fb-a6b6-4864-a65a-38e9e3a9c05f","Type":"ContainerDied","Data":"1758d081570b8116946a169c4a5db9527b35b5cecea86452c37517f52ca46260"} Oct 31 05:15:11.770744 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:11.770658 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-57dc6d-5675c77f6c-8t8zt" event={"ID":"a4beb0fb-a6b6-4864-a65a-38e9e3a9c05f","Type":"ContainerDied","Data":"9473677f0de2b838df8df6eb63c2a39699da8e392e3d2187e181c50ba12d951e"} Oct 31 05:15:11.770744 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:11.770682 4376 scope.go:122] "RemoveContainer" containerID="1758d081570b8116946a169c4a5db9527b35b5cecea86452c37517f52ca46260" Oct 31 05:15:11.772309 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:11.772286 4376 generic.go:356] "Generic (PLEG): container finished" podID="07764cf3-3c73-4f06-82c6-7df4d5e9955a" containerID="cf5938a4da3a2d3c07bd214995e57d887657c35f23802f7eeaa0d0cd98b8be1c" exitCode=0 Oct 31 05:15:11.772466 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:11.772354 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-57dc6d-66998bfb9f-tdltw" Oct 31 05:15:11.772466 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:11.772357 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-57dc6d-66998bfb9f-tdltw" event={"ID":"07764cf3-3c73-4f06-82c6-7df4d5e9955a","Type":"ContainerDied","Data":"cf5938a4da3a2d3c07bd214995e57d887657c35f23802f7eeaa0d0cd98b8be1c"} Oct 31 05:15:11.772466 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:11.772406 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-57dc6d-66998bfb9f-tdltw" event={"ID":"07764cf3-3c73-4f06-82c6-7df4d5e9955a","Type":"ContainerDied","Data":"dee0eecf1c5f019fe0614dac843a0a3db9a11555266b5fd0971c61d1cd535571"} Oct 31 05:15:11.781956 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:11.781935 4376 scope.go:122] "RemoveContainer" containerID="1758d081570b8116946a169c4a5db9527b35b5cecea86452c37517f52ca46260" Oct 31 05:15:11.782268 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:15:11.782241 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1758d081570b8116946a169c4a5db9527b35b5cecea86452c37517f52ca46260\": container with ID starting with 1758d081570b8116946a169c4a5db9527b35b5cecea86452c37517f52ca46260 not found: ID does not exist" containerID="1758d081570b8116946a169c4a5db9527b35b5cecea86452c37517f52ca46260" Oct 31 05:15:11.782352 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:11.782281 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1758d081570b8116946a169c4a5db9527b35b5cecea86452c37517f52ca46260"} err="failed to get container status \"1758d081570b8116946a169c4a5db9527b35b5cecea86452c37517f52ca46260\": rpc error: code = NotFound desc = could not find container \"1758d081570b8116946a169c4a5db9527b35b5cecea86452c37517f52ca46260\": container with ID starting with 1758d081570b8116946a169c4a5db9527b35b5cecea86452c37517f52ca46260 not found: ID does not exist" Oct 31 05:15:11.782352 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:11.782306 4376 scope.go:122] "RemoveContainer" containerID="cf5938a4da3a2d3c07bd214995e57d887657c35f23802f7eeaa0d0cd98b8be1c" Oct 31 05:15:11.793841 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:11.793819 4376 scope.go:122] "RemoveContainer" containerID="cf5938a4da3a2d3c07bd214995e57d887657c35f23802f7eeaa0d0cd98b8be1c" Oct 31 05:15:11.794158 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:15:11.794127 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf5938a4da3a2d3c07bd214995e57d887657c35f23802f7eeaa0d0cd98b8be1c\": container with ID starting with cf5938a4da3a2d3c07bd214995e57d887657c35f23802f7eeaa0d0cd98b8be1c not found: ID does not exist" containerID="cf5938a4da3a2d3c07bd214995e57d887657c35f23802f7eeaa0d0cd98b8be1c" Oct 31 05:15:11.794298 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:11.794167 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf5938a4da3a2d3c07bd214995e57d887657c35f23802f7eeaa0d0cd98b8be1c"} err="failed to get container status \"cf5938a4da3a2d3c07bd214995e57d887657c35f23802f7eeaa0d0cd98b8be1c\": rpc error: code = NotFound desc = could not find container \"cf5938a4da3a2d3c07bd214995e57d887657c35f23802f7eeaa0d0cd98b8be1c\": container with ID starting with cf5938a4da3a2d3c07bd214995e57d887657c35f23802f7eeaa0d0cd98b8be1c not found: ID does not exist" Oct 31 05:15:11.865915 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:11.865883 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4beb0fb-a6b6-4864-a65a-38e9e3a9c05f-kube-api-access-jwc4j\" (UniqueName: \"kubernetes.io/projected/a4beb0fb-a6b6-4864-a65a-38e9e3a9c05f-kube-api-access-jwc4j\") pod \"a4beb0fb-a6b6-4864-a65a-38e9e3a9c05f\" (UID: \"a4beb0fb-a6b6-4864-a65a-38e9e3a9c05f\") " Oct 31 05:15:11.866110 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:11.865959 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07764cf3-3c73-4f06-82c6-7df4d5e9955a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07764cf3-3c73-4f06-82c6-7df4d5e9955a-plugins-config-volume\") pod \"07764cf3-3c73-4f06-82c6-7df4d5e9955a\" (UID: \"07764cf3-3c73-4f06-82c6-7df4d5e9955a\") " Oct 31 05:15:11.866110 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:11.866028 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07764cf3-3c73-4f06-82c6-7df4d5e9955a-kube-api-access-qmkgj\" (UniqueName: \"kubernetes.io/projected/07764cf3-3c73-4f06-82c6-7df4d5e9955a-kube-api-access-qmkgj\") pod \"07764cf3-3c73-4f06-82c6-7df4d5e9955a\" (UID: \"07764cf3-3c73-4f06-82c6-7df4d5e9955a\") " Oct 31 05:15:11.866110 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:11.866102 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4beb0fb-a6b6-4864-a65a-38e9e3a9c05f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4beb0fb-a6b6-4864-a65a-38e9e3a9c05f-plugins-config-volume\") pod \"a4beb0fb-a6b6-4864-a65a-38e9e3a9c05f\" (UID: \"a4beb0fb-a6b6-4864-a65a-38e9e3a9c05f\") " Oct 31 05:15:11.866394 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:11.866363 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07764cf3-3c73-4f06-82c6-7df4d5e9955a-plugins-config-volume" pod "07764cf3-3c73-4f06-82c6-7df4d5e9955a" (UID: "07764cf3-3c73-4f06-82c6-7df4d5e9955a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:15:11.866569 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:11.866461 4376 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07764cf3-3c73-4f06-82c6-7df4d5e9955a-plugins-config-volume\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:15:11.866569 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:11.866478 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4beb0fb-a6b6-4864-a65a-38e9e3a9c05f-plugins-config-volume" pod "a4beb0fb-a6b6-4864-a65a-38e9e3a9c05f" (UID: "a4beb0fb-a6b6-4864-a65a-38e9e3a9c05f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:15:11.868132 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:11.868105 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4beb0fb-a6b6-4864-a65a-38e9e3a9c05f-kube-api-access-jwc4j" pod "a4beb0fb-a6b6-4864-a65a-38e9e3a9c05f" (UID: "a4beb0fb-a6b6-4864-a65a-38e9e3a9c05f"). InnerVolumeSpecName "kube-api-access-jwc4j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:15:11.868132 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:11.868125 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07764cf3-3c73-4f06-82c6-7df4d5e9955a-kube-api-access-qmkgj" pod "07764cf3-3c73-4f06-82c6-7df4d5e9955a" (UID: "07764cf3-3c73-4f06-82c6-7df4d5e9955a"). InnerVolumeSpecName "kube-api-access-qmkgj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:15:11.967633 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:11.967559 4376 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4beb0fb-a6b6-4864-a65a-38e9e3a9c05f-plugins-config-volume\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:15:11.967633 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:11.967589 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwc4j\" (UniqueName: \"kubernetes.io/projected/a4beb0fb-a6b6-4864-a65a-38e9e3a9c05f-kube-api-access-jwc4j\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:15:11.967633 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:11.967599 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmkgj\" (UniqueName: \"kubernetes.io/projected/07764cf3-3c73-4f06-82c6-7df4d5e9955a-kube-api-access-qmkgj\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:15:12.122303 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:12.122271 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-57dc6d-5675c77f6c-8t8zt"] Oct 31 05:15:12.136479 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:12.136445 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-57dc6d-5675c77f6c-8t8zt"] Oct 31 05:15:12.166439 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:12.166409 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-57dc6d-66998bfb9f-tdltw"] Oct 31 05:15:12.169202 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:12.169175 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-57dc6d-66998bfb9f-tdltw"] Oct 31 05:15:12.444201 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:12.444165 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07764cf3-3c73-4f06-82c6-7df4d5e9955a" path="/var/lib/kubelet/pods/07764cf3-3c73-4f06-82c6-7df4d5e9955a/volumes" Oct 31 05:15:12.444643 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:12.444625 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4beb0fb-a6b6-4864-a65a-38e9e3a9c05f" path="/var/lib/kubelet/pods/a4beb0fb-a6b6-4864-a65a-38e9e3a9c05f/volumes" Oct 31 05:15:12.618625 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:12.618587 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-57dc6d-5675c77f6c-8t8zt" podUID="a4beb0fb-a6b6-4864-a65a-38e9e3a9c05f" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.149:50052\" within 1s: context deadline exceeded" Oct 31 05:15:12.618625 ip-10-0-134-155 kubenswrapper[4376]: W1031 05:15:12.618633 4376 logging.go:55] [core] [Channel #23585 SubChannel #23586]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.149:50052", ServerName: "10.133.0.149:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.149:50052: operation was canceled" Oct 31 05:15:14.341391 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:14.341358 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 05:15:15.392970 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:15.392926 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-9633f0-6b8776566f-bbqd7"] Oct 31 05:15:16.436252 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:16.436220 4376 scope.go:122] "RemoveContainer" containerID="255928c4207b5d5275c492af97a79583fb953e3a06ad21210cc1d3d72463f7c9" Oct 31 05:15:17.831639 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:17.831607 4376 generic.go:356] "Generic (PLEG): container finished" podID="72af9c95-36f4-467b-ba78-ec84c1d1d51c" containerID="3482d9034d154459ddf1c26e0562a0c4683279e5a8badc7d938c3a70d7c0c263" exitCode=60 Oct 31 05:15:17.832027 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:17.831655 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a99f2a-29890395-cmqz6" event={"ID":"72af9c95-36f4-467b-ba78-ec84c1d1d51c","Type":"ContainerDied","Data":"3482d9034d154459ddf1c26e0562a0c4683279e5a8badc7d938c3a70d7c0c263"} Oct 31 05:15:17.832027 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:17.831725 4376 scope.go:122] "RemoveContainer" containerID="255928c4207b5d5275c492af97a79583fb953e3a06ad21210cc1d3d72463f7c9" Oct 31 05:15:17.832132 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:17.832071 4376 scope.go:122] "RemoveContainer" containerID="3482d9034d154459ddf1c26e0562a0c4683279e5a8badc7d938c3a70d7c0c263" Oct 31 05:15:17.832445 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:15:17.832423 4376 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-a99f2a-29890395-cmqz6_odh-ai-gateway-infra(72af9c95-36f4-467b-ba78-ec84c1d1d51c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a99f2a-29890395-cmqz6" podUID="72af9c95-36f4-467b-ba78-ec84c1d1d51c" Oct 31 05:15:18.886367 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:18.886314 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a99f2a-29890395-cmqz6"] Oct 31 05:15:19.061957 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:19.061932 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a99f2a-29890395-cmqz6" Oct 31 05:15:19.190037 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:19.189957 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72af9c95-36f4-467b-ba78-ec84c1d1d51c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/72af9c95-36f4-467b-ba78-ec84c1d1d51c-maas-api-service-ca\") pod \"72af9c95-36f4-467b-ba78-ec84c1d1d51c\" (UID: \"72af9c95-36f4-467b-ba78-ec84c1d1d51c\") " Oct 31 05:15:19.190218 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:19.190201 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72af9c95-36f4-467b-ba78-ec84c1d1d51c-kube-api-access-8q2wt\" (UniqueName: \"kubernetes.io/projected/72af9c95-36f4-467b-ba78-ec84c1d1d51c-kube-api-access-8q2wt\") pod \"72af9c95-36f4-467b-ba78-ec84c1d1d51c\" (UID: \"72af9c95-36f4-467b-ba78-ec84c1d1d51c\") " Oct 31 05:15:19.190340 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:19.190301 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72af9c95-36f4-467b-ba78-ec84c1d1d51c-maas-api-service-ca" pod "72af9c95-36f4-467b-ba78-ec84c1d1d51c" (UID: "72af9c95-36f4-467b-ba78-ec84c1d1d51c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:15:19.190612 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:19.190594 4376 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/72af9c95-36f4-467b-ba78-ec84c1d1d51c-maas-api-service-ca\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:15:19.192323 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:19.192292 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72af9c95-36f4-467b-ba78-ec84c1d1d51c-kube-api-access-8q2wt" pod "72af9c95-36f4-467b-ba78-ec84c1d1d51c" (UID: "72af9c95-36f4-467b-ba78-ec84c1d1d51c"). InnerVolumeSpecName "kube-api-access-8q2wt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:15:19.292398 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:19.292354 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8q2wt\" (UniqueName: \"kubernetes.io/projected/72af9c95-36f4-467b-ba78-ec84c1d1d51c-kube-api-access-8q2wt\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:15:19.847432 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:19.847396 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a99f2a-29890395-cmqz6" event={"ID":"72af9c95-36f4-467b-ba78-ec84c1d1d51c","Type":"ContainerDied","Data":"7ea9a2b2c576fcc7c58a7cddd731ae0af4efed449c0466e70233dca482576f11"} Oct 31 05:15:19.847609 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:19.847468 4376 scope.go:122] "RemoveContainer" containerID="3482d9034d154459ddf1c26e0562a0c4683279e5a8badc7d938c3a70d7c0c263" Oct 31 05:15:19.847609 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:19.847482 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a99f2a-29890395-cmqz6" Oct 31 05:15:19.884995 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:15:19.884963 4376 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72af9c95_36f4_467b_ba78_ec84c1d1d51c.slice/crio-7ea9a2b2c576fcc7c58a7cddd731ae0af4efed449c0466e70233dca482576f11\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72af9c95_36f4_467b_ba78_ec84c1d1d51c.slice\": RecentStats: unable to find data in memory cache]" Oct 31 05:15:19.914433 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:19.914401 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a99f2a-29890395-cmqz6"] Oct 31 05:15:19.916978 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:19.916951 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a99f2a-29890395-cmqz6"] Oct 31 05:15:20.394007 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:20.393964 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-9633f0-6b8776566f-bbqd7" podUID="86377cdd-c75b-496b-8fea-1d8bcd541be8" containerName="payload-processing" containerID="cri-o://165729339cf75e2852dfb4e74a25f79dc0456a64c9af3d7428d67cfea8f3f735" gracePeriod=25 Oct 31 05:15:20.473953 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:20.473918 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72af9c95-36f4-467b-ba78-ec84c1d1d51c" path="/var/lib/kubelet/pods/72af9c95-36f4-467b-ba78-ec84c1d1d51c/volumes" Oct 31 05:15:28.270532 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:28.270431 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-9633f0-6b8776566f-bbqd7" podUID="86377cdd-c75b-496b-8fea-1d8bcd541be8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 05:15:36.517535 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:36.517501 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-9633f0-7c5f7dbc86-prfm5"] Oct 31 05:15:36.518030 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:36.517734 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-9633f0-7c5f7dbc86-prfm5" podUID="775b25ed-d591-4102-98b8-59ce485f9f52" containerName="maas-api" containerID="cri-o://409bb134f43af78336e192b2b1f9238e6d60ef9df19f2b69e4faea4c96991a2b" gracePeriod=30 Oct 31 05:15:36.567083 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:36.567044 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-9633f0-7c5f7dbc86-prfm5" podUID="775b25ed-d591-4102-98b8-59ce485f9f52" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.157:8443/health\": dial tcp 10.133.0.157:8443: connect: connection refused" Oct 31 05:15:36.840571 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:36.840542 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-9633f0-7c5f7dbc86-prfm5" Oct 31 05:15:36.913193 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:36.913154 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/775b25ed-d591-4102-98b8-59ce485f9f52-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/775b25ed-d591-4102-98b8-59ce485f9f52-openshift-service-ca\") pod \"775b25ed-d591-4102-98b8-59ce485f9f52\" (UID: \"775b25ed-d591-4102-98b8-59ce485f9f52\") " Oct 31 05:15:36.913404 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:36.913348 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/775b25ed-d591-4102-98b8-59ce485f9f52-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/775b25ed-d591-4102-98b8-59ce485f9f52-maas-api-tls\") pod \"775b25ed-d591-4102-98b8-59ce485f9f52\" (UID: \"775b25ed-d591-4102-98b8-59ce485f9f52\") " Oct 31 05:15:36.913499 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:36.913407 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/775b25ed-d591-4102-98b8-59ce485f9f52-kube-api-access-xktf8\" (UniqueName: \"kubernetes.io/projected/775b25ed-d591-4102-98b8-59ce485f9f52-kube-api-access-xktf8\") pod \"775b25ed-d591-4102-98b8-59ce485f9f52\" (UID: \"775b25ed-d591-4102-98b8-59ce485f9f52\") " Oct 31 05:15:36.913567 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:36.913538 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/775b25ed-d591-4102-98b8-59ce485f9f52-metrics-certs\" (UniqueName: \"kubernetes.io/secret/775b25ed-d591-4102-98b8-59ce485f9f52-metrics-certs\") pod \"775b25ed-d591-4102-98b8-59ce485f9f52\" (UID: \"775b25ed-d591-4102-98b8-59ce485f9f52\") " Oct 31 05:15:36.913693 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:36.913607 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/775b25ed-d591-4102-98b8-59ce485f9f52-openshift-service-ca" pod "775b25ed-d591-4102-98b8-59ce485f9f52" (UID: "775b25ed-d591-4102-98b8-59ce485f9f52"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:15:36.914013 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:36.913980 4376 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/775b25ed-d591-4102-98b8-59ce485f9f52-openshift-service-ca\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:15:36.916180 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:36.916139 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/775b25ed-d591-4102-98b8-59ce485f9f52-maas-api-tls" pod "775b25ed-d591-4102-98b8-59ce485f9f52" (UID: "775b25ed-d591-4102-98b8-59ce485f9f52"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 05:15:36.919373 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:36.919348 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/775b25ed-d591-4102-98b8-59ce485f9f52-metrics-certs" pod "775b25ed-d591-4102-98b8-59ce485f9f52" (UID: "775b25ed-d591-4102-98b8-59ce485f9f52"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 05:15:36.919492 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:36.919421 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/775b25ed-d591-4102-98b8-59ce485f9f52-kube-api-access-xktf8" pod "775b25ed-d591-4102-98b8-59ce485f9f52" (UID: "775b25ed-d591-4102-98b8-59ce485f9f52"). InnerVolumeSpecName "kube-api-access-xktf8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:15:36.950647 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:36.950620 4376 generic.go:356] "Generic (PLEG): container finished" podID="775b25ed-d591-4102-98b8-59ce485f9f52" containerID="409bb134f43af78336e192b2b1f9238e6d60ef9df19f2b69e4faea4c96991a2b" exitCode=0 Oct 31 05:15:36.950767 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:36.950671 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-9633f0-7c5f7dbc86-prfm5" Oct 31 05:15:36.950767 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:36.950712 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-9633f0-7c5f7dbc86-prfm5" event={"ID":"775b25ed-d591-4102-98b8-59ce485f9f52","Type":"ContainerDied","Data":"409bb134f43af78336e192b2b1f9238e6d60ef9df19f2b69e4faea4c96991a2b"} Oct 31 05:15:36.950843 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:36.950772 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-9633f0-7c5f7dbc86-prfm5" event={"ID":"775b25ed-d591-4102-98b8-59ce485f9f52","Type":"ContainerDied","Data":"f75ddcf68fc25f87703c873bacb0bd4792f72a621af63d259c86dea5e8f0af25"} Oct 31 05:15:36.950843 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:36.950802 4376 scope.go:122] "RemoveContainer" containerID="409bb134f43af78336e192b2b1f9238e6d60ef9df19f2b69e4faea4c96991a2b" Oct 31 05:15:36.962575 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:36.962556 4376 scope.go:122] "RemoveContainer" containerID="409bb134f43af78336e192b2b1f9238e6d60ef9df19f2b69e4faea4c96991a2b" Oct 31 05:15:36.962904 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:15:36.962874 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"409bb134f43af78336e192b2b1f9238e6d60ef9df19f2b69e4faea4c96991a2b\": container with ID starting with 409bb134f43af78336e192b2b1f9238e6d60ef9df19f2b69e4faea4c96991a2b not found: ID does not exist" containerID="409bb134f43af78336e192b2b1f9238e6d60ef9df19f2b69e4faea4c96991a2b" Oct 31 05:15:36.962996 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:36.962914 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"409bb134f43af78336e192b2b1f9238e6d60ef9df19f2b69e4faea4c96991a2b"} err="failed to get container status \"409bb134f43af78336e192b2b1f9238e6d60ef9df19f2b69e4faea4c96991a2b\": rpc error: code = NotFound desc = could not find container \"409bb134f43af78336e192b2b1f9238e6d60ef9df19f2b69e4faea4c96991a2b\": container with ID starting with 409bb134f43af78336e192b2b1f9238e6d60ef9df19f2b69e4faea4c96991a2b not found: ID does not exist" Oct 31 05:15:37.037642 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:37.037601 4376 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/775b25ed-d591-4102-98b8-59ce485f9f52-maas-api-tls\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:15:37.037642 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:37.037645 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xktf8\" (UniqueName: \"kubernetes.io/projected/775b25ed-d591-4102-98b8-59ce485f9f52-kube-api-access-xktf8\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:15:37.037908 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:37.037660 4376 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/775b25ed-d591-4102-98b8-59ce485f9f52-metrics-certs\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:15:37.039948 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:37.039888 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-9633f0-7c5f7dbc86-prfm5"] Oct 31 05:15:37.051075 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:37.049927 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-9633f0-7c5f7dbc86-prfm5"] Oct 31 05:15:38.269963 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:38.269918 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-9633f0-6b8776566f-bbqd7" podUID="86377cdd-c75b-496b-8fea-1d8bcd541be8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 05:15:38.442698 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:38.442659 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="775b25ed-d591-4102-98b8-59ce485f9f52" path="/var/lib/kubelet/pods/775b25ed-d591-4102-98b8-59ce485f9f52/volumes" Oct 31 05:15:40.663523 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:40.663458 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-9633f0-6b8776566f-bbqd7" Oct 31 05:15:40.744363 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:40.744321 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86377cdd-c75b-496b-8fea-1d8bcd541be8-kube-api-access-9mhhn\" (UniqueName: \"kubernetes.io/projected/86377cdd-c75b-496b-8fea-1d8bcd541be8-kube-api-access-9mhhn\") pod \"86377cdd-c75b-496b-8fea-1d8bcd541be8\" (UID: \"86377cdd-c75b-496b-8fea-1d8bcd541be8\") " Oct 31 05:15:40.744565 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:40.744383 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86377cdd-c75b-496b-8fea-1d8bcd541be8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86377cdd-c75b-496b-8fea-1d8bcd541be8-plugins-config-volume\") pod \"86377cdd-c75b-496b-8fea-1d8bcd541be8\" (UID: \"86377cdd-c75b-496b-8fea-1d8bcd541be8\") " Oct 31 05:15:40.744808 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:40.744782 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86377cdd-c75b-496b-8fea-1d8bcd541be8-plugins-config-volume" pod "86377cdd-c75b-496b-8fea-1d8bcd541be8" (UID: "86377cdd-c75b-496b-8fea-1d8bcd541be8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:15:40.744953 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:40.744885 4376 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86377cdd-c75b-496b-8fea-1d8bcd541be8-plugins-config-volume\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:15:40.746664 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:40.746636 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86377cdd-c75b-496b-8fea-1d8bcd541be8-kube-api-access-9mhhn" pod "86377cdd-c75b-496b-8fea-1d8bcd541be8" (UID: "86377cdd-c75b-496b-8fea-1d8bcd541be8"). InnerVolumeSpecName "kube-api-access-9mhhn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:15:40.846554 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:40.846513 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mhhn\" (UniqueName: \"kubernetes.io/projected/86377cdd-c75b-496b-8fea-1d8bcd541be8-kube-api-access-9mhhn\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:15:40.987373 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:40.987263 4376 generic.go:356] "Generic (PLEG): container finished" podID="86377cdd-c75b-496b-8fea-1d8bcd541be8" containerID="165729339cf75e2852dfb4e74a25f79dc0456a64c9af3d7428d67cfea8f3f735" exitCode=0 Oct 31 05:15:40.987373 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:40.987311 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-9633f0-6b8776566f-bbqd7" Oct 31 05:15:40.987373 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:40.987351 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-9633f0-6b8776566f-bbqd7" event={"ID":"86377cdd-c75b-496b-8fea-1d8bcd541be8","Type":"ContainerDied","Data":"165729339cf75e2852dfb4e74a25f79dc0456a64c9af3d7428d67cfea8f3f735"} Oct 31 05:15:40.987666 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:40.987418 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-9633f0-6b8776566f-bbqd7" event={"ID":"86377cdd-c75b-496b-8fea-1d8bcd541be8","Type":"ContainerDied","Data":"0bef063e1d9fe6367ee8f908929e78106f3549afcbd8960d6a1f58929b9c1370"} Oct 31 05:15:40.987666 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:40.987452 4376 scope.go:122] "RemoveContainer" containerID="165729339cf75e2852dfb4e74a25f79dc0456a64c9af3d7428d67cfea8f3f735" Oct 31 05:15:41.001488 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:41.001467 4376 scope.go:122] "RemoveContainer" containerID="165729339cf75e2852dfb4e74a25f79dc0456a64c9af3d7428d67cfea8f3f735" Oct 31 05:15:41.001873 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:15:41.001840 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"165729339cf75e2852dfb4e74a25f79dc0456a64c9af3d7428d67cfea8f3f735\": container with ID starting with 165729339cf75e2852dfb4e74a25f79dc0456a64c9af3d7428d67cfea8f3f735 not found: ID does not exist" containerID="165729339cf75e2852dfb4e74a25f79dc0456a64c9af3d7428d67cfea8f3f735" Oct 31 05:15:41.002000 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:41.001883 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"165729339cf75e2852dfb4e74a25f79dc0456a64c9af3d7428d67cfea8f3f735"} err="failed to get container status \"165729339cf75e2852dfb4e74a25f79dc0456a64c9af3d7428d67cfea8f3f735\": rpc error: code = NotFound desc = could not find container \"165729339cf75e2852dfb4e74a25f79dc0456a64c9af3d7428d67cfea8f3f735\": container with ID starting with 165729339cf75e2852dfb4e74a25f79dc0456a64c9af3d7428d67cfea8f3f735 not found: ID does not exist" Oct 31 05:15:41.004647 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:15:41.004617 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 05:15:41.059175 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:41.058371 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-9633f0-6b8776566f-bbqd7"] Oct 31 05:15:41.064024 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:41.063535 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-9633f0-6b8776566f-bbqd7"] Oct 31 05:15:42.442721 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:42.442688 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86377cdd-c75b-496b-8fea-1d8bcd541be8" path="/var/lib/kubelet/pods/86377cdd-c75b-496b-8fea-1d8bcd541be8/volumes" Oct 31 05:15:43.626132 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:43.626098 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 05:15:45.262970 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:45.262940 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-c4006d/distinct-2-w4-c4006d-kserve-776b5b899-kpl6z"] Oct 31 05:15:45.296199 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:45.296167 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-c4006d/simulator-w4-c4006d-kserve-595455485b-4h46q"] Oct 31 05:15:52.252255 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:52.252224 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-afc519-d746ff6cc-npmp9"] Oct 31 05:15:52.289179 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:52.289145 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-afc519-69cbc98467-l7w68"] Oct 31 05:15:57.253573 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:57.253525 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-afc519-d746ff6cc-npmp9" podUID="e90146f8-ed41-49b8-bf8b-c92bee5f8cdd" containerName="payload-processing" containerID="cri-o://2a7a08c67ee09683a03dcda6277fc82fa6c5690a205555171d927bbb2ef54aa7" gracePeriod=25 Oct 31 05:15:57.289750 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:15:57.289700 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-afc519-69cbc98467-l7w68" podUID="84d95c12-4dca-44d2-a2cb-fdbfe919135b" containerName="payload-pre-processing" containerID="cri-o://53c4edd38ab68b826ecbed62aa5bff0dd9e26fd76265fa486ad478d6b0967fa0" gracePeriod=25 Oct 31 05:16:00.283738 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:00.283620 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-c4006d/distinct-2-w4-c4006d-kserve-776b5b899-kpl6z" podUID="dc53083d-eadd-4cb8-8633-35e02d400b9c" containerName="main" containerID="cri-o://6c20446894934dcfa9b40993be9fa2a23769d15a81224a240f6cbcc76caba357" gracePeriod=45 Oct 31 05:16:00.317122 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:00.317091 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-c4006d/simulator-w4-c4006d-kserve-595455485b-4h46q" podUID="576e2c1f-1c80-43a9-a136-3692776642ae" containerName="main" containerID="cri-o://91b226397f85d99ec63a941b055c3984264717467120c5a5fa7954382d6cc2fe" gracePeriod=45 Oct 31 05:16:00.582868 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:00.582841 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c4006d/distinct-2-w4-c4006d-kserve-776b5b899-kpl6z" Oct 31 05:16:00.604177 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:00.604156 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c4006d/simulator-w4-c4006d-kserve-595455485b-4h46q" Oct 31 05:16:00.753247 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:00.753201 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/576e2c1f-1c80-43a9-a136-3692776642ae-dshm\" (UniqueName: \"kubernetes.io/empty-dir/576e2c1f-1c80-43a9-a136-3692776642ae-dshm\") pod \"576e2c1f-1c80-43a9-a136-3692776642ae\" (UID: \"576e2c1f-1c80-43a9-a136-3692776642ae\") " Oct 31 05:16:00.753683 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:00.753313 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/576e2c1f-1c80-43a9-a136-3692776642ae-kube-api-access-nn6xd\" (UniqueName: \"kubernetes.io/projected/576e2c1f-1c80-43a9-a136-3692776642ae-kube-api-access-nn6xd\") pod \"576e2c1f-1c80-43a9-a136-3692776642ae\" (UID: \"576e2c1f-1c80-43a9-a136-3692776642ae\") " Oct 31 05:16:00.753683 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:00.753385 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc53083d-eadd-4cb8-8633-35e02d400b9c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc53083d-eadd-4cb8-8633-35e02d400b9c-model-cache\") pod \"dc53083d-eadd-4cb8-8633-35e02d400b9c\" (UID: \"dc53083d-eadd-4cb8-8633-35e02d400b9c\") " Oct 31 05:16:00.753683 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:00.753485 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/576e2c1f-1c80-43a9-a136-3692776642ae-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/576e2c1f-1c80-43a9-a136-3692776642ae-tmp-dir\") pod \"576e2c1f-1c80-43a9-a136-3692776642ae\" (UID: \"576e2c1f-1c80-43a9-a136-3692776642ae\") " Oct 31 05:16:00.753683 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:00.753526 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc53083d-eadd-4cb8-8633-35e02d400b9c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc53083d-eadd-4cb8-8633-35e02d400b9c-dshm\") pod \"dc53083d-eadd-4cb8-8633-35e02d400b9c\" (UID: \"dc53083d-eadd-4cb8-8633-35e02d400b9c\") " Oct 31 05:16:00.753683 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:00.753557 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc53083d-eadd-4cb8-8633-35e02d400b9c-tls-certs\" (UniqueName: \"kubernetes.io/secret/dc53083d-eadd-4cb8-8633-35e02d400b9c-tls-certs\") pod \"dc53083d-eadd-4cb8-8633-35e02d400b9c\" (UID: \"dc53083d-eadd-4cb8-8633-35e02d400b9c\") " Oct 31 05:16:00.753683 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:00.753608 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/576e2c1f-1c80-43a9-a136-3692776642ae-home\" (UniqueName: \"kubernetes.io/empty-dir/576e2c1f-1c80-43a9-a136-3692776642ae-home\") pod \"576e2c1f-1c80-43a9-a136-3692776642ae\" (UID: \"576e2c1f-1c80-43a9-a136-3692776642ae\") " Oct 31 05:16:00.753683 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:00.753643 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc53083d-eadd-4cb8-8633-35e02d400b9c-kube-api-access-mn7kq\" (UniqueName: \"kubernetes.io/projected/dc53083d-eadd-4cb8-8633-35e02d400b9c-kube-api-access-mn7kq\") pod \"dc53083d-eadd-4cb8-8633-35e02d400b9c\" (UID: \"dc53083d-eadd-4cb8-8633-35e02d400b9c\") " Oct 31 05:16:00.753683 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:00.753681 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/576e2c1f-1c80-43a9-a136-3692776642ae-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/576e2c1f-1c80-43a9-a136-3692776642ae-model-cache\") pod \"576e2c1f-1c80-43a9-a136-3692776642ae\" (UID: \"576e2c1f-1c80-43a9-a136-3692776642ae\") " Oct 31 05:16:00.754176 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:00.753728 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc53083d-eadd-4cb8-8633-35e02d400b9c-home\" (UniqueName: \"kubernetes.io/empty-dir/dc53083d-eadd-4cb8-8633-35e02d400b9c-home\") pod \"dc53083d-eadd-4cb8-8633-35e02d400b9c\" (UID: \"dc53083d-eadd-4cb8-8633-35e02d400b9c\") " Oct 31 05:16:00.754176 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:00.753763 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/576e2c1f-1c80-43a9-a136-3692776642ae-tls-certs\" (UniqueName: \"kubernetes.io/secret/576e2c1f-1c80-43a9-a136-3692776642ae-tls-certs\") pod \"576e2c1f-1c80-43a9-a136-3692776642ae\" (UID: \"576e2c1f-1c80-43a9-a136-3692776642ae\") " Oct 31 05:16:00.754176 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:00.753802 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc53083d-eadd-4cb8-8633-35e02d400b9c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc53083d-eadd-4cb8-8633-35e02d400b9c-tmp-dir\") pod \"dc53083d-eadd-4cb8-8633-35e02d400b9c\" (UID: \"dc53083d-eadd-4cb8-8633-35e02d400b9c\") " Oct 31 05:16:00.754176 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:00.753835 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/576e2c1f-1c80-43a9-a136-3692776642ae-tmp-dir" pod "576e2c1f-1c80-43a9-a136-3692776642ae" (UID: "576e2c1f-1c80-43a9-a136-3692776642ae"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:16:00.754435 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:00.754315 4376 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/576e2c1f-1c80-43a9-a136-3692776642ae-tmp-dir\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:16:00.754655 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:00.754629 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc53083d-eadd-4cb8-8633-35e02d400b9c-home" pod "dc53083d-eadd-4cb8-8633-35e02d400b9c" (UID: "dc53083d-eadd-4cb8-8633-35e02d400b9c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:16:00.754768 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:00.754754 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc53083d-eadd-4cb8-8633-35e02d400b9c-tmp-dir" pod "dc53083d-eadd-4cb8-8633-35e02d400b9c" (UID: "dc53083d-eadd-4cb8-8633-35e02d400b9c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:16:00.756607 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:00.755075 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/576e2c1f-1c80-43a9-a136-3692776642ae-home" pod "576e2c1f-1c80-43a9-a136-3692776642ae" (UID: "576e2c1f-1c80-43a9-a136-3692776642ae"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:16:00.756607 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:00.755715 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc53083d-eadd-4cb8-8633-35e02d400b9c-model-cache" pod "dc53083d-eadd-4cb8-8633-35e02d400b9c" (UID: "dc53083d-eadd-4cb8-8633-35e02d400b9c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:16:00.756607 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:00.755770 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/576e2c1f-1c80-43a9-a136-3692776642ae-model-cache" pod "576e2c1f-1c80-43a9-a136-3692776642ae" (UID: "576e2c1f-1c80-43a9-a136-3692776642ae"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:16:00.756607 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:00.756227 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/576e2c1f-1c80-43a9-a136-3692776642ae-kube-api-access-nn6xd" pod "576e2c1f-1c80-43a9-a136-3692776642ae" (UID: "576e2c1f-1c80-43a9-a136-3692776642ae"). InnerVolumeSpecName "kube-api-access-nn6xd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:16:00.756607 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:00.756397 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/576e2c1f-1c80-43a9-a136-3692776642ae-dshm" pod "576e2c1f-1c80-43a9-a136-3692776642ae" (UID: "576e2c1f-1c80-43a9-a136-3692776642ae"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:16:00.756970 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:00.756618 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc53083d-eadd-4cb8-8633-35e02d400b9c-dshm" pod "dc53083d-eadd-4cb8-8633-35e02d400b9c" (UID: "dc53083d-eadd-4cb8-8633-35e02d400b9c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:16:00.756970 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:00.756668 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc53083d-eadd-4cb8-8633-35e02d400b9c-kube-api-access-mn7kq" pod "dc53083d-eadd-4cb8-8633-35e02d400b9c" (UID: "dc53083d-eadd-4cb8-8633-35e02d400b9c"). InnerVolumeSpecName "kube-api-access-mn7kq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:16:00.757137 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:00.757112 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/576e2c1f-1c80-43a9-a136-3692776642ae-tls-certs" pod "576e2c1f-1c80-43a9-a136-3692776642ae" (UID: "576e2c1f-1c80-43a9-a136-3692776642ae"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 05:16:00.757992 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:00.757971 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc53083d-eadd-4cb8-8633-35e02d400b9c-tls-certs" pod "dc53083d-eadd-4cb8-8633-35e02d400b9c" (UID: "dc53083d-eadd-4cb8-8633-35e02d400b9c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 05:16:00.855210 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:00.855133 4376 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc53083d-eadd-4cb8-8633-35e02d400b9c-dshm\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:16:00.855210 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:00.855163 4376 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc53083d-eadd-4cb8-8633-35e02d400b9c-tls-certs\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:16:00.855210 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:00.855179 4376 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/576e2c1f-1c80-43a9-a136-3692776642ae-home\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:16:00.855210 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:00.855192 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mn7kq\" (UniqueName: \"kubernetes.io/projected/dc53083d-eadd-4cb8-8633-35e02d400b9c-kube-api-access-mn7kq\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:16:00.855210 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:00.855207 4376 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/576e2c1f-1c80-43a9-a136-3692776642ae-model-cache\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:16:00.855529 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:00.855219 4376 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc53083d-eadd-4cb8-8633-35e02d400b9c-home\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:16:00.855529 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:00.855230 4376 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/576e2c1f-1c80-43a9-a136-3692776642ae-tls-certs\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:16:00.855529 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:00.855241 4376 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc53083d-eadd-4cb8-8633-35e02d400b9c-tmp-dir\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:16:00.855529 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:00.855252 4376 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/576e2c1f-1c80-43a9-a136-3692776642ae-dshm\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:16:00.855529 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:00.855264 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nn6xd\" (UniqueName: \"kubernetes.io/projected/576e2c1f-1c80-43a9-a136-3692776642ae-kube-api-access-nn6xd\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:16:00.855529 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:00.855277 4376 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc53083d-eadd-4cb8-8633-35e02d400b9c-model-cache\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:16:01.107317 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:01.107226 4376 generic.go:356] "Generic (PLEG): container finished" podID="dc53083d-eadd-4cb8-8633-35e02d400b9c" containerID="6c20446894934dcfa9b40993be9fa2a23769d15a81224a240f6cbcc76caba357" exitCode=0 Oct 31 05:16:01.107317 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:01.107268 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c4006d/distinct-2-w4-c4006d-kserve-776b5b899-kpl6z" Oct 31 05:16:01.107558 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:01.107310 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c4006d/distinct-2-w4-c4006d-kserve-776b5b899-kpl6z" event={"ID":"dc53083d-eadd-4cb8-8633-35e02d400b9c","Type":"ContainerDied","Data":"6c20446894934dcfa9b40993be9fa2a23769d15a81224a240f6cbcc76caba357"} Oct 31 05:16:01.107558 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:01.107377 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c4006d/distinct-2-w4-c4006d-kserve-776b5b899-kpl6z" event={"ID":"dc53083d-eadd-4cb8-8633-35e02d400b9c","Type":"ContainerDied","Data":"d94caeb527da1f56f2dca6f903744bf860fea59a34a0ab66b8615737df500ea9"} Oct 31 05:16:01.107558 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:01.107396 4376 scope.go:122] "RemoveContainer" containerID="6c20446894934dcfa9b40993be9fa2a23769d15a81224a240f6cbcc76caba357" Oct 31 05:16:01.109063 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:01.109033 4376 generic.go:356] "Generic (PLEG): container finished" podID="576e2c1f-1c80-43a9-a136-3692776642ae" containerID="91b226397f85d99ec63a941b055c3984264717467120c5a5fa7954382d6cc2fe" exitCode=0 Oct 31 05:16:01.109157 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:01.109070 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c4006d/simulator-w4-c4006d-kserve-595455485b-4h46q" Oct 31 05:16:01.109157 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:01.109100 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c4006d/simulator-w4-c4006d-kserve-595455485b-4h46q" event={"ID":"576e2c1f-1c80-43a9-a136-3692776642ae","Type":"ContainerDied","Data":"91b226397f85d99ec63a941b055c3984264717467120c5a5fa7954382d6cc2fe"} Oct 31 05:16:01.109157 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:01.109125 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c4006d/simulator-w4-c4006d-kserve-595455485b-4h46q" event={"ID":"576e2c1f-1c80-43a9-a136-3692776642ae","Type":"ContainerDied","Data":"78a655ba0aafddf6f2d1f028387bbf418f903e71b3baefc4c69287f9d8fea67b"} Oct 31 05:16:01.117893 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:01.117877 4376 scope.go:122] "RemoveContainer" containerID="6c20446894934dcfa9b40993be9fa2a23769d15a81224a240f6cbcc76caba357" Oct 31 05:16:01.118141 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:16:01.118122 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c20446894934dcfa9b40993be9fa2a23769d15a81224a240f6cbcc76caba357\": container with ID starting with 6c20446894934dcfa9b40993be9fa2a23769d15a81224a240f6cbcc76caba357 not found: ID does not exist" containerID="6c20446894934dcfa9b40993be9fa2a23769d15a81224a240f6cbcc76caba357" Oct 31 05:16:01.118212 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:01.118149 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c20446894934dcfa9b40993be9fa2a23769d15a81224a240f6cbcc76caba357"} err="failed to get container status \"6c20446894934dcfa9b40993be9fa2a23769d15a81224a240f6cbcc76caba357\": rpc error: code = NotFound desc = could not find container \"6c20446894934dcfa9b40993be9fa2a23769d15a81224a240f6cbcc76caba357\": container with ID starting with 6c20446894934dcfa9b40993be9fa2a23769d15a81224a240f6cbcc76caba357 not found: ID does not exist" Oct 31 05:16:01.118212 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:01.118172 4376 scope.go:122] "RemoveContainer" containerID="91b226397f85d99ec63a941b055c3984264717467120c5a5fa7954382d6cc2fe" Oct 31 05:16:01.126445 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:01.126431 4376 scope.go:122] "RemoveContainer" containerID="91b226397f85d99ec63a941b055c3984264717467120c5a5fa7954382d6cc2fe" Oct 31 05:16:01.126702 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:16:01.126681 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91b226397f85d99ec63a941b055c3984264717467120c5a5fa7954382d6cc2fe\": container with ID starting with 91b226397f85d99ec63a941b055c3984264717467120c5a5fa7954382d6cc2fe not found: ID does not exist" containerID="91b226397f85d99ec63a941b055c3984264717467120c5a5fa7954382d6cc2fe" Oct 31 05:16:01.126750 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:01.126709 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91b226397f85d99ec63a941b055c3984264717467120c5a5fa7954382d6cc2fe"} err="failed to get container status \"91b226397f85d99ec63a941b055c3984264717467120c5a5fa7954382d6cc2fe\": rpc error: code = NotFound desc = could not find container \"91b226397f85d99ec63a941b055c3984264717467120c5a5fa7954382d6cc2fe\": container with ID starting with 91b226397f85d99ec63a941b055c3984264717467120c5a5fa7954382d6cc2fe not found: ID does not exist" Oct 31 05:16:01.165039 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:01.165017 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-c4006d/simulator-w4-c4006d-kserve-595455485b-4h46q"] Oct 31 05:16:01.179465 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:01.179446 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-c4006d/simulator-w4-c4006d-kserve-595455485b-4h46q"] Oct 31 05:16:01.208409 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:01.208390 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-c4006d/distinct-2-w4-c4006d-kserve-776b5b899-kpl6z"] Oct 31 05:16:01.212372 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:01.212354 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-c4006d/distinct-2-w4-c4006d-kserve-776b5b899-kpl6z"] Oct 31 05:16:01.539390 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:01.539354 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-afc519-d746ff6cc-npmp9" podUID="e90146f8-ed41-49b8-bf8b-c92bee5f8cdd" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 05:16:01.881733 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:01.881700 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 05:16:02.443144 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:02.443109 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="576e2c1f-1c80-43a9-a136-3692776642ae" path="/var/lib/kubelet/pods/576e2c1f-1c80-43a9-a136-3692776642ae/volumes" Oct 31 05:16:02.443555 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:02.443540 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc53083d-eadd-4cb8-8633-35e02d400b9c" path="/var/lib/kubelet/pods/dc53083d-eadd-4cb8-8633-35e02d400b9c/volumes" Oct 31 05:16:02.549495 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:02.549453 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-afc519-69cbc98467-l7w68" podUID="84d95c12-4dca-44d2-a2cb-fdbfe919135b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 05:16:05.282136 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:05.282100 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-afc519-68c697db57-tkv9n"] Oct 31 05:16:05.282914 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:05.282872 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-afc519-68c697db57-tkv9n" podUID="7b65451e-2626-4b42-9340-d6db06b74760" containerName="maas-api" containerID="cri-o://69b6eebacfbc54c2253bd6f8930c73f35aa4948bae2cf3f93702cb17ccba0979" gracePeriod=30 Oct 31 05:16:05.616941 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:05.616911 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-afc519-68c697db57-tkv9n" Oct 31 05:16:05.652693 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:05.652669 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b65451e-2626-4b42-9340-d6db06b74760-kube-api-access-rzh74\" (UniqueName: \"kubernetes.io/projected/7b65451e-2626-4b42-9340-d6db06b74760-kube-api-access-rzh74\") pod \"7b65451e-2626-4b42-9340-d6db06b74760\" (UID: \"7b65451e-2626-4b42-9340-d6db06b74760\") " Oct 31 05:16:05.652840 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:05.652719 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7b65451e-2626-4b42-9340-d6db06b74760-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b65451e-2626-4b42-9340-d6db06b74760-maas-api-tls\") pod \"7b65451e-2626-4b42-9340-d6db06b74760\" (UID: \"7b65451e-2626-4b42-9340-d6db06b74760\") " Oct 31 05:16:05.652840 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:05.652765 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b65451e-2626-4b42-9340-d6db06b74760-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b65451e-2626-4b42-9340-d6db06b74760-openshift-service-ca\") pod \"7b65451e-2626-4b42-9340-d6db06b74760\" (UID: \"7b65451e-2626-4b42-9340-d6db06b74760\") " Oct 31 05:16:05.652840 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:05.652807 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7b65451e-2626-4b42-9340-d6db06b74760-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b65451e-2626-4b42-9340-d6db06b74760-metrics-certs\") pod \"7b65451e-2626-4b42-9340-d6db06b74760\" (UID: \"7b65451e-2626-4b42-9340-d6db06b74760\") " Oct 31 05:16:05.653153 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:05.653118 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b65451e-2626-4b42-9340-d6db06b74760-openshift-service-ca" pod "7b65451e-2626-4b42-9340-d6db06b74760" (UID: "7b65451e-2626-4b42-9340-d6db06b74760"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:16:05.653275 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:05.653241 4376 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b65451e-2626-4b42-9340-d6db06b74760-openshift-service-ca\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:16:05.654980 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:05.654957 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b65451e-2626-4b42-9340-d6db06b74760-kube-api-access-rzh74" pod "7b65451e-2626-4b42-9340-d6db06b74760" (UID: "7b65451e-2626-4b42-9340-d6db06b74760"). InnerVolumeSpecName "kube-api-access-rzh74". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:16:05.655092 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:05.654991 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b65451e-2626-4b42-9340-d6db06b74760-metrics-certs" pod "7b65451e-2626-4b42-9340-d6db06b74760" (UID: "7b65451e-2626-4b42-9340-d6db06b74760"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 05:16:05.655092 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:05.655041 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b65451e-2626-4b42-9340-d6db06b74760-maas-api-tls" pod "7b65451e-2626-4b42-9340-d6db06b74760" (UID: "7b65451e-2626-4b42-9340-d6db06b74760"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 05:16:05.776911 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:05.776800 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzh74\" (UniqueName: \"kubernetes.io/projected/7b65451e-2626-4b42-9340-d6db06b74760-kube-api-access-rzh74\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:16:05.776911 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:05.776840 4376 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b65451e-2626-4b42-9340-d6db06b74760-maas-api-tls\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:16:05.776911 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:05.776856 4376 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b65451e-2626-4b42-9340-d6db06b74760-metrics-certs\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:16:06.156342 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:06.155914 4376 generic.go:356] "Generic (PLEG): container finished" podID="7b65451e-2626-4b42-9340-d6db06b74760" containerID="69b6eebacfbc54c2253bd6f8930c73f35aa4948bae2cf3f93702cb17ccba0979" exitCode=0 Oct 31 05:16:06.156342 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:06.156161 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-afc519-68c697db57-tkv9n" event={"ID":"7b65451e-2626-4b42-9340-d6db06b74760","Type":"ContainerDied","Data":"69b6eebacfbc54c2253bd6f8930c73f35aa4948bae2cf3f93702cb17ccba0979"} Oct 31 05:16:06.156342 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:06.156288 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-afc519-68c697db57-tkv9n" event={"ID":"7b65451e-2626-4b42-9340-d6db06b74760","Type":"ContainerDied","Data":"bb805b6c5d81cbd928ea8683dd5d8a30fc04758aa7b685697079611a97dfadb5"} Oct 31 05:16:06.156659 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:06.156377 4376 scope.go:122] "RemoveContainer" containerID="69b6eebacfbc54c2253bd6f8930c73f35aa4948bae2cf3f93702cb17ccba0979" Oct 31 05:16:06.156659 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:06.156607 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-afc519-68c697db57-tkv9n" Oct 31 05:16:06.174431 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:06.174402 4376 scope.go:122] "RemoveContainer" containerID="69b6eebacfbc54c2253bd6f8930c73f35aa4948bae2cf3f93702cb17ccba0979" Oct 31 05:16:06.174919 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:16:06.174765 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69b6eebacfbc54c2253bd6f8930c73f35aa4948bae2cf3f93702cb17ccba0979\": container with ID starting with 69b6eebacfbc54c2253bd6f8930c73f35aa4948bae2cf3f93702cb17ccba0979 not found: ID does not exist" containerID="69b6eebacfbc54c2253bd6f8930c73f35aa4948bae2cf3f93702cb17ccba0979" Oct 31 05:16:06.174919 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:06.174810 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69b6eebacfbc54c2253bd6f8930c73f35aa4948bae2cf3f93702cb17ccba0979"} err="failed to get container status \"69b6eebacfbc54c2253bd6f8930c73f35aa4948bae2cf3f93702cb17ccba0979\": rpc error: code = NotFound desc = could not find container \"69b6eebacfbc54c2253bd6f8930c73f35aa4948bae2cf3f93702cb17ccba0979\": container with ID starting with 69b6eebacfbc54c2253bd6f8930c73f35aa4948bae2cf3f93702cb17ccba0979 not found: ID does not exist" Oct 31 05:16:06.219922 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:06.219894 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-afc519-68c697db57-tkv9n"] Oct 31 05:16:06.225472 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:06.225447 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-afc519-68c697db57-tkv9n"] Oct 31 05:16:06.443601 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:06.443529 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b65451e-2626-4b42-9340-d6db06b74760" path="/var/lib/kubelet/pods/7b65451e-2626-4b42-9340-d6db06b74760/volumes" Oct 31 05:16:10.998685 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:16:10.998647 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 05:16:11.539567 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:11.539525 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-afc519-d746ff6cc-npmp9" podUID="e90146f8-ed41-49b8-bf8b-c92bee5f8cdd" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 05:16:12.549271 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:12.549236 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-afc519-69cbc98467-l7w68" podUID="84d95c12-4dca-44d2-a2cb-fdbfe919135b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 05:16:13.810185 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:13.810152 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-8973e2-849bd8687-5kxc2"] Oct 31 05:16:17.381600 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:17.381565 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-afc519-openshift-default-dfdbf9b97-qvdkl"] Oct 31 05:16:17.382054 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:17.381873 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-afc519-openshift-default-dfdbf9b97-qvdkl" podUID="80cdb0ed-7002-415c-9916-7e1650f0481f" containerName="istio-proxy" containerID="cri-o://18a8b0020f24aa5f77fd5ee2049d6452e048d21989c1bb5f75fa399f6b8afcd9" gracePeriod=30 Oct 31 05:16:17.632549 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:17.632519 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-afc519-d746ff6cc-npmp9" Oct 31 05:16:17.671099 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:17.671067 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e90146f8-ed41-49b8-bf8b-c92bee5f8cdd-kube-api-access-wf84q\" (UniqueName: \"kubernetes.io/projected/e90146f8-ed41-49b8-bf8b-c92bee5f8cdd-kube-api-access-wf84q\") pod \"e90146f8-ed41-49b8-bf8b-c92bee5f8cdd\" (UID: \"e90146f8-ed41-49b8-bf8b-c92bee5f8cdd\") " Oct 31 05:16:17.671293 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:17.671143 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e90146f8-ed41-49b8-bf8b-c92bee5f8cdd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e90146f8-ed41-49b8-bf8b-c92bee5f8cdd-plugins-config-volume\") pod \"e90146f8-ed41-49b8-bf8b-c92bee5f8cdd\" (UID: \"e90146f8-ed41-49b8-bf8b-c92bee5f8cdd\") " Oct 31 05:16:17.672369 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:17.672321 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e90146f8-ed41-49b8-bf8b-c92bee5f8cdd-plugins-config-volume" pod "e90146f8-ed41-49b8-bf8b-c92bee5f8cdd" (UID: "e90146f8-ed41-49b8-bf8b-c92bee5f8cdd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:16:17.675215 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:17.675188 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e90146f8-ed41-49b8-bf8b-c92bee5f8cdd-kube-api-access-wf84q" pod "e90146f8-ed41-49b8-bf8b-c92bee5f8cdd" (UID: "e90146f8-ed41-49b8-bf8b-c92bee5f8cdd"). InnerVolumeSpecName "kube-api-access-wf84q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:16:17.689924 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:17.689900 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-afc519-69cbc98467-l7w68" Oct 31 05:16:17.773291 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:17.773260 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84d95c12-4dca-44d2-a2cb-fdbfe919135b-kube-api-access-kpnvh\" (UniqueName: \"kubernetes.io/projected/84d95c12-4dca-44d2-a2cb-fdbfe919135b-kube-api-access-kpnvh\") pod \"84d95c12-4dca-44d2-a2cb-fdbfe919135b\" (UID: \"84d95c12-4dca-44d2-a2cb-fdbfe919135b\") " Oct 31 05:16:17.773464 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:17.773321 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84d95c12-4dca-44d2-a2cb-fdbfe919135b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84d95c12-4dca-44d2-a2cb-fdbfe919135b-plugins-config-volume\") pod \"84d95c12-4dca-44d2-a2cb-fdbfe919135b\" (UID: \"84d95c12-4dca-44d2-a2cb-fdbfe919135b\") " Oct 31 05:16:17.773813 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:17.773776 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84d95c12-4dca-44d2-a2cb-fdbfe919135b-plugins-config-volume" pod "84d95c12-4dca-44d2-a2cb-fdbfe919135b" (UID: "84d95c12-4dca-44d2-a2cb-fdbfe919135b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:16:17.773927 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:17.773912 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wf84q\" (UniqueName: \"kubernetes.io/projected/e90146f8-ed41-49b8-bf8b-c92bee5f8cdd-kube-api-access-wf84q\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:16:17.773991 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:17.773934 4376 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e90146f8-ed41-49b8-bf8b-c92bee5f8cdd-plugins-config-volume\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:16:17.773991 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:17.773949 4376 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84d95c12-4dca-44d2-a2cb-fdbfe919135b-plugins-config-volume\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:16:17.775554 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:17.775531 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84d95c12-4dca-44d2-a2cb-fdbfe919135b-kube-api-access-kpnvh" pod "84d95c12-4dca-44d2-a2cb-fdbfe919135b" (UID: "84d95c12-4dca-44d2-a2cb-fdbfe919135b"). InnerVolumeSpecName "kube-api-access-kpnvh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:16:17.874913 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:17.874890 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kpnvh\" (UniqueName: \"kubernetes.io/projected/84d95c12-4dca-44d2-a2cb-fdbfe919135b-kube-api-access-kpnvh\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:16:18.229023 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:18.228932 4376 generic.go:356] "Generic (PLEG): container finished" podID="84d95c12-4dca-44d2-a2cb-fdbfe919135b" containerID="53c4edd38ab68b826ecbed62aa5bff0dd9e26fd76265fa486ad478d6b0967fa0" exitCode=0 Oct 31 05:16:18.229023 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:18.228985 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-afc519-69cbc98467-l7w68" Oct 31 05:16:18.229280 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:18.229018 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-afc519-69cbc98467-l7w68" event={"ID":"84d95c12-4dca-44d2-a2cb-fdbfe919135b","Type":"ContainerDied","Data":"53c4edd38ab68b826ecbed62aa5bff0dd9e26fd76265fa486ad478d6b0967fa0"} Oct 31 05:16:18.229280 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:18.229094 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-afc519-69cbc98467-l7w68" event={"ID":"84d95c12-4dca-44d2-a2cb-fdbfe919135b","Type":"ContainerDied","Data":"af8b3658eb011aae6b133d27fa92b5fbc7d8fa781c8c3e5502b79d50f67d3564"} Oct 31 05:16:18.229280 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:18.229132 4376 scope.go:122] "RemoveContainer" containerID="53c4edd38ab68b826ecbed62aa5bff0dd9e26fd76265fa486ad478d6b0967fa0" Oct 31 05:16:18.230389 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:18.230344 4376 generic.go:356] "Generic (PLEG): container finished" podID="e90146f8-ed41-49b8-bf8b-c92bee5f8cdd" containerID="2a7a08c67ee09683a03dcda6277fc82fa6c5690a205555171d927bbb2ef54aa7" exitCode=0 Oct 31 05:16:18.230482 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:18.230389 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-afc519-d746ff6cc-npmp9" Oct 31 05:16:18.230482 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:18.230416 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-afc519-d746ff6cc-npmp9" event={"ID":"e90146f8-ed41-49b8-bf8b-c92bee5f8cdd","Type":"ContainerDied","Data":"2a7a08c67ee09683a03dcda6277fc82fa6c5690a205555171d927bbb2ef54aa7"} Oct 31 05:16:18.230482 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:18.230472 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-afc519-d746ff6cc-npmp9" event={"ID":"e90146f8-ed41-49b8-bf8b-c92bee5f8cdd","Type":"ContainerDied","Data":"08694f61e7057f1688d6b89f1a4dab93c4996799c941c7679a38fe83c72f29e7"} Oct 31 05:16:18.239169 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:18.239155 4376 scope.go:122] "RemoveContainer" containerID="53c4edd38ab68b826ecbed62aa5bff0dd9e26fd76265fa486ad478d6b0967fa0" Oct 31 05:16:18.239519 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:16:18.239444 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53c4edd38ab68b826ecbed62aa5bff0dd9e26fd76265fa486ad478d6b0967fa0\": container with ID starting with 53c4edd38ab68b826ecbed62aa5bff0dd9e26fd76265fa486ad478d6b0967fa0 not found: ID does not exist" containerID="53c4edd38ab68b826ecbed62aa5bff0dd9e26fd76265fa486ad478d6b0967fa0" Oct 31 05:16:18.239519 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:18.239467 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53c4edd38ab68b826ecbed62aa5bff0dd9e26fd76265fa486ad478d6b0967fa0"} err="failed to get container status \"53c4edd38ab68b826ecbed62aa5bff0dd9e26fd76265fa486ad478d6b0967fa0\": rpc error: code = NotFound desc = could not find container \"53c4edd38ab68b826ecbed62aa5bff0dd9e26fd76265fa486ad478d6b0967fa0\": container with ID starting with 53c4edd38ab68b826ecbed62aa5bff0dd9e26fd76265fa486ad478d6b0967fa0 not found: ID does not exist" Oct 31 05:16:18.239519 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:18.239487 4376 scope.go:122] "RemoveContainer" containerID="2a7a08c67ee09683a03dcda6277fc82fa6c5690a205555171d927bbb2ef54aa7" Oct 31 05:16:18.248388 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:18.248372 4376 scope.go:122] "RemoveContainer" containerID="2a7a08c67ee09683a03dcda6277fc82fa6c5690a205555171d927bbb2ef54aa7" Oct 31 05:16:18.248633 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:16:18.248615 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a7a08c67ee09683a03dcda6277fc82fa6c5690a205555171d927bbb2ef54aa7\": container with ID starting with 2a7a08c67ee09683a03dcda6277fc82fa6c5690a205555171d927bbb2ef54aa7 not found: ID does not exist" containerID="2a7a08c67ee09683a03dcda6277fc82fa6c5690a205555171d927bbb2ef54aa7" Oct 31 05:16:18.248694 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:18.248640 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a7a08c67ee09683a03dcda6277fc82fa6c5690a205555171d927bbb2ef54aa7"} err="failed to get container status \"2a7a08c67ee09683a03dcda6277fc82fa6c5690a205555171d927bbb2ef54aa7\": rpc error: code = NotFound desc = could not find container \"2a7a08c67ee09683a03dcda6277fc82fa6c5690a205555171d927bbb2ef54aa7\": container with ID starting with 2a7a08c67ee09683a03dcda6277fc82fa6c5690a205555171d927bbb2ef54aa7 not found: ID does not exist" Oct 31 05:16:18.285879 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:18.285857 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-afc519-d746ff6cc-npmp9"] Oct 31 05:16:18.299929 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:18.299907 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-afc519-d746ff6cc-npmp9"] Oct 31 05:16:18.327068 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:18.327047 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-afc519-69cbc98467-l7w68"] Oct 31 05:16:18.330433 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:18.330413 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-afc519-69cbc98467-l7w68"] Oct 31 05:16:18.442867 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:18.442842 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84d95c12-4dca-44d2-a2cb-fdbfe919135b" path="/var/lib/kubelet/pods/84d95c12-4dca-44d2-a2cb-fdbfe919135b/volumes" Oct 31 05:16:18.443232 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:18.443218 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e90146f8-ed41-49b8-bf8b-c92bee5f8cdd" path="/var/lib/kubelet/pods/e90146f8-ed41-49b8-bf8b-c92bee5f8cdd/volumes" Oct 31 05:16:18.810890 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:18.810851 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-8973e2-849bd8687-5kxc2" podUID="2a9e4834-09d5-4add-9004-04c30da0527c" containerName="payload-processing" containerID="cri-o://cf0df277d55a85c5e084bb3a35e3afd985254f6fe7bdb826bd7737a7e3a50f86" gracePeriod=25 Oct 31 05:16:19.044318 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:19.044279 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-c4006d-b7ccb647b-v7tsw"] Oct 31 05:16:19.044603 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:19.044551 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c4006d-b7ccb647b-v7tsw" podUID="ba6f1fd8-1985-4ea5-9097-36263d48aa13" containerName="maas-api" containerID="cri-o://4e6065771759e51f9b8e0f1cb6ce9ff5710a05f7842243ffa01a13f582b4d6c3" gracePeriod=30 Oct 31 05:16:19.241141 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:19.241108 4376 generic.go:356] "Generic (PLEG): container finished" podID="ba6f1fd8-1985-4ea5-9097-36263d48aa13" containerID="4e6065771759e51f9b8e0f1cb6ce9ff5710a05f7842243ffa01a13f582b4d6c3" exitCode=0 Oct 31 05:16:19.241290 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:19.241174 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c4006d-b7ccb647b-v7tsw" event={"ID":"ba6f1fd8-1985-4ea5-9097-36263d48aa13","Type":"ContainerDied","Data":"4e6065771759e51f9b8e0f1cb6ce9ff5710a05f7842243ffa01a13f582b4d6c3"} Oct 31 05:16:19.292476 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:19.292455 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c4006d-b7ccb647b-v7tsw" Oct 31 05:16:19.393749 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:19.393718 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ba6f1fd8-1985-4ea5-9097-36263d48aa13-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba6f1fd8-1985-4ea5-9097-36263d48aa13-metrics-certs\") pod \"ba6f1fd8-1985-4ea5-9097-36263d48aa13\" (UID: \"ba6f1fd8-1985-4ea5-9097-36263d48aa13\") " Oct 31 05:16:19.393953 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:19.393789 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba6f1fd8-1985-4ea5-9097-36263d48aa13-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba6f1fd8-1985-4ea5-9097-36263d48aa13-openshift-service-ca\") pod \"ba6f1fd8-1985-4ea5-9097-36263d48aa13\" (UID: \"ba6f1fd8-1985-4ea5-9097-36263d48aa13\") " Oct 31 05:16:19.393953 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:19.393858 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ba6f1fd8-1985-4ea5-9097-36263d48aa13-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ba6f1fd8-1985-4ea5-9097-36263d48aa13-maas-api-tls\") pod \"ba6f1fd8-1985-4ea5-9097-36263d48aa13\" (UID: \"ba6f1fd8-1985-4ea5-9097-36263d48aa13\") " Oct 31 05:16:19.393953 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:19.393912 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba6f1fd8-1985-4ea5-9097-36263d48aa13-kube-api-access-bp65b\" (UniqueName: \"kubernetes.io/projected/ba6f1fd8-1985-4ea5-9097-36263d48aa13-kube-api-access-bp65b\") pod \"ba6f1fd8-1985-4ea5-9097-36263d48aa13\" (UID: \"ba6f1fd8-1985-4ea5-9097-36263d48aa13\") " Oct 31 05:16:19.394242 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:19.394212 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba6f1fd8-1985-4ea5-9097-36263d48aa13-openshift-service-ca" pod "ba6f1fd8-1985-4ea5-9097-36263d48aa13" (UID: "ba6f1fd8-1985-4ea5-9097-36263d48aa13"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:16:19.394402 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:19.394385 4376 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba6f1fd8-1985-4ea5-9097-36263d48aa13-openshift-service-ca\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:16:19.396364 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:19.396306 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba6f1fd8-1985-4ea5-9097-36263d48aa13-kube-api-access-bp65b" pod "ba6f1fd8-1985-4ea5-9097-36263d48aa13" (UID: "ba6f1fd8-1985-4ea5-9097-36263d48aa13"). InnerVolumeSpecName "kube-api-access-bp65b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:16:19.396638 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:19.396612 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba6f1fd8-1985-4ea5-9097-36263d48aa13-maas-api-tls" pod "ba6f1fd8-1985-4ea5-9097-36263d48aa13" (UID: "ba6f1fd8-1985-4ea5-9097-36263d48aa13"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 05:16:19.396801 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:19.396778 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba6f1fd8-1985-4ea5-9097-36263d48aa13-metrics-certs" pod "ba6f1fd8-1985-4ea5-9097-36263d48aa13" (UID: "ba6f1fd8-1985-4ea5-9097-36263d48aa13"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 05:16:19.513358 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:19.509242 4376 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ba6f1fd8-1985-4ea5-9097-36263d48aa13-maas-api-tls\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:16:19.513358 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:19.509279 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bp65b\" (UniqueName: \"kubernetes.io/projected/ba6f1fd8-1985-4ea5-9097-36263d48aa13-kube-api-access-bp65b\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:16:19.513358 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:19.509295 4376 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba6f1fd8-1985-4ea5-9097-36263d48aa13-metrics-certs\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:16:20.247747 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:20.247719 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c4006d-b7ccb647b-v7tsw" Oct 31 05:16:20.247970 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:20.247753 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c4006d-b7ccb647b-v7tsw" event={"ID":"ba6f1fd8-1985-4ea5-9097-36263d48aa13","Type":"ContainerDied","Data":"16850602a80a7fb971fc33fc22e09155f9f1d7a274bb091db47dce0362a26252"} Oct 31 05:16:20.247970 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:20.247822 4376 scope.go:122] "RemoveContainer" containerID="4e6065771759e51f9b8e0f1cb6ce9ff5710a05f7842243ffa01a13f582b4d6c3" Oct 31 05:16:20.304919 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:20.304895 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-c4006d-b7ccb647b-v7tsw"] Oct 31 05:16:20.319079 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:20.319055 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-c4006d-b7ccb647b-v7tsw"] Oct 31 05:16:20.443577 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:20.443545 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba6f1fd8-1985-4ea5-9097-36263d48aa13" path="/var/lib/kubelet/pods/ba6f1fd8-1985-4ea5-9097-36263d48aa13/volumes" Oct 31 05:16:22.672038 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:22.671942 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-afc519-openshift-default-dfdbf9b97-qvdkl" Oct 31 05:16:22.750663 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:22.750628 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80cdb0ed-7002-415c-9916-7e1650f0481f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80cdb0ed-7002-415c-9916-7e1650f0481f-workload-certs\") pod \"80cdb0ed-7002-415c-9916-7e1650f0481f\" (UID: \"80cdb0ed-7002-415c-9916-7e1650f0481f\") " Oct 31 05:16:22.750850 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:22.750679 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80cdb0ed-7002-415c-9916-7e1650f0481f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80cdb0ed-7002-415c-9916-7e1650f0481f-istio-data\") pod \"80cdb0ed-7002-415c-9916-7e1650f0481f\" (UID: \"80cdb0ed-7002-415c-9916-7e1650f0481f\") " Oct 31 05:16:22.750850 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:22.750708 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80cdb0ed-7002-415c-9916-7e1650f0481f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80cdb0ed-7002-415c-9916-7e1650f0481f-istiod-ca-cert\") pod \"80cdb0ed-7002-415c-9916-7e1650f0481f\" (UID: \"80cdb0ed-7002-415c-9916-7e1650f0481f\") " Oct 31 05:16:22.750850 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:22.750755 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80cdb0ed-7002-415c-9916-7e1650f0481f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80cdb0ed-7002-415c-9916-7e1650f0481f-workload-socket\") pod \"80cdb0ed-7002-415c-9916-7e1650f0481f\" (UID: \"80cdb0ed-7002-415c-9916-7e1650f0481f\") " Oct 31 05:16:22.750850 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:22.750800 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80cdb0ed-7002-415c-9916-7e1650f0481f-istio-token\" (UniqueName: \"kubernetes.io/projected/80cdb0ed-7002-415c-9916-7e1650f0481f-istio-token\") pod \"80cdb0ed-7002-415c-9916-7e1650f0481f\" (UID: \"80cdb0ed-7002-415c-9916-7e1650f0481f\") " Oct 31 05:16:22.751068 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:22.750861 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80cdb0ed-7002-415c-9916-7e1650f0481f-kube-api-access-jqdqk\" (UniqueName: \"kubernetes.io/projected/80cdb0ed-7002-415c-9916-7e1650f0481f-kube-api-access-jqdqk\") pod \"80cdb0ed-7002-415c-9916-7e1650f0481f\" (UID: \"80cdb0ed-7002-415c-9916-7e1650f0481f\") " Oct 31 05:16:22.751068 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:22.750965 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80cdb0ed-7002-415c-9916-7e1650f0481f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80cdb0ed-7002-415c-9916-7e1650f0481f-istio-envoy\") pod \"80cdb0ed-7002-415c-9916-7e1650f0481f\" (UID: \"80cdb0ed-7002-415c-9916-7e1650f0481f\") " Oct 31 05:16:22.751068 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:22.751033 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/80cdb0ed-7002-415c-9916-7e1650f0481f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80cdb0ed-7002-415c-9916-7e1650f0481f-istio-podinfo\") pod \"80cdb0ed-7002-415c-9916-7e1650f0481f\" (UID: \"80cdb0ed-7002-415c-9916-7e1650f0481f\") " Oct 31 05:16:22.751223 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:22.751067 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80cdb0ed-7002-415c-9916-7e1650f0481f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80cdb0ed-7002-415c-9916-7e1650f0481f-credential-socket\") pod \"80cdb0ed-7002-415c-9916-7e1650f0481f\" (UID: \"80cdb0ed-7002-415c-9916-7e1650f0481f\") " Oct 31 05:16:22.751649 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:22.751621 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80cdb0ed-7002-415c-9916-7e1650f0481f-credential-socket" pod "80cdb0ed-7002-415c-9916-7e1650f0481f" (UID: "80cdb0ed-7002-415c-9916-7e1650f0481f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:16:22.751842 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:22.751824 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80cdb0ed-7002-415c-9916-7e1650f0481f-workload-certs" pod "80cdb0ed-7002-415c-9916-7e1650f0481f" (UID: "80cdb0ed-7002-415c-9916-7e1650f0481f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:16:22.751999 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:22.751983 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80cdb0ed-7002-415c-9916-7e1650f0481f-istio-data" pod "80cdb0ed-7002-415c-9916-7e1650f0481f" (UID: "80cdb0ed-7002-415c-9916-7e1650f0481f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:16:22.752227 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:22.752210 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80cdb0ed-7002-415c-9916-7e1650f0481f-istiod-ca-cert" pod "80cdb0ed-7002-415c-9916-7e1650f0481f" (UID: "80cdb0ed-7002-415c-9916-7e1650f0481f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:16:22.752433 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:22.752416 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80cdb0ed-7002-415c-9916-7e1650f0481f-workload-socket" pod "80cdb0ed-7002-415c-9916-7e1650f0481f" (UID: "80cdb0ed-7002-415c-9916-7e1650f0481f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:16:22.755028 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:22.755001 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80cdb0ed-7002-415c-9916-7e1650f0481f-istio-token" pod "80cdb0ed-7002-415c-9916-7e1650f0481f" (UID: "80cdb0ed-7002-415c-9916-7e1650f0481f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:16:22.756236 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:22.756201 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/80cdb0ed-7002-415c-9916-7e1650f0481f-istio-podinfo" pod "80cdb0ed-7002-415c-9916-7e1650f0481f" (UID: "80cdb0ed-7002-415c-9916-7e1650f0481f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 05:16:22.756587 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:22.756535 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80cdb0ed-7002-415c-9916-7e1650f0481f-istio-envoy" pod "80cdb0ed-7002-415c-9916-7e1650f0481f" (UID: "80cdb0ed-7002-415c-9916-7e1650f0481f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:16:22.756587 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:22.756540 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80cdb0ed-7002-415c-9916-7e1650f0481f-kube-api-access-jqdqk" pod "80cdb0ed-7002-415c-9916-7e1650f0481f" (UID: "80cdb0ed-7002-415c-9916-7e1650f0481f"). InnerVolumeSpecName "kube-api-access-jqdqk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:16:22.852282 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:22.852255 4376 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80cdb0ed-7002-415c-9916-7e1650f0481f-workload-certs\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:16:22.852282 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:22.852281 4376 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80cdb0ed-7002-415c-9916-7e1650f0481f-istio-data\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:16:22.852282 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:22.852289 4376 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80cdb0ed-7002-415c-9916-7e1650f0481f-istiod-ca-cert\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:16:22.852537 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:22.852299 4376 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80cdb0ed-7002-415c-9916-7e1650f0481f-workload-socket\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:16:22.852537 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:22.852308 4376 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80cdb0ed-7002-415c-9916-7e1650f0481f-istio-token\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:16:22.852537 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:22.852317 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqdqk\" (UniqueName: \"kubernetes.io/projected/80cdb0ed-7002-415c-9916-7e1650f0481f-kube-api-access-jqdqk\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:16:22.852537 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:22.852338 4376 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80cdb0ed-7002-415c-9916-7e1650f0481f-istio-envoy\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:16:22.852537 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:22.852348 4376 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80cdb0ed-7002-415c-9916-7e1650f0481f-istio-podinfo\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:16:22.852537 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:22.852355 4376 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80cdb0ed-7002-415c-9916-7e1650f0481f-credential-socket\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:16:23.298765 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:23.298725 4376 generic.go:356] "Generic (PLEG): container finished" podID="80cdb0ed-7002-415c-9916-7e1650f0481f" containerID="18a8b0020f24aa5f77fd5ee2049d6452e048d21989c1bb5f75fa399f6b8afcd9" exitCode=0 Oct 31 05:16:23.298946 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:23.298918 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-afc519-openshift-default-dfdbf9b97-qvdkl" event={"ID":"80cdb0ed-7002-415c-9916-7e1650f0481f","Type":"ContainerDied","Data":"18a8b0020f24aa5f77fd5ee2049d6452e048d21989c1bb5f75fa399f6b8afcd9"} Oct 31 05:16:23.299020 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:23.298971 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-afc519-openshift-default-dfdbf9b97-qvdkl" event={"ID":"80cdb0ed-7002-415c-9916-7e1650f0481f","Type":"ContainerDied","Data":"7231d9c2596b6d0665d5ee72a2280c2236a915adb081f2a04af30e3a21654cb3"} Oct 31 05:16:23.299020 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:23.298999 4376 scope.go:122] "RemoveContainer" containerID="18a8b0020f24aa5f77fd5ee2049d6452e048d21989c1bb5f75fa399f6b8afcd9" Oct 31 05:16:23.299189 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:23.299165 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-afc519-openshift-default-dfdbf9b97-qvdkl" Oct 31 05:16:23.313735 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:23.313712 4376 scope.go:122] "RemoveContainer" containerID="f7752764b4fabf3ad4cc5192c8e0217f044eed7de2108ea7b914eecd9885e170" Oct 31 05:16:23.329156 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:23.329069 4376 scope.go:122] "RemoveContainer" containerID="18a8b0020f24aa5f77fd5ee2049d6452e048d21989c1bb5f75fa399f6b8afcd9" Oct 31 05:16:23.329496 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:16:23.329420 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18a8b0020f24aa5f77fd5ee2049d6452e048d21989c1bb5f75fa399f6b8afcd9\": container with ID starting with 18a8b0020f24aa5f77fd5ee2049d6452e048d21989c1bb5f75fa399f6b8afcd9 not found: ID does not exist" containerID="18a8b0020f24aa5f77fd5ee2049d6452e048d21989c1bb5f75fa399f6b8afcd9" Oct 31 05:16:23.329496 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:23.329464 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18a8b0020f24aa5f77fd5ee2049d6452e048d21989c1bb5f75fa399f6b8afcd9"} err="failed to get container status \"18a8b0020f24aa5f77fd5ee2049d6452e048d21989c1bb5f75fa399f6b8afcd9\": rpc error: code = NotFound desc = could not find container \"18a8b0020f24aa5f77fd5ee2049d6452e048d21989c1bb5f75fa399f6b8afcd9\": container with ID starting with 18a8b0020f24aa5f77fd5ee2049d6452e048d21989c1bb5f75fa399f6b8afcd9 not found: ID does not exist" Oct 31 05:16:23.329496 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:23.329487 4376 scope.go:122] "RemoveContainer" containerID="f7752764b4fabf3ad4cc5192c8e0217f044eed7de2108ea7b914eecd9885e170" Oct 31 05:16:23.329967 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:16:23.329923 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7752764b4fabf3ad4cc5192c8e0217f044eed7de2108ea7b914eecd9885e170\": container with ID starting with f7752764b4fabf3ad4cc5192c8e0217f044eed7de2108ea7b914eecd9885e170 not found: ID does not exist" containerID="f7752764b4fabf3ad4cc5192c8e0217f044eed7de2108ea7b914eecd9885e170" Oct 31 05:16:23.330095 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:23.329977 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7752764b4fabf3ad4cc5192c8e0217f044eed7de2108ea7b914eecd9885e170"} err="failed to get container status \"f7752764b4fabf3ad4cc5192c8e0217f044eed7de2108ea7b914eecd9885e170\": rpc error: code = NotFound desc = could not find container \"f7752764b4fabf3ad4cc5192c8e0217f044eed7de2108ea7b914eecd9885e170\": container with ID starting with f7752764b4fabf3ad4cc5192c8e0217f044eed7de2108ea7b914eecd9885e170 not found: ID does not exist" Oct 31 05:16:23.365533 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:23.365511 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-afc519-openshift-default-dfdbf9b97-qvdkl"] Oct 31 05:16:23.379855 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:23.379831 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-afc519-openshift-default-dfdbf9b97-qvdkl"] Oct 31 05:16:24.443675 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:24.443639 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80cdb0ed-7002-415c-9916-7e1650f0481f" path="/var/lib/kubelet/pods/80cdb0ed-7002-415c-9916-7e1650f0481f/volumes" Oct 31 05:16:26.484726 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:26.484690 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-brpvl"] Oct 31 05:16:26.485241 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:26.484937 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-brpvl" podUID="6957320f-9d86-4430-bca7-85db00d7a099" containerName="istio-proxy" containerID="cri-o://ca88dfd607dbbf2878cd20e8f583c5ecee8952003e85169731c0a12d701e9315" gracePeriod=30 Oct 31 05:16:27.717149 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:27.717114 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-8973e2-849bd8687-5kxc2" podUID="2a9e4834-09d5-4add-9004-04c30da0527c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 05:16:29.031176 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:29.031136 4376 patch_prober.go:29] interesting pod/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-brpvl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 05:16:29.031608 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:29.031197 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-brpvl" podUID="6957320f-9d86-4430-bca7-85db00d7a099" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 05:16:29.836419 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:29.836387 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-227c93-74b5c495fc-v4rxp"] Oct 31 05:16:30.035309 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:30.035275 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-a99f2a-6df854df67-qh6wn"] Oct 31 05:16:30.035786 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:30.035482 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a99f2a-6df854df67-qh6wn" podUID="b3e4def5-df8d-42d2-810c-8fa6764d338b" containerName="maas-api" containerID="cri-o://ea82adb5b66aaaa5e43624dfb39d35e0b275a57c8eb9b3661aff0a890ee75b35" gracePeriod=30 Oct 31 05:16:30.317024 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:30.317001 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a99f2a-6df854df67-qh6wn" Oct 31 05:16:30.352711 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:30.352683 4376 generic.go:356] "Generic (PLEG): container finished" podID="b3e4def5-df8d-42d2-810c-8fa6764d338b" containerID="ea82adb5b66aaaa5e43624dfb39d35e0b275a57c8eb9b3661aff0a890ee75b35" exitCode=0 Oct 31 05:16:30.352891 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:30.352734 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a99f2a-6df854df67-qh6wn" Oct 31 05:16:30.352891 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:30.352777 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a99f2a-6df854df67-qh6wn" event={"ID":"b3e4def5-df8d-42d2-810c-8fa6764d338b","Type":"ContainerDied","Data":"ea82adb5b66aaaa5e43624dfb39d35e0b275a57c8eb9b3661aff0a890ee75b35"} Oct 31 05:16:30.352891 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:30.352850 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a99f2a-6df854df67-qh6wn" event={"ID":"b3e4def5-df8d-42d2-810c-8fa6764d338b","Type":"ContainerDied","Data":"44c9cf30b79ebe8205d5a17fdf58ad2c4e8a3dd40633e46f7ffd52009b86114d"} Oct 31 05:16:30.352891 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:30.352886 4376 scope.go:122] "RemoveContainer" containerID="ea82adb5b66aaaa5e43624dfb39d35e0b275a57c8eb9b3661aff0a890ee75b35" Oct 31 05:16:30.362392 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:30.362372 4376 scope.go:122] "RemoveContainer" containerID="ea82adb5b66aaaa5e43624dfb39d35e0b275a57c8eb9b3661aff0a890ee75b35" Oct 31 05:16:30.368180 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:16:30.363244 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea82adb5b66aaaa5e43624dfb39d35e0b275a57c8eb9b3661aff0a890ee75b35\": container with ID starting with ea82adb5b66aaaa5e43624dfb39d35e0b275a57c8eb9b3661aff0a890ee75b35 not found: ID does not exist" containerID="ea82adb5b66aaaa5e43624dfb39d35e0b275a57c8eb9b3661aff0a890ee75b35" Oct 31 05:16:30.368180 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:30.363288 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea82adb5b66aaaa5e43624dfb39d35e0b275a57c8eb9b3661aff0a890ee75b35"} err="failed to get container status \"ea82adb5b66aaaa5e43624dfb39d35e0b275a57c8eb9b3661aff0a890ee75b35\": rpc error: code = NotFound desc = could not find container \"ea82adb5b66aaaa5e43624dfb39d35e0b275a57c8eb9b3661aff0a890ee75b35\": container with ID starting with ea82adb5b66aaaa5e43624dfb39d35e0b275a57c8eb9b3661aff0a890ee75b35 not found: ID does not exist" Oct 31 05:16:30.368180 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:30.367030 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b3e4def5-df8d-42d2-810c-8fa6764d338b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3e4def5-df8d-42d2-810c-8fa6764d338b-maas-api-tls\") pod \"b3e4def5-df8d-42d2-810c-8fa6764d338b\" (UID: \"b3e4def5-df8d-42d2-810c-8fa6764d338b\") " Oct 31 05:16:30.368180 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:30.367140 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3e4def5-df8d-42d2-810c-8fa6764d338b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3e4def5-df8d-42d2-810c-8fa6764d338b-openshift-service-ca\") pod \"b3e4def5-df8d-42d2-810c-8fa6764d338b\" (UID: \"b3e4def5-df8d-42d2-810c-8fa6764d338b\") " Oct 31 05:16:30.368180 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:30.367221 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3e4def5-df8d-42d2-810c-8fa6764d338b-kube-api-access-9gjrh\" (UniqueName: \"kubernetes.io/projected/b3e4def5-df8d-42d2-810c-8fa6764d338b-kube-api-access-9gjrh\") pod \"b3e4def5-df8d-42d2-810c-8fa6764d338b\" (UID: \"b3e4def5-df8d-42d2-810c-8fa6764d338b\") " Oct 31 05:16:30.368180 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:30.367306 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b3e4def5-df8d-42d2-810c-8fa6764d338b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3e4def5-df8d-42d2-810c-8fa6764d338b-metrics-certs\") pod \"b3e4def5-df8d-42d2-810c-8fa6764d338b\" (UID: \"b3e4def5-df8d-42d2-810c-8fa6764d338b\") " Oct 31 05:16:30.369070 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:30.368796 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3e4def5-df8d-42d2-810c-8fa6764d338b-openshift-service-ca" pod "b3e4def5-df8d-42d2-810c-8fa6764d338b" (UID: "b3e4def5-df8d-42d2-810c-8fa6764d338b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:16:30.372999 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:30.371272 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3e4def5-df8d-42d2-810c-8fa6764d338b-metrics-certs" pod "b3e4def5-df8d-42d2-810c-8fa6764d338b" (UID: "b3e4def5-df8d-42d2-810c-8fa6764d338b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 05:16:30.372999 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:30.371308 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3e4def5-df8d-42d2-810c-8fa6764d338b-maas-api-tls" pod "b3e4def5-df8d-42d2-810c-8fa6764d338b" (UID: "b3e4def5-df8d-42d2-810c-8fa6764d338b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 05:16:30.373424 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:30.373386 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3e4def5-df8d-42d2-810c-8fa6764d338b-kube-api-access-9gjrh" pod "b3e4def5-df8d-42d2-810c-8fa6764d338b" (UID: "b3e4def5-df8d-42d2-810c-8fa6764d338b"). InnerVolumeSpecName "kube-api-access-9gjrh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:16:30.468619 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:30.468586 4376 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3e4def5-df8d-42d2-810c-8fa6764d338b-metrics-certs\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:16:30.468619 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:30.468615 4376 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3e4def5-df8d-42d2-810c-8fa6764d338b-maas-api-tls\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:16:30.468619 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:30.468625 4376 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3e4def5-df8d-42d2-810c-8fa6764d338b-openshift-service-ca\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:16:30.468872 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:30.468634 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9gjrh\" (UniqueName: \"kubernetes.io/projected/b3e4def5-df8d-42d2-810c-8fa6764d338b-kube-api-access-9gjrh\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:16:30.697177 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:30.697142 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-a99f2a-6df854df67-qh6wn"] Oct 31 05:16:30.711151 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:30.711088 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-a99f2a-6df854df67-qh6wn"] Oct 31 05:16:31.744511 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:31.744486 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-brpvl" Oct 31 05:16:31.888089 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:31.888061 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6957320f-9d86-4430-bca7-85db00d7a099-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6957320f-9d86-4430-bca7-85db00d7a099-workload-socket\") pod \"6957320f-9d86-4430-bca7-85db00d7a099\" (UID: \"6957320f-9d86-4430-bca7-85db00d7a099\") " Oct 31 05:16:31.888252 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:31.888108 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6957320f-9d86-4430-bca7-85db00d7a099-kube-api-access-dq2k2\" (UniqueName: \"kubernetes.io/projected/6957320f-9d86-4430-bca7-85db00d7a099-kube-api-access-dq2k2\") pod \"6957320f-9d86-4430-bca7-85db00d7a099\" (UID: \"6957320f-9d86-4430-bca7-85db00d7a099\") " Oct 31 05:16:31.888252 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:31.888175 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6957320f-9d86-4430-bca7-85db00d7a099-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6957320f-9d86-4430-bca7-85db00d7a099-istiod-ca-cert\") pod \"6957320f-9d86-4430-bca7-85db00d7a099\" (UID: \"6957320f-9d86-4430-bca7-85db00d7a099\") " Oct 31 05:16:31.888359 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:31.888320 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6957320f-9d86-4430-bca7-85db00d7a099-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6957320f-9d86-4430-bca7-85db00d7a099-istio-envoy\") pod \"6957320f-9d86-4430-bca7-85db00d7a099\" (UID: \"6957320f-9d86-4430-bca7-85db00d7a099\") " Oct 31 05:16:31.888414 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:31.888378 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6957320f-9d86-4430-bca7-85db00d7a099-workload-socket" pod "6957320f-9d86-4430-bca7-85db00d7a099" (UID: "6957320f-9d86-4430-bca7-85db00d7a099"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:16:31.888414 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:31.888400 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6957320f-9d86-4430-bca7-85db00d7a099-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6957320f-9d86-4430-bca7-85db00d7a099-workload-certs\") pod \"6957320f-9d86-4430-bca7-85db00d7a099\" (UID: \"6957320f-9d86-4430-bca7-85db00d7a099\") " Oct 31 05:16:31.888500 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:31.888445 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6957320f-9d86-4430-bca7-85db00d7a099-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6957320f-9d86-4430-bca7-85db00d7a099-istio-data\") pod \"6957320f-9d86-4430-bca7-85db00d7a099\" (UID: \"6957320f-9d86-4430-bca7-85db00d7a099\") " Oct 31 05:16:31.888500 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:31.888476 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6957320f-9d86-4430-bca7-85db00d7a099-istio-token\" (UniqueName: \"kubernetes.io/projected/6957320f-9d86-4430-bca7-85db00d7a099-istio-token\") pod \"6957320f-9d86-4430-bca7-85db00d7a099\" (UID: \"6957320f-9d86-4430-bca7-85db00d7a099\") " Oct 31 05:16:31.888602 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:31.888516 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6957320f-9d86-4430-bca7-85db00d7a099-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6957320f-9d86-4430-bca7-85db00d7a099-istio-podinfo\") pod \"6957320f-9d86-4430-bca7-85db00d7a099\" (UID: \"6957320f-9d86-4430-bca7-85db00d7a099\") " Oct 31 05:16:31.888602 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:31.888558 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6957320f-9d86-4430-bca7-85db00d7a099-istiod-ca-cert" pod "6957320f-9d86-4430-bca7-85db00d7a099" (UID: "6957320f-9d86-4430-bca7-85db00d7a099"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:16:31.888602 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:31.888570 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6957320f-9d86-4430-bca7-85db00d7a099-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6957320f-9d86-4430-bca7-85db00d7a099-credential-socket\") pod \"6957320f-9d86-4430-bca7-85db00d7a099\" (UID: \"6957320f-9d86-4430-bca7-85db00d7a099\") " Oct 31 05:16:31.888784 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:31.888652 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6957320f-9d86-4430-bca7-85db00d7a099-workload-certs" pod "6957320f-9d86-4430-bca7-85db00d7a099" (UID: "6957320f-9d86-4430-bca7-85db00d7a099"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:16:31.888847 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:31.888775 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6957320f-9d86-4430-bca7-85db00d7a099-istio-data" pod "6957320f-9d86-4430-bca7-85db00d7a099" (UID: "6957320f-9d86-4430-bca7-85db00d7a099"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:16:31.888985 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:31.888958 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6957320f-9d86-4430-bca7-85db00d7a099-credential-socket" pod "6957320f-9d86-4430-bca7-85db00d7a099" (UID: "6957320f-9d86-4430-bca7-85db00d7a099"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:16:31.889204 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:31.889185 4376 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6957320f-9d86-4430-bca7-85db00d7a099-istio-data\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:16:31.889311 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:31.889210 4376 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6957320f-9d86-4430-bca7-85db00d7a099-credential-socket\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:16:31.889311 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:31.889224 4376 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6957320f-9d86-4430-bca7-85db00d7a099-workload-socket\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:16:31.889311 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:31.889239 4376 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6957320f-9d86-4430-bca7-85db00d7a099-istiod-ca-cert\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:16:31.889311 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:31.889252 4376 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6957320f-9d86-4430-bca7-85db00d7a099-workload-certs\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:16:31.890685 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:31.890661 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6957320f-9d86-4430-bca7-85db00d7a099-istio-podinfo" pod "6957320f-9d86-4430-bca7-85db00d7a099" (UID: "6957320f-9d86-4430-bca7-85db00d7a099"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 05:16:31.890777 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:31.890686 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6957320f-9d86-4430-bca7-85db00d7a099-istio-token" pod "6957320f-9d86-4430-bca7-85db00d7a099" (UID: "6957320f-9d86-4430-bca7-85db00d7a099"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:16:31.890892 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:31.890874 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6957320f-9d86-4430-bca7-85db00d7a099-istio-envoy" pod "6957320f-9d86-4430-bca7-85db00d7a099" (UID: "6957320f-9d86-4430-bca7-85db00d7a099"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:16:31.890932 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:31.890889 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6957320f-9d86-4430-bca7-85db00d7a099-kube-api-access-dq2k2" pod "6957320f-9d86-4430-bca7-85db00d7a099" (UID: "6957320f-9d86-4430-bca7-85db00d7a099"). InnerVolumeSpecName "kube-api-access-dq2k2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:16:31.990470 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:31.990450 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dq2k2\" (UniqueName: \"kubernetes.io/projected/6957320f-9d86-4430-bca7-85db00d7a099-kube-api-access-dq2k2\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:16:31.990470 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:31.990470 4376 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6957320f-9d86-4430-bca7-85db00d7a099-istio-envoy\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:16:31.990602 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:31.990479 4376 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6957320f-9d86-4430-bca7-85db00d7a099-istio-token\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:16:31.990602 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:31.990486 4376 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6957320f-9d86-4430-bca7-85db00d7a099-istio-podinfo\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:16:32.367087 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:32.367053 4376 generic.go:356] "Generic (PLEG): container finished" podID="6957320f-9d86-4430-bca7-85db00d7a099" containerID="ca88dfd607dbbf2878cd20e8f583c5ecee8952003e85169731c0a12d701e9315" exitCode=0 Oct 31 05:16:32.367250 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:32.367099 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-brpvl" Oct 31 05:16:32.367250 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:32.367105 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-brpvl" event={"ID":"6957320f-9d86-4430-bca7-85db00d7a099","Type":"ContainerDied","Data":"ca88dfd607dbbf2878cd20e8f583c5ecee8952003e85169731c0a12d701e9315"} Oct 31 05:16:32.367250 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:32.367161 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-brpvl" event={"ID":"6957320f-9d86-4430-bca7-85db00d7a099","Type":"ContainerDied","Data":"8e850ede4e9ea44b5e1aae8f38af2dfe694dfc8e51562cf708e65ad31b6103db"} Oct 31 05:16:32.367250 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:32.367192 4376 scope.go:122] "RemoveContainer" containerID="ca88dfd607dbbf2878cd20e8f583c5ecee8952003e85169731c0a12d701e9315" Oct 31 05:16:32.380495 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:32.380470 4376 scope.go:122] "RemoveContainer" containerID="6f62c4f1a8beb2890f61a48c09ca5cd6a4e392a731114cdf4c1babee7bfda920" Oct 31 05:16:32.390830 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:32.390806 4376 scope.go:122] "RemoveContainer" containerID="ca88dfd607dbbf2878cd20e8f583c5ecee8952003e85169731c0a12d701e9315" Oct 31 05:16:32.391091 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:16:32.391071 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca88dfd607dbbf2878cd20e8f583c5ecee8952003e85169731c0a12d701e9315\": container with ID starting with ca88dfd607dbbf2878cd20e8f583c5ecee8952003e85169731c0a12d701e9315 not found: ID does not exist" containerID="ca88dfd607dbbf2878cd20e8f583c5ecee8952003e85169731c0a12d701e9315" Oct 31 05:16:32.391164 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:32.391106 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca88dfd607dbbf2878cd20e8f583c5ecee8952003e85169731c0a12d701e9315"} err="failed to get container status \"ca88dfd607dbbf2878cd20e8f583c5ecee8952003e85169731c0a12d701e9315\": rpc error: code = NotFound desc = could not find container \"ca88dfd607dbbf2878cd20e8f583c5ecee8952003e85169731c0a12d701e9315\": container with ID starting with ca88dfd607dbbf2878cd20e8f583c5ecee8952003e85169731c0a12d701e9315 not found: ID does not exist" Oct 31 05:16:32.391164 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:32.391123 4376 scope.go:122] "RemoveContainer" containerID="6f62c4f1a8beb2890f61a48c09ca5cd6a4e392a731114cdf4c1babee7bfda920" Oct 31 05:16:32.391370 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:16:32.391347 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f62c4f1a8beb2890f61a48c09ca5cd6a4e392a731114cdf4c1babee7bfda920\": container with ID starting with 6f62c4f1a8beb2890f61a48c09ca5cd6a4e392a731114cdf4c1babee7bfda920 not found: ID does not exist" containerID="6f62c4f1a8beb2890f61a48c09ca5cd6a4e392a731114cdf4c1babee7bfda920" Oct 31 05:16:32.391435 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:32.391375 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f62c4f1a8beb2890f61a48c09ca5cd6a4e392a731114cdf4c1babee7bfda920"} err="failed to get container status \"6f62c4f1a8beb2890f61a48c09ca5cd6a4e392a731114cdf4c1babee7bfda920\": rpc error: code = NotFound desc = could not find container \"6f62c4f1a8beb2890f61a48c09ca5cd6a4e392a731114cdf4c1babee7bfda920\": container with ID starting with 6f62c4f1a8beb2890f61a48c09ca5cd6a4e392a731114cdf4c1babee7bfda920 not found: ID does not exist" Oct 31 05:16:32.445588 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:32.445550 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3e4def5-df8d-42d2-810c-8fa6764d338b" path="/var/lib/kubelet/pods/b3e4def5-df8d-42d2-810c-8fa6764d338b/volumes" Oct 31 05:16:32.445967 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:32.445948 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-brpvl"] Oct 31 05:16:32.457312 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:32.457281 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-brpvl"] Oct 31 05:16:34.443280 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:34.443232 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6957320f-9d86-4430-bca7-85db00d7a099" path="/var/lib/kubelet/pods/6957320f-9d86-4430-bca7-85db00d7a099/volumes" Oct 31 05:16:34.837318 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:34.837208 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-227c93-74b5c495fc-v4rxp" podUID="8e0104d4-9c91-4dc6-93ae-a7844c9d89c8" containerName="payload-processing" containerID="cri-o://d5184cac0849763780d398b0d1e264c99217941e194b41ae3f9d4e6eb5fd9059" gracePeriod=25 Oct 31 05:16:35.655474 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:35.655437 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-227c93-74b5c495fc-v4rxp" podUID="8e0104d4-9c91-4dc6-93ae-a7844c9d89c8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 05:16:36.659291 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:36.659256 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 05:16:37.717142 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:37.717103 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-8973e2-849bd8687-5kxc2" podUID="2a9e4834-09d5-4add-9004-04c30da0527c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 05:16:39.094457 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:39.094431 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-8973e2-849bd8687-5kxc2" Oct 31 05:16:39.194893 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:39.194859 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a9e4834-09d5-4add-9004-04c30da0527c-kube-api-access-m9qc5\" (UniqueName: \"kubernetes.io/projected/2a9e4834-09d5-4add-9004-04c30da0527c-kube-api-access-m9qc5\") pod \"2a9e4834-09d5-4add-9004-04c30da0527c\" (UID: \"2a9e4834-09d5-4add-9004-04c30da0527c\") " Oct 31 05:16:39.195065 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:39.194909 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a9e4834-09d5-4add-9004-04c30da0527c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a9e4834-09d5-4add-9004-04c30da0527c-plugins-config-volume\") pod \"2a9e4834-09d5-4add-9004-04c30da0527c\" (UID: \"2a9e4834-09d5-4add-9004-04c30da0527c\") " Oct 31 05:16:39.195498 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:39.195472 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a9e4834-09d5-4add-9004-04c30da0527c-plugins-config-volume" pod "2a9e4834-09d5-4add-9004-04c30da0527c" (UID: "2a9e4834-09d5-4add-9004-04c30da0527c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:16:39.197493 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:39.197466 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a9e4834-09d5-4add-9004-04c30da0527c-kube-api-access-m9qc5" pod "2a9e4834-09d5-4add-9004-04c30da0527c" (UID: "2a9e4834-09d5-4add-9004-04c30da0527c"). InnerVolumeSpecName "kube-api-access-m9qc5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:16:39.296314 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:39.296277 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9qc5\" (UniqueName: \"kubernetes.io/projected/2a9e4834-09d5-4add-9004-04c30da0527c-kube-api-access-m9qc5\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:16:39.296314 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:39.296311 4376 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a9e4834-09d5-4add-9004-04c30da0527c-plugins-config-volume\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:16:39.410367 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:39.410322 4376 generic.go:356] "Generic (PLEG): container finished" podID="2a9e4834-09d5-4add-9004-04c30da0527c" containerID="cf0df277d55a85c5e084bb3a35e3afd985254f6fe7bdb826bd7737a7e3a50f86" exitCode=0 Oct 31 05:16:39.410542 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:39.410367 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-8973e2-849bd8687-5kxc2" event={"ID":"2a9e4834-09d5-4add-9004-04c30da0527c","Type":"ContainerDied","Data":"cf0df277d55a85c5e084bb3a35e3afd985254f6fe7bdb826bd7737a7e3a50f86"} Oct 31 05:16:39.410542 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:39.410417 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-8973e2-849bd8687-5kxc2" event={"ID":"2a9e4834-09d5-4add-9004-04c30da0527c","Type":"ContainerDied","Data":"89a0bc099a5458770d546a0e675e74b99eefb31366ff3741050fe27581170daa"} Oct 31 05:16:39.410542 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:39.410439 4376 scope.go:122] "RemoveContainer" containerID="cf0df277d55a85c5e084bb3a35e3afd985254f6fe7bdb826bd7737a7e3a50f86" Oct 31 05:16:39.410542 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:39.410375 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-8973e2-849bd8687-5kxc2" Oct 31 05:16:39.419779 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:39.419763 4376 scope.go:122] "RemoveContainer" containerID="cf0df277d55a85c5e084bb3a35e3afd985254f6fe7bdb826bd7737a7e3a50f86" Oct 31 05:16:39.420057 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:16:39.420037 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf0df277d55a85c5e084bb3a35e3afd985254f6fe7bdb826bd7737a7e3a50f86\": container with ID starting with cf0df277d55a85c5e084bb3a35e3afd985254f6fe7bdb826bd7737a7e3a50f86 not found: ID does not exist" containerID="cf0df277d55a85c5e084bb3a35e3afd985254f6fe7bdb826bd7737a7e3a50f86" Oct 31 05:16:39.420108 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:39.420065 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf0df277d55a85c5e084bb3a35e3afd985254f6fe7bdb826bd7737a7e3a50f86"} err="failed to get container status \"cf0df277d55a85c5e084bb3a35e3afd985254f6fe7bdb826bd7737a7e3a50f86\": rpc error: code = NotFound desc = could not find container \"cf0df277d55a85c5e084bb3a35e3afd985254f6fe7bdb826bd7737a7e3a50f86\": container with ID starting with cf0df277d55a85c5e084bb3a35e3afd985254f6fe7bdb826bd7737a7e3a50f86 not found: ID does not exist" Oct 31 05:16:39.471757 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:39.471715 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-8973e2-849bd8687-5kxc2"] Oct 31 05:16:39.485855 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:39.485824 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-8973e2-849bd8687-5kxc2"] Oct 31 05:16:40.444154 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:40.444119 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a9e4834-09d5-4add-9004-04c30da0527c" path="/var/lib/kubelet/pods/2a9e4834-09d5-4add-9004-04c30da0527c/volumes" Oct 31 05:16:41.008142 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:16:41.008082 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 05:16:45.655380 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:45.655325 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-227c93-74b5c495fc-v4rxp" podUID="8e0104d4-9c91-4dc6-93ae-a7844c9d89c8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 05:16:49.858508 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:49.858473 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-338f2e-849686b74-9bcnc"] Oct 31 05:16:49.859310 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:49.859278 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-338f2e-849686b74-9bcnc" podUID="7cffa2e2-93ef-4f2b-a512-107f5d14b698" containerName="maas-api" containerID="cri-o://df8432fa1463660d33bd46848e0ec0659352c10e8b8f09528915d8808c9f3138" gracePeriod=30 Oct 31 05:16:50.144358 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:50.142560 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-338f2e-849686b74-9bcnc" Oct 31 05:16:50.226404 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:50.226371 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7cffa2e2-93ef-4f2b-a512-107f5d14b698-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7cffa2e2-93ef-4f2b-a512-107f5d14b698-metrics-certs\") pod \"7cffa2e2-93ef-4f2b-a512-107f5d14b698\" (UID: \"7cffa2e2-93ef-4f2b-a512-107f5d14b698\") " Oct 31 05:16:50.226589 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:50.226443 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7cffa2e2-93ef-4f2b-a512-107f5d14b698-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7cffa2e2-93ef-4f2b-a512-107f5d14b698-openshift-service-ca\") pod \"7cffa2e2-93ef-4f2b-a512-107f5d14b698\" (UID: \"7cffa2e2-93ef-4f2b-a512-107f5d14b698\") " Oct 31 05:16:50.226589 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:50.226525 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7cffa2e2-93ef-4f2b-a512-107f5d14b698-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7cffa2e2-93ef-4f2b-a512-107f5d14b698-maas-api-tls\") pod \"7cffa2e2-93ef-4f2b-a512-107f5d14b698\" (UID: \"7cffa2e2-93ef-4f2b-a512-107f5d14b698\") " Oct 31 05:16:50.226589 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:50.226578 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7cffa2e2-93ef-4f2b-a512-107f5d14b698-kube-api-access-twk6m\" (UniqueName: \"kubernetes.io/projected/7cffa2e2-93ef-4f2b-a512-107f5d14b698-kube-api-access-twk6m\") pod \"7cffa2e2-93ef-4f2b-a512-107f5d14b698\" (UID: \"7cffa2e2-93ef-4f2b-a512-107f5d14b698\") " Oct 31 05:16:50.226852 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:50.226827 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7cffa2e2-93ef-4f2b-a512-107f5d14b698-openshift-service-ca" pod "7cffa2e2-93ef-4f2b-a512-107f5d14b698" (UID: "7cffa2e2-93ef-4f2b-a512-107f5d14b698"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:16:50.230470 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:50.230442 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7cffa2e2-93ef-4f2b-a512-107f5d14b698-maas-api-tls" pod "7cffa2e2-93ef-4f2b-a512-107f5d14b698" (UID: "7cffa2e2-93ef-4f2b-a512-107f5d14b698"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 05:16:50.230568 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:50.230466 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7cffa2e2-93ef-4f2b-a512-107f5d14b698-metrics-certs" pod "7cffa2e2-93ef-4f2b-a512-107f5d14b698" (UID: "7cffa2e2-93ef-4f2b-a512-107f5d14b698"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 05:16:50.230568 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:50.230482 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cffa2e2-93ef-4f2b-a512-107f5d14b698-kube-api-access-twk6m" pod "7cffa2e2-93ef-4f2b-a512-107f5d14b698" (UID: "7cffa2e2-93ef-4f2b-a512-107f5d14b698"). InnerVolumeSpecName "kube-api-access-twk6m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:16:50.328788 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:50.328742 4376 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7cffa2e2-93ef-4f2b-a512-107f5d14b698-maas-api-tls\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:16:50.328788 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:50.328787 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twk6m\" (UniqueName: \"kubernetes.io/projected/7cffa2e2-93ef-4f2b-a512-107f5d14b698-kube-api-access-twk6m\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:16:50.329000 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:50.328803 4376 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7cffa2e2-93ef-4f2b-a512-107f5d14b698-metrics-certs\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:16:50.329000 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:50.328815 4376 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7cffa2e2-93ef-4f2b-a512-107f5d14b698-openshift-service-ca\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:16:50.382414 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:50.382380 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 05:16:50.453087 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:50.452995 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-766728-d66dffc49-llqqr"] Oct 31 05:16:50.469996 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:50.469962 4376 generic.go:356] "Generic (PLEG): container finished" podID="7cffa2e2-93ef-4f2b-a512-107f5d14b698" containerID="df8432fa1463660d33bd46848e0ec0659352c10e8b8f09528915d8808c9f3138" exitCode=0 Oct 31 05:16:50.470141 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:50.470090 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-338f2e-849686b74-9bcnc" event={"ID":"7cffa2e2-93ef-4f2b-a512-107f5d14b698","Type":"ContainerDied","Data":"df8432fa1463660d33bd46848e0ec0659352c10e8b8f09528915d8808c9f3138"} Oct 31 05:16:50.470141 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:50.470138 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-338f2e-849686b74-9bcnc" event={"ID":"7cffa2e2-93ef-4f2b-a512-107f5d14b698","Type":"ContainerDied","Data":"b0b05d723d91511c08d3f6e3bd06a0dd6725fce484636ebeeae101dc6f0a972c"} Oct 31 05:16:50.470234 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:50.470161 4376 scope.go:122] "RemoveContainer" containerID="df8432fa1463660d33bd46848e0ec0659352c10e8b8f09528915d8808c9f3138" Oct 31 05:16:50.470234 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:50.470098 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-338f2e-849686b74-9bcnc" Oct 31 05:16:50.480681 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:50.480653 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-766728-69794bbd5f-s8zfz"] Oct 31 05:16:50.485727 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:50.485706 4376 scope.go:122] "RemoveContainer" containerID="df8432fa1463660d33bd46848e0ec0659352c10e8b8f09528915d8808c9f3138" Oct 31 05:16:50.486035 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:16:50.486006 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df8432fa1463660d33bd46848e0ec0659352c10e8b8f09528915d8808c9f3138\": container with ID starting with df8432fa1463660d33bd46848e0ec0659352c10e8b8f09528915d8808c9f3138 not found: ID does not exist" containerID="df8432fa1463660d33bd46848e0ec0659352c10e8b8f09528915d8808c9f3138" Oct 31 05:16:50.486138 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:50.486039 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df8432fa1463660d33bd46848e0ec0659352c10e8b8f09528915d8808c9f3138"} err="failed to get container status \"df8432fa1463660d33bd46848e0ec0659352c10e8b8f09528915d8808c9f3138\": rpc error: code = NotFound desc = could not find container \"df8432fa1463660d33bd46848e0ec0659352c10e8b8f09528915d8808c9f3138\": container with ID starting with df8432fa1463660d33bd46848e0ec0659352c10e8b8f09528915d8808c9f3138 not found: ID does not exist" Oct 31 05:16:50.538737 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:50.538697 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-338f2e-849686b74-9bcnc"] Oct 31 05:16:50.549765 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:50.549738 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-338f2e-849686b74-9bcnc"] Oct 31 05:16:52.445134 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:52.445094 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7cffa2e2-93ef-4f2b-a512-107f5d14b698" path="/var/lib/kubelet/pods/7cffa2e2-93ef-4f2b-a512-107f5d14b698/volumes" Oct 31 05:16:55.153422 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:55.153392 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-227c93-74b5c495fc-v4rxp" Oct 31 05:16:55.318296 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:55.318262 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e0104d4-9c91-4dc6-93ae-a7844c9d89c8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e0104d4-9c91-4dc6-93ae-a7844c9d89c8-plugins-config-volume\") pod \"8e0104d4-9c91-4dc6-93ae-a7844c9d89c8\" (UID: \"8e0104d4-9c91-4dc6-93ae-a7844c9d89c8\") " Oct 31 05:16:55.318522 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:55.318305 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e0104d4-9c91-4dc6-93ae-a7844c9d89c8-kube-api-access-mkts2\" (UniqueName: \"kubernetes.io/projected/8e0104d4-9c91-4dc6-93ae-a7844c9d89c8-kube-api-access-mkts2\") pod \"8e0104d4-9c91-4dc6-93ae-a7844c9d89c8\" (UID: \"8e0104d4-9c91-4dc6-93ae-a7844c9d89c8\") " Oct 31 05:16:55.318678 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:55.318659 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e0104d4-9c91-4dc6-93ae-a7844c9d89c8-plugins-config-volume" pod "8e0104d4-9c91-4dc6-93ae-a7844c9d89c8" (UID: "8e0104d4-9c91-4dc6-93ae-a7844c9d89c8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:16:55.318800 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:55.318786 4376 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e0104d4-9c91-4dc6-93ae-a7844c9d89c8-plugins-config-volume\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:16:55.320471 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:55.320444 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e0104d4-9c91-4dc6-93ae-a7844c9d89c8-kube-api-access-mkts2" pod "8e0104d4-9c91-4dc6-93ae-a7844c9d89c8" (UID: "8e0104d4-9c91-4dc6-93ae-a7844c9d89c8"). InnerVolumeSpecName "kube-api-access-mkts2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:16:55.419608 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:55.419578 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mkts2\" (UniqueName: \"kubernetes.io/projected/8e0104d4-9c91-4dc6-93ae-a7844c9d89c8-kube-api-access-mkts2\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:16:55.453487 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:55.453449 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-766728-d66dffc49-llqqr" podUID="f05ba444-d067-4e81-a4ca-cb9ef426c2a1" containerName="payload-processing" containerID="cri-o://6c89e8848671e9859f7b4c56e2e5e709a6ff1cd0c8818e41a10467a489e3afda" gracePeriod=25 Oct 31 05:16:55.481934 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:55.481894 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-766728-69794bbd5f-s8zfz" podUID="b9faf32d-45c1-4167-bb23-663987f3e602" containerName="payload-pre-processing" containerID="cri-o://a672b1062cd591142810625dd353b3fd9354a6c9bd3b42763d8cd1c2e727f474" gracePeriod=25 Oct 31 05:16:55.514122 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:55.514088 4376 generic.go:356] "Generic (PLEG): container finished" podID="8e0104d4-9c91-4dc6-93ae-a7844c9d89c8" containerID="d5184cac0849763780d398b0d1e264c99217941e194b41ae3f9d4e6eb5fd9059" exitCode=0 Oct 31 05:16:55.514262 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:55.514138 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-227c93-74b5c495fc-v4rxp" Oct 31 05:16:55.514262 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:55.514171 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-227c93-74b5c495fc-v4rxp" event={"ID":"8e0104d4-9c91-4dc6-93ae-a7844c9d89c8","Type":"ContainerDied","Data":"d5184cac0849763780d398b0d1e264c99217941e194b41ae3f9d4e6eb5fd9059"} Oct 31 05:16:55.514262 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:55.514225 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-227c93-74b5c495fc-v4rxp" event={"ID":"8e0104d4-9c91-4dc6-93ae-a7844c9d89c8","Type":"ContainerDied","Data":"56656f11975716a78a93973f340db8d8fbc5ac99b544b43c22ccd3c52b36a84a"} Oct 31 05:16:55.514262 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:55.514248 4376 scope.go:122] "RemoveContainer" containerID="d5184cac0849763780d398b0d1e264c99217941e194b41ae3f9d4e6eb5fd9059" Oct 31 05:16:55.524064 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:55.524043 4376 scope.go:122] "RemoveContainer" containerID="d5184cac0849763780d398b0d1e264c99217941e194b41ae3f9d4e6eb5fd9059" Oct 31 05:16:55.524365 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:16:55.524344 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5184cac0849763780d398b0d1e264c99217941e194b41ae3f9d4e6eb5fd9059\": container with ID starting with d5184cac0849763780d398b0d1e264c99217941e194b41ae3f9d4e6eb5fd9059 not found: ID does not exist" containerID="d5184cac0849763780d398b0d1e264c99217941e194b41ae3f9d4e6eb5fd9059" Oct 31 05:16:55.524438 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:55.524379 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5184cac0849763780d398b0d1e264c99217941e194b41ae3f9d4e6eb5fd9059"} err="failed to get container status \"d5184cac0849763780d398b0d1e264c99217941e194b41ae3f9d4e6eb5fd9059\": rpc error: code = NotFound desc = could not find container \"d5184cac0849763780d398b0d1e264c99217941e194b41ae3f9d4e6eb5fd9059\": container with ID starting with d5184cac0849763780d398b0d1e264c99217941e194b41ae3f9d4e6eb5fd9059 not found: ID does not exist" Oct 31 05:16:55.573173 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:55.573145 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-227c93-74b5c495fc-v4rxp"] Oct 31 05:16:55.586778 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:55.586747 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-227c93-74b5c495fc-v4rxp"] Oct 31 05:16:56.443485 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:56.443447 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e0104d4-9c91-4dc6-93ae-a7844c9d89c8" path="/var/lib/kubelet/pods/8e0104d4-9c91-4dc6-93ae-a7844c9d89c8/volumes" Oct 31 05:16:58.421156 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:58.421112 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-766728-69794bbd5f-s8zfz" podUID="b9faf32d-45c1-4167-bb23-663987f3e602" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 05:16:58.421156 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:16:58.421145 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-766728-d66dffc49-llqqr" podUID="f05ba444-d067-4e81-a4ca-cb9ef426c2a1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 05:17:01.792510 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:01.792478 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b2a372-84cc9765b7-8q729"] Oct 31 05:17:04.678843 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:04.678811 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-25f56e-bd47bbbd9-r4fhf"] Oct 31 05:17:04.709481 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:04.709446 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-25f56e-69577647c5-wvb9t"] Oct 31 05:17:05.526960 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:05.526928 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 05:17:06.793443 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:06.793375 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b2a372-84cc9765b7-8q729" podUID="8414bff7-0d22-4ccd-8c0d-e6c2dd294022" containerName="payload-pre-processing" containerID="cri-o://49a566f597506c107acb92205ef5700da38eadccc16245a23304d33651a6be56" gracePeriod=25 Oct 31 05:17:08.420944 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:08.420898 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-766728-d66dffc49-llqqr" podUID="f05ba444-d067-4e81-a4ca-cb9ef426c2a1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 05:17:08.420944 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:08.420940 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-766728-69794bbd5f-s8zfz" podUID="b9faf32d-45c1-4167-bb23-663987f3e602" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 05:17:09.680051 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:09.680007 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-25f56e-bd47bbbd9-r4fhf" podUID="43d89e12-3149-41df-825f-40801de7576a" containerName="payload-processing" containerID="cri-o://9aa705e067a14e38ce68e66c9b17454a1612d5d2566fd7ca585063485bbec729" gracePeriod=25 Oct 31 05:17:09.710619 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:09.710575 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-25f56e-69577647c5-wvb9t" podUID="69d5f412-aaf9-4d2a-afb0-3f76f0861259" containerName="payload-pre-processing" containerID="cri-o://90b51dfd6d3297a2dbbc7bfc2f535824c07608cd387cb1587293c431c18c9dc0" gracePeriod=25 Oct 31 05:17:10.998689 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:17:10.998653 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 05:17:11.135936 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:11.135899 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b2a372-84cc9765b7-8q729" podUID="8414bff7-0d22-4ccd-8c0d-e6c2dd294022" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 05:17:11.788921 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:11.788882 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-25f56e-85b84d5b9b-5n9vf"] Oct 31 05:17:11.789194 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:11.789166 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-25f56e-85b84d5b9b-5n9vf" podUID="994761dc-71d9-4334-a0aa-891d7287a690" containerName="maas-api" containerID="cri-o://e7ee00005b072dc8127f49f2ded0369bbd6f7b7a2cbda557c8c42a479f262c4b" gracePeriod=30 Oct 31 05:17:12.061962 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:12.061872 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-25f56e-85b84d5b9b-5n9vf" Oct 31 05:17:12.145398 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:12.145363 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/994761dc-71d9-4334-a0aa-891d7287a690-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/994761dc-71d9-4334-a0aa-891d7287a690-openshift-service-ca\") pod \"994761dc-71d9-4334-a0aa-891d7287a690\" (UID: \"994761dc-71d9-4334-a0aa-891d7287a690\") " Oct 31 05:17:12.145579 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:12.145560 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/994761dc-71d9-4334-a0aa-891d7287a690-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/994761dc-71d9-4334-a0aa-891d7287a690-maas-api-tls\") pod \"994761dc-71d9-4334-a0aa-891d7287a690\" (UID: \"994761dc-71d9-4334-a0aa-891d7287a690\") " Oct 31 05:17:12.145626 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:12.145608 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/994761dc-71d9-4334-a0aa-891d7287a690-kube-api-access-4nj9t\" (UniqueName: \"kubernetes.io/projected/994761dc-71d9-4334-a0aa-891d7287a690-kube-api-access-4nj9t\") pod \"994761dc-71d9-4334-a0aa-891d7287a690\" (UID: \"994761dc-71d9-4334-a0aa-891d7287a690\") " Oct 31 05:17:12.145668 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:12.145643 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/994761dc-71d9-4334-a0aa-891d7287a690-metrics-certs\" (UniqueName: \"kubernetes.io/secret/994761dc-71d9-4334-a0aa-891d7287a690-metrics-certs\") pod \"994761dc-71d9-4334-a0aa-891d7287a690\" (UID: \"994761dc-71d9-4334-a0aa-891d7287a690\") " Oct 31 05:17:12.145735 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:12.145708 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/994761dc-71d9-4334-a0aa-891d7287a690-openshift-service-ca" pod "994761dc-71d9-4334-a0aa-891d7287a690" (UID: "994761dc-71d9-4334-a0aa-891d7287a690"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:17:12.146042 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:12.146019 4376 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/994761dc-71d9-4334-a0aa-891d7287a690-openshift-service-ca\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:17:12.147831 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:12.147805 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/994761dc-71d9-4334-a0aa-891d7287a690-maas-api-tls" pod "994761dc-71d9-4334-a0aa-891d7287a690" (UID: "994761dc-71d9-4334-a0aa-891d7287a690"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 05:17:12.148048 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:12.148030 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/994761dc-71d9-4334-a0aa-891d7287a690-kube-api-access-4nj9t" pod "994761dc-71d9-4334-a0aa-891d7287a690" (UID: "994761dc-71d9-4334-a0aa-891d7287a690"). InnerVolumeSpecName "kube-api-access-4nj9t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:17:12.148048 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:12.148035 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/994761dc-71d9-4334-a0aa-891d7287a690-metrics-certs" pod "994761dc-71d9-4334-a0aa-891d7287a690" (UID: "994761dc-71d9-4334-a0aa-891d7287a690"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 05:17:12.249230 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:12.249178 4376 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/994761dc-71d9-4334-a0aa-891d7287a690-maas-api-tls\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:17:12.249230 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:12.249230 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4nj9t\" (UniqueName: \"kubernetes.io/projected/994761dc-71d9-4334-a0aa-891d7287a690-kube-api-access-4nj9t\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:17:12.249555 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:12.249247 4376 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/994761dc-71d9-4334-a0aa-891d7287a690-metrics-certs\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:17:12.624270 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:12.624236 4376 generic.go:356] "Generic (PLEG): container finished" podID="994761dc-71d9-4334-a0aa-891d7287a690" containerID="e7ee00005b072dc8127f49f2ded0369bbd6f7b7a2cbda557c8c42a479f262c4b" exitCode=0 Oct 31 05:17:12.624476 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:12.624280 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-25f56e-85b84d5b9b-5n9vf" Oct 31 05:17:12.624476 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:12.624299 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-25f56e-85b84d5b9b-5n9vf" event={"ID":"994761dc-71d9-4334-a0aa-891d7287a690","Type":"ContainerDied","Data":"e7ee00005b072dc8127f49f2ded0369bbd6f7b7a2cbda557c8c42a479f262c4b"} Oct 31 05:17:12.624476 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:12.624372 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-25f56e-85b84d5b9b-5n9vf" event={"ID":"994761dc-71d9-4334-a0aa-891d7287a690","Type":"ContainerDied","Data":"88e14b49aeff2cd3990335ce3a3cddf12d20ee16bfd3d0316e63f6492f41923d"} Oct 31 05:17:12.624476 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:12.624395 4376 scope.go:122] "RemoveContainer" containerID="e7ee00005b072dc8127f49f2ded0369bbd6f7b7a2cbda557c8c42a479f262c4b" Oct 31 05:17:12.633941 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:12.633920 4376 scope.go:122] "RemoveContainer" containerID="e7ee00005b072dc8127f49f2ded0369bbd6f7b7a2cbda557c8c42a479f262c4b" Oct 31 05:17:12.634201 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:17:12.634180 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7ee00005b072dc8127f49f2ded0369bbd6f7b7a2cbda557c8c42a479f262c4b\": container with ID starting with e7ee00005b072dc8127f49f2ded0369bbd6f7b7a2cbda557c8c42a479f262c4b not found: ID does not exist" containerID="e7ee00005b072dc8127f49f2ded0369bbd6f7b7a2cbda557c8c42a479f262c4b" Oct 31 05:17:12.634271 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:12.634214 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7ee00005b072dc8127f49f2ded0369bbd6f7b7a2cbda557c8c42a479f262c4b"} err="failed to get container status \"e7ee00005b072dc8127f49f2ded0369bbd6f7b7a2cbda557c8c42a479f262c4b\": rpc error: code = NotFound desc = could not find container \"e7ee00005b072dc8127f49f2ded0369bbd6f7b7a2cbda557c8c42a479f262c4b\": container with ID starting with e7ee00005b072dc8127f49f2ded0369bbd6f7b7a2cbda557c8c42a479f262c4b not found: ID does not exist" Oct 31 05:17:12.682193 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:12.682163 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-25f56e-85b84d5b9b-5n9vf"] Oct 31 05:17:12.684299 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:12.684269 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-25f56e-85b84d5b9b-5n9vf"] Oct 31 05:17:14.238352 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:14.222283 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 05:17:14.442713 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:14.442678 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="994761dc-71d9-4334-a0aa-891d7287a690" path="/var/lib/kubelet/pods/994761dc-71d9-4334-a0aa-891d7287a690/volumes" Oct 31 05:17:14.681380 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:14.678425 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-25f56e-69577647c5-wvb9t" podUID="69d5f412-aaf9-4d2a-afb0-3f76f0861259" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 05:17:14.681380 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:14.679054 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-25f56e-bd47bbbd9-r4fhf" podUID="43d89e12-3149-41df-825f-40801de7576a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 05:17:15.412930 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:15.412887 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-5n2gs"] Oct 31 05:17:15.413446 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:15.413137 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-5n2gs" podUID="d057ba0f-e65d-44de-949f-bc94f3c20550" containerName="istio-proxy" containerID="cri-o://6320e141db89af9e001de1e607995acab227e72490f754e8f73b16fddce9f78d" gracePeriod=30 Oct 31 05:17:15.656368 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:15.656264 4376 generic.go:356] "Generic (PLEG): container finished" podID="b9faf32d-45c1-4167-bb23-663987f3e602" containerID="a672b1062cd591142810625dd353b3fd9354a6c9bd3b42763d8cd1c2e727f474" exitCode=0 Oct 31 05:17:15.656536 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:15.656378 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-766728-69794bbd5f-s8zfz" event={"ID":"b9faf32d-45c1-4167-bb23-663987f3e602","Type":"ContainerDied","Data":"a672b1062cd591142810625dd353b3fd9354a6c9bd3b42763d8cd1c2e727f474"} Oct 31 05:17:15.658541 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:15.658512 4376 generic.go:356] "Generic (PLEG): container finished" podID="f05ba444-d067-4e81-a4ca-cb9ef426c2a1" containerID="6c89e8848671e9859f7b4c56e2e5e709a6ff1cd0c8818e41a10467a489e3afda" exitCode=0 Oct 31 05:17:15.658740 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:15.658721 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-766728-d66dffc49-llqqr" event={"ID":"f05ba444-d067-4e81-a4ca-cb9ef426c2a1","Type":"ContainerDied","Data":"6c89e8848671e9859f7b4c56e2e5e709a6ff1cd0c8818e41a10467a489e3afda"} Oct 31 05:17:15.749632 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:15.749601 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-766728-d66dffc49-llqqr" Oct 31 05:17:15.782422 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:15.782392 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-766728-69794bbd5f-s8zfz" Oct 31 05:17:15.890050 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:15.890021 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f05ba444-d067-4e81-a4ca-cb9ef426c2a1-kube-api-access-6n58b\" (UniqueName: \"kubernetes.io/projected/f05ba444-d067-4e81-a4ca-cb9ef426c2a1-kube-api-access-6n58b\") pod \"f05ba444-d067-4e81-a4ca-cb9ef426c2a1\" (UID: \"f05ba444-d067-4e81-a4ca-cb9ef426c2a1\") " Oct 31 05:17:15.890199 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:15.890062 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f05ba444-d067-4e81-a4ca-cb9ef426c2a1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f05ba444-d067-4e81-a4ca-cb9ef426c2a1-plugins-config-volume\") pod \"f05ba444-d067-4e81-a4ca-cb9ef426c2a1\" (UID: \"f05ba444-d067-4e81-a4ca-cb9ef426c2a1\") " Oct 31 05:17:15.890199 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:15.890106 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9faf32d-45c1-4167-bb23-663987f3e602-kube-api-access-vd59h\" (UniqueName: \"kubernetes.io/projected/b9faf32d-45c1-4167-bb23-663987f3e602-kube-api-access-vd59h\") pod \"b9faf32d-45c1-4167-bb23-663987f3e602\" (UID: \"b9faf32d-45c1-4167-bb23-663987f3e602\") " Oct 31 05:17:15.890286 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:15.890266 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9faf32d-45c1-4167-bb23-663987f3e602-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9faf32d-45c1-4167-bb23-663987f3e602-plugins-config-volume\") pod \"b9faf32d-45c1-4167-bb23-663987f3e602\" (UID: \"b9faf32d-45c1-4167-bb23-663987f3e602\") " Oct 31 05:17:15.890661 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:15.890626 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f05ba444-d067-4e81-a4ca-cb9ef426c2a1-plugins-config-volume" pod "f05ba444-d067-4e81-a4ca-cb9ef426c2a1" (UID: "f05ba444-d067-4e81-a4ca-cb9ef426c2a1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:17:15.890661 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:15.890643 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9faf32d-45c1-4167-bb23-663987f3e602-plugins-config-volume" pod "b9faf32d-45c1-4167-bb23-663987f3e602" (UID: "b9faf32d-45c1-4167-bb23-663987f3e602"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:17:15.892190 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:15.892160 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f05ba444-d067-4e81-a4ca-cb9ef426c2a1-kube-api-access-6n58b" pod "f05ba444-d067-4e81-a4ca-cb9ef426c2a1" (UID: "f05ba444-d067-4e81-a4ca-cb9ef426c2a1"). InnerVolumeSpecName "kube-api-access-6n58b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:17:15.892290 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:15.892195 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9faf32d-45c1-4167-bb23-663987f3e602-kube-api-access-vd59h" pod "b9faf32d-45c1-4167-bb23-663987f3e602" (UID: "b9faf32d-45c1-4167-bb23-663987f3e602"). InnerVolumeSpecName "kube-api-access-vd59h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:17:15.992002 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:15.991929 4376 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f05ba444-d067-4e81-a4ca-cb9ef426c2a1-plugins-config-volume\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:17:15.992002 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:15.991956 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vd59h\" (UniqueName: \"kubernetes.io/projected/b9faf32d-45c1-4167-bb23-663987f3e602-kube-api-access-vd59h\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:17:15.992002 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:15.991966 4376 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9faf32d-45c1-4167-bb23-663987f3e602-plugins-config-volume\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:17:15.992002 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:15.991977 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6n58b\" (UniqueName: \"kubernetes.io/projected/f05ba444-d067-4e81-a4ca-cb9ef426c2a1-kube-api-access-6n58b\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:17:16.670025 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:16.669982 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-766728-69794bbd5f-s8zfz" event={"ID":"b9faf32d-45c1-4167-bb23-663987f3e602","Type":"ContainerDied","Data":"5286b69d00501647ee6dc7bc285f551a6f0918ef7ed706050737f4608a9fbb2c"} Oct 31 05:17:16.670811 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:16.670789 4376 scope.go:122] "RemoveContainer" containerID="a672b1062cd591142810625dd353b3fd9354a6c9bd3b42763d8cd1c2e727f474" Oct 31 05:17:16.671126 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:16.671107 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-766728-69794bbd5f-s8zfz" Oct 31 05:17:16.675310 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:16.675274 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-766728-d66dffc49-llqqr" event={"ID":"f05ba444-d067-4e81-a4ca-cb9ef426c2a1","Type":"ContainerDied","Data":"178b91ab5a71aed1689d629a670a3dbebed60a8e2b41b0b0cd111e0f31544010"} Oct 31 05:17:16.675310 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:16.675300 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-766728-d66dffc49-llqqr" Oct 31 05:17:16.685717 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:16.685677 4376 scope.go:122] "RemoveContainer" containerID="6c89e8848671e9859f7b4c56e2e5e709a6ff1cd0c8818e41a10467a489e3afda" Oct 31 05:17:16.729419 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:16.729341 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-766728-69794bbd5f-s8zfz"] Oct 31 05:17:16.743651 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:16.743626 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-766728-69794bbd5f-s8zfz"] Oct 31 05:17:16.770590 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:16.770558 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-766728-d66dffc49-llqqr"] Oct 31 05:17:16.774389 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:16.774367 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-766728-d66dffc49-llqqr"] Oct 31 05:17:18.444388 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:18.442739 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9faf32d-45c1-4167-bb23-663987f3e602" path="/var/lib/kubelet/pods/b9faf32d-45c1-4167-bb23-663987f3e602/volumes" Oct 31 05:17:18.444388 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:18.443306 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f05ba444-d067-4e81-a4ca-cb9ef426c2a1" path="/var/lib/kubelet/pods/f05ba444-d067-4e81-a4ca-cb9ef426c2a1/volumes" Oct 31 05:17:19.532235 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:19.532204 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-25f56e-openshift-default-6465df847-6j725"] Oct 31 05:17:19.532718 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:19.532459 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-25f56e-openshift-default-6465df847-6j725" podUID="caf8772a-4894-4924-9f18-f2b191aadad7" containerName="istio-proxy" containerID="cri-o://f404f437b5bd699a47c12a58483993d7b7b47ad54b19ee53bb2b7863526c98f2" gracePeriod=30 Oct 31 05:17:20.677920 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:20.677895 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-5n2gs" Oct 31 05:17:20.695146 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:20.695077 4376 generic.go:356] "Generic (PLEG): container finished" podID="d057ba0f-e65d-44de-949f-bc94f3c20550" containerID="6320e141db89af9e001de1e607995acab227e72490f754e8f73b16fddce9f78d" exitCode=0 Oct 31 05:17:20.695146 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:20.695139 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-5n2gs" Oct 31 05:17:20.695369 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:20.695158 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-5n2gs" event={"ID":"d057ba0f-e65d-44de-949f-bc94f3c20550","Type":"ContainerDied","Data":"6320e141db89af9e001de1e607995acab227e72490f754e8f73b16fddce9f78d"} Oct 31 05:17:20.695369 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:20.695212 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-5n2gs" event={"ID":"d057ba0f-e65d-44de-949f-bc94f3c20550","Type":"ContainerDied","Data":"86fa4999a692d1530210154a293aef2a9cb10538982e0f09d60e7978b81948a5"} Oct 31 05:17:20.695369 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:20.695234 4376 scope.go:122] "RemoveContainer" containerID="6320e141db89af9e001de1e607995acab227e72490f754e8f73b16fddce9f78d" Oct 31 05:17:20.704136 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:20.704113 4376 scope.go:122] "RemoveContainer" containerID="6320e141db89af9e001de1e607995acab227e72490f754e8f73b16fddce9f78d" Oct 31 05:17:20.704434 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:17:20.704412 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6320e141db89af9e001de1e607995acab227e72490f754e8f73b16fddce9f78d\": container with ID starting with 6320e141db89af9e001de1e607995acab227e72490f754e8f73b16fddce9f78d not found: ID does not exist" containerID="6320e141db89af9e001de1e607995acab227e72490f754e8f73b16fddce9f78d" Oct 31 05:17:20.704521 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:20.704465 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6320e141db89af9e001de1e607995acab227e72490f754e8f73b16fddce9f78d"} err="failed to get container status \"6320e141db89af9e001de1e607995acab227e72490f754e8f73b16fddce9f78d\": rpc error: code = NotFound desc = could not find container \"6320e141db89af9e001de1e607995acab227e72490f754e8f73b16fddce9f78d\": container with ID starting with 6320e141db89af9e001de1e607995acab227e72490f754e8f73b16fddce9f78d not found: ID does not exist" Oct 31 05:17:20.844053 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:20.844023 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d057ba0f-e65d-44de-949f-bc94f3c20550-istio-token\" (UniqueName: \"kubernetes.io/projected/d057ba0f-e65d-44de-949f-bc94f3c20550-istio-token\") pod \"d057ba0f-e65d-44de-949f-bc94f3c20550\" (UID: \"d057ba0f-e65d-44de-949f-bc94f3c20550\") " Oct 31 05:17:20.844229 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:20.844070 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d057ba0f-e65d-44de-949f-bc94f3c20550-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d057ba0f-e65d-44de-949f-bc94f3c20550-workload-socket\") pod \"d057ba0f-e65d-44de-949f-bc94f3c20550\" (UID: \"d057ba0f-e65d-44de-949f-bc94f3c20550\") " Oct 31 05:17:20.844229 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:20.844131 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d057ba0f-e65d-44de-949f-bc94f3c20550-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d057ba0f-e65d-44de-949f-bc94f3c20550-istiod-ca-cert\") pod \"d057ba0f-e65d-44de-949f-bc94f3c20550\" (UID: \"d057ba0f-e65d-44de-949f-bc94f3c20550\") " Oct 31 05:17:20.844229 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:20.844163 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d057ba0f-e65d-44de-949f-bc94f3c20550-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d057ba0f-e65d-44de-949f-bc94f3c20550-istio-podinfo\") pod \"d057ba0f-e65d-44de-949f-bc94f3c20550\" (UID: \"d057ba0f-e65d-44de-949f-bc94f3c20550\") " Oct 31 05:17:20.844229 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:20.844192 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d057ba0f-e65d-44de-949f-bc94f3c20550-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d057ba0f-e65d-44de-949f-bc94f3c20550-credential-socket\") pod \"d057ba0f-e65d-44de-949f-bc94f3c20550\" (UID: \"d057ba0f-e65d-44de-949f-bc94f3c20550\") " Oct 31 05:17:20.844453 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:20.844236 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d057ba0f-e65d-44de-949f-bc94f3c20550-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d057ba0f-e65d-44de-949f-bc94f3c20550-istio-data\") pod \"d057ba0f-e65d-44de-949f-bc94f3c20550\" (UID: \"d057ba0f-e65d-44de-949f-bc94f3c20550\") " Oct 31 05:17:20.844453 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:20.844316 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d057ba0f-e65d-44de-949f-bc94f3c20550-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d057ba0f-e65d-44de-949f-bc94f3c20550-workload-certs\") pod \"d057ba0f-e65d-44de-949f-bc94f3c20550\" (UID: \"d057ba0f-e65d-44de-949f-bc94f3c20550\") " Oct 31 05:17:20.844453 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:20.844369 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d057ba0f-e65d-44de-949f-bc94f3c20550-kube-api-access-q8vjx\" (UniqueName: \"kubernetes.io/projected/d057ba0f-e65d-44de-949f-bc94f3c20550-kube-api-access-q8vjx\") pod \"d057ba0f-e65d-44de-949f-bc94f3c20550\" (UID: \"d057ba0f-e65d-44de-949f-bc94f3c20550\") " Oct 31 05:17:20.844453 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:20.844411 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d057ba0f-e65d-44de-949f-bc94f3c20550-workload-socket" pod "d057ba0f-e65d-44de-949f-bc94f3c20550" (UID: "d057ba0f-e65d-44de-949f-bc94f3c20550"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:17:20.844722 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:20.844457 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d057ba0f-e65d-44de-949f-bc94f3c20550-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d057ba0f-e65d-44de-949f-bc94f3c20550-istio-envoy\") pod \"d057ba0f-e65d-44de-949f-bc94f3c20550\" (UID: \"d057ba0f-e65d-44de-949f-bc94f3c20550\") " Oct 31 05:17:20.844722 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:20.844494 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d057ba0f-e65d-44de-949f-bc94f3c20550-credential-socket" pod "d057ba0f-e65d-44de-949f-bc94f3c20550" (UID: "d057ba0f-e65d-44de-949f-bc94f3c20550"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:17:20.844722 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:20.844577 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d057ba0f-e65d-44de-949f-bc94f3c20550-istio-data" pod "d057ba0f-e65d-44de-949f-bc94f3c20550" (UID: "d057ba0f-e65d-44de-949f-bc94f3c20550"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:17:20.844722 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:20.844652 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d057ba0f-e65d-44de-949f-bc94f3c20550-workload-certs" pod "d057ba0f-e65d-44de-949f-bc94f3c20550" (UID: "d057ba0f-e65d-44de-949f-bc94f3c20550"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:17:20.844722 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:20.844579 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d057ba0f-e65d-44de-949f-bc94f3c20550-istiod-ca-cert" pod "d057ba0f-e65d-44de-949f-bc94f3c20550" (UID: "d057ba0f-e65d-44de-949f-bc94f3c20550"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:17:20.844998 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:20.844882 4376 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d057ba0f-e65d-44de-949f-bc94f3c20550-workload-socket\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:17:20.844998 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:20.844900 4376 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d057ba0f-e65d-44de-949f-bc94f3c20550-istiod-ca-cert\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:17:20.844998 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:20.844910 4376 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d057ba0f-e65d-44de-949f-bc94f3c20550-credential-socket\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:17:20.844998 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:20.844918 4376 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d057ba0f-e65d-44de-949f-bc94f3c20550-istio-data\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:17:20.844998 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:20.844928 4376 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d057ba0f-e65d-44de-949f-bc94f3c20550-workload-certs\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:17:20.846625 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:20.846591 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d057ba0f-e65d-44de-949f-bc94f3c20550-kube-api-access-q8vjx" pod "d057ba0f-e65d-44de-949f-bc94f3c20550" (UID: "d057ba0f-e65d-44de-949f-bc94f3c20550"). InnerVolumeSpecName "kube-api-access-q8vjx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:17:20.846807 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:20.846777 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d057ba0f-e65d-44de-949f-bc94f3c20550-istio-token" pod "d057ba0f-e65d-44de-949f-bc94f3c20550" (UID: "d057ba0f-e65d-44de-949f-bc94f3c20550"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:17:20.846807 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:20.846789 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d057ba0f-e65d-44de-949f-bc94f3c20550-istio-podinfo" pod "d057ba0f-e65d-44de-949f-bc94f3c20550" (UID: "d057ba0f-e65d-44de-949f-bc94f3c20550"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 05:17:20.846939 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:20.846854 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d057ba0f-e65d-44de-949f-bc94f3c20550-istio-envoy" pod "d057ba0f-e65d-44de-949f-bc94f3c20550" (UID: "d057ba0f-e65d-44de-949f-bc94f3c20550"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:17:20.876048 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:20.876010 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-4c2387/distinct-w3-4c2387-kserve-768bdd84c7-6jmtl"] Oct 31 05:17:20.945996 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:20.945918 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8vjx\" (UniqueName: \"kubernetes.io/projected/d057ba0f-e65d-44de-949f-bc94f3c20550-kube-api-access-q8vjx\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:17:20.945996 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:20.945949 4376 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d057ba0f-e65d-44de-949f-bc94f3c20550-istio-envoy\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:17:20.945996 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:20.945960 4376 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d057ba0f-e65d-44de-949f-bc94f3c20550-istio-token\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:17:20.945996 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:20.945968 4376 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d057ba0f-e65d-44de-949f-bc94f3c20550-istio-podinfo\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:17:21.046503 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:21.046469 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-5n2gs"] Oct 31 05:17:21.051131 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:21.051104 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-5n2gs"] Oct 31 05:17:21.135312 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:21.135280 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b2a372-84cc9765b7-8q729" podUID="8414bff7-0d22-4ccd-8c0d-e6c2dd294022" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 05:17:22.442812 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:22.442780 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d057ba0f-e65d-44de-949f-bc94f3c20550" path="/var/lib/kubelet/pods/d057ba0f-e65d-44de-949f-bc94f3c20550/volumes" Oct 31 05:17:24.673831 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:24.673791 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-25f56e-69577647c5-wvb9t" podUID="69d5f412-aaf9-4d2a-afb0-3f76f0861259" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 05:17:24.674199 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:24.673803 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-25f56e-bd47bbbd9-r4fhf" podUID="43d89e12-3149-41df-825f-40801de7576a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 05:17:24.718759 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:24.718723 4376 generic.go:356] "Generic (PLEG): container finished" podID="caf8772a-4894-4924-9f18-f2b191aadad7" containerID="f404f437b5bd699a47c12a58483993d7b7b47ad54b19ee53bb2b7863526c98f2" exitCode=0 Oct 31 05:17:24.718934 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:24.718822 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-25f56e-openshift-default-6465df847-6j725" event={"ID":"caf8772a-4894-4924-9f18-f2b191aadad7","Type":"ContainerDied","Data":"f404f437b5bd699a47c12a58483993d7b7b47ad54b19ee53bb2b7863526c98f2"} Oct 31 05:17:24.718934 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:24.718882 4376 scope.go:122] "RemoveContainer" containerID="440604183b7edcc8cd815d9149aa46b04f0ad06c051c8897a66a7ef0bd0d4761" Oct 31 05:17:24.787267 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:24.787244 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-25f56e-openshift-default-6465df847-6j725" Oct 31 05:17:24.886854 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:24.886821 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/caf8772a-4894-4924-9f18-f2b191aadad7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/caf8772a-4894-4924-9f18-f2b191aadad7-credential-socket\") pod \"caf8772a-4894-4924-9f18-f2b191aadad7\" (UID: \"caf8772a-4894-4924-9f18-f2b191aadad7\") " Oct 31 05:17:24.887083 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:24.886864 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/caf8772a-4894-4924-9f18-f2b191aadad7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/caf8772a-4894-4924-9f18-f2b191aadad7-istiod-ca-cert\") pod \"caf8772a-4894-4924-9f18-f2b191aadad7\" (UID: \"caf8772a-4894-4924-9f18-f2b191aadad7\") " Oct 31 05:17:24.887083 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:24.886931 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/caf8772a-4894-4924-9f18-f2b191aadad7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/caf8772a-4894-4924-9f18-f2b191aadad7-workload-socket\") pod \"caf8772a-4894-4924-9f18-f2b191aadad7\" (UID: \"caf8772a-4894-4924-9f18-f2b191aadad7\") " Oct 31 05:17:24.887083 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:24.886957 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/caf8772a-4894-4924-9f18-f2b191aadad7-kube-api-access-55vn7\" (UniqueName: \"kubernetes.io/projected/caf8772a-4894-4924-9f18-f2b191aadad7-kube-api-access-55vn7\") pod \"caf8772a-4894-4924-9f18-f2b191aadad7\" (UID: \"caf8772a-4894-4924-9f18-f2b191aadad7\") " Oct 31 05:17:24.887083 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:24.886999 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/caf8772a-4894-4924-9f18-f2b191aadad7-istio-token\" (UniqueName: \"kubernetes.io/projected/caf8772a-4894-4924-9f18-f2b191aadad7-istio-token\") pod \"caf8772a-4894-4924-9f18-f2b191aadad7\" (UID: \"caf8772a-4894-4924-9f18-f2b191aadad7\") " Oct 31 05:17:24.887307 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:24.887154 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/caf8772a-4894-4924-9f18-f2b191aadad7-credential-socket" pod "caf8772a-4894-4924-9f18-f2b191aadad7" (UID: "caf8772a-4894-4924-9f18-f2b191aadad7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:17:24.887307 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:24.887196 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/caf8772a-4894-4924-9f18-f2b191aadad7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/caf8772a-4894-4924-9f18-f2b191aadad7-workload-certs\") pod \"caf8772a-4894-4924-9f18-f2b191aadad7\" (UID: \"caf8772a-4894-4924-9f18-f2b191aadad7\") " Oct 31 05:17:24.887307 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:24.887251 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/caf8772a-4894-4924-9f18-f2b191aadad7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/caf8772a-4894-4924-9f18-f2b191aadad7-istio-envoy\") pod \"caf8772a-4894-4924-9f18-f2b191aadad7\" (UID: \"caf8772a-4894-4924-9f18-f2b191aadad7\") " Oct 31 05:17:24.887515 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:24.887309 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/caf8772a-4894-4924-9f18-f2b191aadad7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/caf8772a-4894-4924-9f18-f2b191aadad7-istio-podinfo\") pod \"caf8772a-4894-4924-9f18-f2b191aadad7\" (UID: \"caf8772a-4894-4924-9f18-f2b191aadad7\") " Oct 31 05:17:24.887515 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:24.887348 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/caf8772a-4894-4924-9f18-f2b191aadad7-workload-socket" pod "caf8772a-4894-4924-9f18-f2b191aadad7" (UID: "caf8772a-4894-4924-9f18-f2b191aadad7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:17:24.887515 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:24.887382 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/caf8772a-4894-4924-9f18-f2b191aadad7-istiod-ca-cert" pod "caf8772a-4894-4924-9f18-f2b191aadad7" (UID: "caf8772a-4894-4924-9f18-f2b191aadad7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:17:24.887515 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:24.887439 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/caf8772a-4894-4924-9f18-f2b191aadad7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/caf8772a-4894-4924-9f18-f2b191aadad7-istio-data\") pod \"caf8772a-4894-4924-9f18-f2b191aadad7\" (UID: \"caf8772a-4894-4924-9f18-f2b191aadad7\") " Oct 31 05:17:24.887731 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:24.887539 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/caf8772a-4894-4924-9f18-f2b191aadad7-workload-certs" pod "caf8772a-4894-4924-9f18-f2b191aadad7" (UID: "caf8772a-4894-4924-9f18-f2b191aadad7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:17:24.887788 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:24.887744 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/caf8772a-4894-4924-9f18-f2b191aadad7-istio-data" pod "caf8772a-4894-4924-9f18-f2b191aadad7" (UID: "caf8772a-4894-4924-9f18-f2b191aadad7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:17:24.887892 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:24.887874 4376 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/caf8772a-4894-4924-9f18-f2b191aadad7-credential-socket\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:17:24.887964 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:24.887895 4376 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/caf8772a-4894-4924-9f18-f2b191aadad7-istiod-ca-cert\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:17:24.887964 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:24.887905 4376 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/caf8772a-4894-4924-9f18-f2b191aadad7-workload-socket\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:17:24.887964 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:24.887913 4376 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/caf8772a-4894-4924-9f18-f2b191aadad7-workload-certs\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:17:24.887964 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:24.887924 4376 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/caf8772a-4894-4924-9f18-f2b191aadad7-istio-data\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:17:24.889510 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:24.889474 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/caf8772a-4894-4924-9f18-f2b191aadad7-istio-envoy" pod "caf8772a-4894-4924-9f18-f2b191aadad7" (UID: "caf8772a-4894-4924-9f18-f2b191aadad7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:17:24.889643 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:24.889601 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/caf8772a-4894-4924-9f18-f2b191aadad7-istio-token" pod "caf8772a-4894-4924-9f18-f2b191aadad7" (UID: "caf8772a-4894-4924-9f18-f2b191aadad7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:17:24.889750 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:24.889730 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/caf8772a-4894-4924-9f18-f2b191aadad7-kube-api-access-55vn7" pod "caf8772a-4894-4924-9f18-f2b191aadad7" (UID: "caf8772a-4894-4924-9f18-f2b191aadad7"). InnerVolumeSpecName "kube-api-access-55vn7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:17:24.889801 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:24.889759 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/caf8772a-4894-4924-9f18-f2b191aadad7-istio-podinfo" pod "caf8772a-4894-4924-9f18-f2b191aadad7" (UID: "caf8772a-4894-4924-9f18-f2b191aadad7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 05:17:24.989472 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:24.989437 4376 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/caf8772a-4894-4924-9f18-f2b191aadad7-istio-podinfo\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:17:24.989472 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:24.989464 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55vn7\" (UniqueName: \"kubernetes.io/projected/caf8772a-4894-4924-9f18-f2b191aadad7-kube-api-access-55vn7\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:17:24.989472 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:24.989476 4376 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/caf8772a-4894-4924-9f18-f2b191aadad7-istio-token\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:17:24.989472 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:24.989484 4376 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/caf8772a-4894-4924-9f18-f2b191aadad7-istio-envoy\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:17:25.725407 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:25.725374 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-25f56e-openshift-default-6465df847-6j725" event={"ID":"caf8772a-4894-4924-9f18-f2b191aadad7","Type":"ContainerDied","Data":"40b252fe9e99746909a575a9ee76452605feaee2da3a1e6df9b33da8548421fd"} Oct 31 05:17:25.725946 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:25.725426 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-25f56e-openshift-default-6465df847-6j725" Oct 31 05:17:25.725946 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:25.725434 4376 scope.go:122] "RemoveContainer" containerID="f404f437b5bd699a47c12a58483993d7b7b47ad54b19ee53bb2b7863526c98f2" Oct 31 05:17:25.798640 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:25.798606 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-25f56e-openshift-default-6465df847-6j725"] Oct 31 05:17:25.812786 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:25.812748 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-25f56e-openshift-default-6465df847-6j725"] Oct 31 05:17:26.443624 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:26.443589 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="caf8772a-4894-4924-9f18-f2b191aadad7" path="/var/lib/kubelet/pods/caf8772a-4894-4924-9f18-f2b191aadad7/volumes" Oct 31 05:17:27.032930 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:27.032906 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b2a372-84cc9765b7-8q729" Oct 31 05:17:27.155030 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:27.154986 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 05:17:27.213138 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:27.213101 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8414bff7-0d22-4ccd-8c0d-e6c2dd294022-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8414bff7-0d22-4ccd-8c0d-e6c2dd294022-plugins-config-volume\") pod \"8414bff7-0d22-4ccd-8c0d-e6c2dd294022\" (UID: \"8414bff7-0d22-4ccd-8c0d-e6c2dd294022\") " Oct 31 05:17:27.213349 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:27.213181 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8414bff7-0d22-4ccd-8c0d-e6c2dd294022-kube-api-access-j5pmw\" (UniqueName: \"kubernetes.io/projected/8414bff7-0d22-4ccd-8c0d-e6c2dd294022-kube-api-access-j5pmw\") pod \"8414bff7-0d22-4ccd-8c0d-e6c2dd294022\" (UID: \"8414bff7-0d22-4ccd-8c0d-e6c2dd294022\") " Oct 31 05:17:27.213561 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:27.213537 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8414bff7-0d22-4ccd-8c0d-e6c2dd294022-plugins-config-volume" pod "8414bff7-0d22-4ccd-8c0d-e6c2dd294022" (UID: "8414bff7-0d22-4ccd-8c0d-e6c2dd294022"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:17:27.215373 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:27.215343 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8414bff7-0d22-4ccd-8c0d-e6c2dd294022-kube-api-access-j5pmw" pod "8414bff7-0d22-4ccd-8c0d-e6c2dd294022" (UID: "8414bff7-0d22-4ccd-8c0d-e6c2dd294022"). InnerVolumeSpecName "kube-api-access-j5pmw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:17:27.314656 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:27.314617 4376 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8414bff7-0d22-4ccd-8c0d-e6c2dd294022-plugins-config-volume\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:17:27.314656 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:27.314647 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j5pmw\" (UniqueName: \"kubernetes.io/projected/8414bff7-0d22-4ccd-8c0d-e6c2dd294022-kube-api-access-j5pmw\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:17:27.738239 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:27.738192 4376 generic.go:356] "Generic (PLEG): container finished" podID="8414bff7-0d22-4ccd-8c0d-e6c2dd294022" containerID="49a566f597506c107acb92205ef5700da38eadccc16245a23304d33651a6be56" exitCode=0 Oct 31 05:17:27.738452 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:27.738262 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b2a372-84cc9765b7-8q729" event={"ID":"8414bff7-0d22-4ccd-8c0d-e6c2dd294022","Type":"ContainerDied","Data":"49a566f597506c107acb92205ef5700da38eadccc16245a23304d33651a6be56"} Oct 31 05:17:27.738452 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:27.738256 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b2a372-84cc9765b7-8q729" Oct 31 05:17:27.738452 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:27.738370 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b2a372-84cc9765b7-8q729" event={"ID":"8414bff7-0d22-4ccd-8c0d-e6c2dd294022","Type":"ContainerDied","Data":"016939757b72b4cfae531417bc6b1b7d2fea8a18b83ab819a61b0d2b1c430485"} Oct 31 05:17:27.738452 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:27.738399 4376 scope.go:122] "RemoveContainer" containerID="49a566f597506c107acb92205ef5700da38eadccc16245a23304d33651a6be56" Oct 31 05:17:27.748635 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:27.748607 4376 scope.go:122] "RemoveContainer" containerID="49a566f597506c107acb92205ef5700da38eadccc16245a23304d33651a6be56" Oct 31 05:17:27.752558 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:17:27.752531 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49a566f597506c107acb92205ef5700da38eadccc16245a23304d33651a6be56\": container with ID starting with 49a566f597506c107acb92205ef5700da38eadccc16245a23304d33651a6be56 not found: ID does not exist" containerID="49a566f597506c107acb92205ef5700da38eadccc16245a23304d33651a6be56" Oct 31 05:17:27.752656 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:27.752573 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49a566f597506c107acb92205ef5700da38eadccc16245a23304d33651a6be56"} err="failed to get container status \"49a566f597506c107acb92205ef5700da38eadccc16245a23304d33651a6be56\": rpc error: code = NotFound desc = could not find container \"49a566f597506c107acb92205ef5700da38eadccc16245a23304d33651a6be56\": container with ID starting with 49a566f597506c107acb92205ef5700da38eadccc16245a23304d33651a6be56 not found: ID does not exist" Oct 31 05:17:27.801980 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:27.801941 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b2a372-84cc9765b7-8q729"] Oct 31 05:17:27.805368 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:27.805319 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b2a372-84cc9765b7-8q729"] Oct 31 05:17:28.443837 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:28.443789 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8414bff7-0d22-4ccd-8c0d-e6c2dd294022" path="/var/lib/kubelet/pods/8414bff7-0d22-4ccd-8c0d-e6c2dd294022/volumes" Oct 31 05:17:29.959344 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:29.959301 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-25f56e-bd47bbbd9-r4fhf" Oct 31 05:17:29.999430 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:29.999408 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-25f56e-69577647c5-wvb9t" Oct 31 05:17:30.043620 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:30.043592 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43d89e12-3149-41df-825f-40801de7576a-kube-api-access-d8tvs\" (UniqueName: \"kubernetes.io/projected/43d89e12-3149-41df-825f-40801de7576a-kube-api-access-d8tvs\") pod \"43d89e12-3149-41df-825f-40801de7576a\" (UID: \"43d89e12-3149-41df-825f-40801de7576a\") " Oct 31 05:17:30.043804 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:30.043643 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69d5f412-aaf9-4d2a-afb0-3f76f0861259-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69d5f412-aaf9-4d2a-afb0-3f76f0861259-plugins-config-volume\") pod \"69d5f412-aaf9-4d2a-afb0-3f76f0861259\" (UID: \"69d5f412-aaf9-4d2a-afb0-3f76f0861259\") " Oct 31 05:17:30.043804 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:30.043692 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43d89e12-3149-41df-825f-40801de7576a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43d89e12-3149-41df-825f-40801de7576a-plugins-config-volume\") pod \"43d89e12-3149-41df-825f-40801de7576a\" (UID: \"43d89e12-3149-41df-825f-40801de7576a\") " Oct 31 05:17:30.043804 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:30.043715 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69d5f412-aaf9-4d2a-afb0-3f76f0861259-kube-api-access-s99jg\" (UniqueName: \"kubernetes.io/projected/69d5f412-aaf9-4d2a-afb0-3f76f0861259-kube-api-access-s99jg\") pod \"69d5f412-aaf9-4d2a-afb0-3f76f0861259\" (UID: \"69d5f412-aaf9-4d2a-afb0-3f76f0861259\") " Oct 31 05:17:30.044083 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:30.044053 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69d5f412-aaf9-4d2a-afb0-3f76f0861259-plugins-config-volume" pod "69d5f412-aaf9-4d2a-afb0-3f76f0861259" (UID: "69d5f412-aaf9-4d2a-afb0-3f76f0861259"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:17:30.044207 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:30.044098 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43d89e12-3149-41df-825f-40801de7576a-plugins-config-volume" pod "43d89e12-3149-41df-825f-40801de7576a" (UID: "43d89e12-3149-41df-825f-40801de7576a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:17:30.045822 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:30.045794 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43d89e12-3149-41df-825f-40801de7576a-kube-api-access-d8tvs" pod "43d89e12-3149-41df-825f-40801de7576a" (UID: "43d89e12-3149-41df-825f-40801de7576a"). InnerVolumeSpecName "kube-api-access-d8tvs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:17:30.045888 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:30.045860 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69d5f412-aaf9-4d2a-afb0-3f76f0861259-kube-api-access-s99jg" pod "69d5f412-aaf9-4d2a-afb0-3f76f0861259" (UID: "69d5f412-aaf9-4d2a-afb0-3f76f0861259"). InnerVolumeSpecName "kube-api-access-s99jg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:17:30.144719 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:30.144684 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8tvs\" (UniqueName: \"kubernetes.io/projected/43d89e12-3149-41df-825f-40801de7576a-kube-api-access-d8tvs\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:17:30.144719 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:30.144715 4376 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69d5f412-aaf9-4d2a-afb0-3f76f0861259-plugins-config-volume\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:17:30.144719 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:30.144725 4376 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43d89e12-3149-41df-825f-40801de7576a-plugins-config-volume\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:17:30.144976 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:30.144734 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s99jg\" (UniqueName: \"kubernetes.io/projected/69d5f412-aaf9-4d2a-afb0-3f76f0861259-kube-api-access-s99jg\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:17:30.755963 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:30.755877 4376 generic.go:356] "Generic (PLEG): container finished" podID="43d89e12-3149-41df-825f-40801de7576a" containerID="9aa705e067a14e38ce68e66c9b17454a1612d5d2566fd7ca585063485bbec729" exitCode=0 Oct 31 05:17:30.755963 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:30.755926 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-25f56e-bd47bbbd9-r4fhf" Oct 31 05:17:30.756213 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:30.755969 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-25f56e-bd47bbbd9-r4fhf" event={"ID":"43d89e12-3149-41df-825f-40801de7576a","Type":"ContainerDied","Data":"9aa705e067a14e38ce68e66c9b17454a1612d5d2566fd7ca585063485bbec729"} Oct 31 05:17:30.756213 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:30.756028 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-25f56e-bd47bbbd9-r4fhf" event={"ID":"43d89e12-3149-41df-825f-40801de7576a","Type":"ContainerDied","Data":"0af22bd332e4a66b41d2baee11349ed366462ceae9504e004025c0dfbf0ca70d"} Oct 31 05:17:30.756213 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:30.756055 4376 scope.go:122] "RemoveContainer" containerID="9aa705e067a14e38ce68e66c9b17454a1612d5d2566fd7ca585063485bbec729" Oct 31 05:17:30.757309 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:30.757286 4376 generic.go:356] "Generic (PLEG): container finished" podID="69d5f412-aaf9-4d2a-afb0-3f76f0861259" containerID="90b51dfd6d3297a2dbbc7bfc2f535824c07608cd387cb1587293c431c18c9dc0" exitCode=0 Oct 31 05:17:30.757451 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:30.757314 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-25f56e-69577647c5-wvb9t" event={"ID":"69d5f412-aaf9-4d2a-afb0-3f76f0861259","Type":"ContainerDied","Data":"90b51dfd6d3297a2dbbc7bfc2f535824c07608cd387cb1587293c431c18c9dc0"} Oct 31 05:17:30.757451 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:30.757345 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-25f56e-69577647c5-wvb9t" Oct 31 05:17:30.757451 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:30.757382 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-25f56e-69577647c5-wvb9t" event={"ID":"69d5f412-aaf9-4d2a-afb0-3f76f0861259","Type":"ContainerDied","Data":"9f7233d559ca8f4166d1295101f33b7a64337d2b75aa911322467ce167e184ad"} Oct 31 05:17:30.764877 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:30.764859 4376 scope.go:122] "RemoveContainer" containerID="9aa705e067a14e38ce68e66c9b17454a1612d5d2566fd7ca585063485bbec729" Oct 31 05:17:30.765162 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:17:30.765143 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9aa705e067a14e38ce68e66c9b17454a1612d5d2566fd7ca585063485bbec729\": container with ID starting with 9aa705e067a14e38ce68e66c9b17454a1612d5d2566fd7ca585063485bbec729 not found: ID does not exist" containerID="9aa705e067a14e38ce68e66c9b17454a1612d5d2566fd7ca585063485bbec729" Oct 31 05:17:30.765257 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:30.765170 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9aa705e067a14e38ce68e66c9b17454a1612d5d2566fd7ca585063485bbec729"} err="failed to get container status \"9aa705e067a14e38ce68e66c9b17454a1612d5d2566fd7ca585063485bbec729\": rpc error: code = NotFound desc = could not find container \"9aa705e067a14e38ce68e66c9b17454a1612d5d2566fd7ca585063485bbec729\": container with ID starting with 9aa705e067a14e38ce68e66c9b17454a1612d5d2566fd7ca585063485bbec729 not found: ID does not exist" Oct 31 05:17:30.765257 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:30.765185 4376 scope.go:122] "RemoveContainer" containerID="90b51dfd6d3297a2dbbc7bfc2f535824c07608cd387cb1587293c431c18c9dc0" Oct 31 05:17:30.772823 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:30.772801 4376 scope.go:122] "RemoveContainer" containerID="90b51dfd6d3297a2dbbc7bfc2f535824c07608cd387cb1587293c431c18c9dc0" Oct 31 05:17:30.773091 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:17:30.773067 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90b51dfd6d3297a2dbbc7bfc2f535824c07608cd387cb1587293c431c18c9dc0\": container with ID starting with 90b51dfd6d3297a2dbbc7bfc2f535824c07608cd387cb1587293c431c18c9dc0 not found: ID does not exist" containerID="90b51dfd6d3297a2dbbc7bfc2f535824c07608cd387cb1587293c431c18c9dc0" Oct 31 05:17:30.773161 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:30.773098 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90b51dfd6d3297a2dbbc7bfc2f535824c07608cd387cb1587293c431c18c9dc0"} err="failed to get container status \"90b51dfd6d3297a2dbbc7bfc2f535824c07608cd387cb1587293c431c18c9dc0\": rpc error: code = NotFound desc = could not find container \"90b51dfd6d3297a2dbbc7bfc2f535824c07608cd387cb1587293c431c18c9dc0\": container with ID starting with 90b51dfd6d3297a2dbbc7bfc2f535824c07608cd387cb1587293c431c18c9dc0 not found: ID does not exist" Oct 31 05:17:30.813525 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:30.813496 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-25f56e-69577647c5-wvb9t"] Oct 31 05:17:30.817669 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:30.817643 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-25f56e-69577647c5-wvb9t"] Oct 31 05:17:30.862270 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:30.862232 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-25f56e-bd47bbbd9-r4fhf"] Oct 31 05:17:30.868155 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:30.868126 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-25f56e-bd47bbbd9-r4fhf"] Oct 31 05:17:31.903961 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:31.903919 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-965f61-576d68cfbf-ffkvw"] Oct 31 05:17:31.938264 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:31.938226 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-965f61-84cf84d479-gq7q5"] Oct 31 05:17:32.442497 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:32.442461 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43d89e12-3149-41df-825f-40801de7576a" path="/var/lib/kubelet/pods/43d89e12-3149-41df-825f-40801de7576a/volumes" Oct 31 05:17:32.442861 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:32.442845 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69d5f412-aaf9-4d2a-afb0-3f76f0861259" path="/var/lib/kubelet/pods/69d5f412-aaf9-4d2a-afb0-3f76f0861259/volumes" Oct 31 05:17:35.896385 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:35.896306 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-4c2387/distinct-w3-4c2387-kserve-768bdd84c7-6jmtl" podUID="b2c75691-4cf4-484b-96e2-3c2503b8d78c" containerName="main" containerID="cri-o://e601b65852ff9096dfc957fb7cdb571aef3670633546efe666ee6fc47a19f36b" gracePeriod=45 Oct 31 05:17:35.938121 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:35.938079 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4c2387/distinct-w3-4c2387-kserve-768bdd84c7-6jmtl" podUID="b2c75691-4cf4-484b-96e2-3c2503b8d78c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": dial tcp 10.133.0.72:8000: connect: connection refused" Oct 31 05:17:36.144571 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:36.144547 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-4c2387/distinct-w3-4c2387-kserve-768bdd84c7-6jmtl" Oct 31 05:17:36.209153 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:36.209075 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2c75691-4cf4-484b-96e2-3c2503b8d78c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2c75691-4cf4-484b-96e2-3c2503b8d78c-model-cache\") pod \"b2c75691-4cf4-484b-96e2-3c2503b8d78c\" (UID: \"b2c75691-4cf4-484b-96e2-3c2503b8d78c\") " Oct 31 05:17:36.209153 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:36.209137 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b2c75691-4cf4-484b-96e2-3c2503b8d78c-tls-certs\" (UniqueName: \"kubernetes.io/secret/b2c75691-4cf4-484b-96e2-3c2503b8d78c-tls-certs\") pod \"b2c75691-4cf4-484b-96e2-3c2503b8d78c\" (UID: \"b2c75691-4cf4-484b-96e2-3c2503b8d78c\") " Oct 31 05:17:36.209389 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:36.209207 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2c75691-4cf4-484b-96e2-3c2503b8d78c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2c75691-4cf4-484b-96e2-3c2503b8d78c-dshm\") pod \"b2c75691-4cf4-484b-96e2-3c2503b8d78c\" (UID: \"b2c75691-4cf4-484b-96e2-3c2503b8d78c\") " Oct 31 05:17:36.209389 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:36.209322 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2c75691-4cf4-484b-96e2-3c2503b8d78c-kube-api-access-k2bdk\" (UniqueName: \"kubernetes.io/projected/b2c75691-4cf4-484b-96e2-3c2503b8d78c-kube-api-access-k2bdk\") pod \"b2c75691-4cf4-484b-96e2-3c2503b8d78c\" (UID: \"b2c75691-4cf4-484b-96e2-3c2503b8d78c\") " Oct 31 05:17:36.209389 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:36.209373 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2c75691-4cf4-484b-96e2-3c2503b8d78c-model-cache" pod "b2c75691-4cf4-484b-96e2-3c2503b8d78c" (UID: "b2c75691-4cf4-484b-96e2-3c2503b8d78c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:17:36.209564 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:36.209412 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2c75691-4cf4-484b-96e2-3c2503b8d78c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2c75691-4cf4-484b-96e2-3c2503b8d78c-tmp-dir\") pod \"b2c75691-4cf4-484b-96e2-3c2503b8d78c\" (UID: \"b2c75691-4cf4-484b-96e2-3c2503b8d78c\") " Oct 31 05:17:36.209564 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:36.209446 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2c75691-4cf4-484b-96e2-3c2503b8d78c-home\" (UniqueName: \"kubernetes.io/empty-dir/b2c75691-4cf4-484b-96e2-3c2503b8d78c-home\") pod \"b2c75691-4cf4-484b-96e2-3c2503b8d78c\" (UID: \"b2c75691-4cf4-484b-96e2-3c2503b8d78c\") " Oct 31 05:17:36.209699 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:36.209668 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2c75691-4cf4-484b-96e2-3c2503b8d78c-tmp-dir" pod "b2c75691-4cf4-484b-96e2-3c2503b8d78c" (UID: "b2c75691-4cf4-484b-96e2-3c2503b8d78c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:17:36.209909 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:36.209878 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2c75691-4cf4-484b-96e2-3c2503b8d78c-home" pod "b2c75691-4cf4-484b-96e2-3c2503b8d78c" (UID: "b2c75691-4cf4-484b-96e2-3c2503b8d78c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:17:36.209992 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:36.209911 4376 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2c75691-4cf4-484b-96e2-3c2503b8d78c-tmp-dir\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:17:36.209992 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:36.209938 4376 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2c75691-4cf4-484b-96e2-3c2503b8d78c-model-cache\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:17:36.211799 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:36.211766 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2c75691-4cf4-484b-96e2-3c2503b8d78c-kube-api-access-k2bdk" pod "b2c75691-4cf4-484b-96e2-3c2503b8d78c" (UID: "b2c75691-4cf4-484b-96e2-3c2503b8d78c"). InnerVolumeSpecName "kube-api-access-k2bdk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:17:36.211929 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:36.211852 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2c75691-4cf4-484b-96e2-3c2503b8d78c-dshm" pod "b2c75691-4cf4-484b-96e2-3c2503b8d78c" (UID: "b2c75691-4cf4-484b-96e2-3c2503b8d78c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:17:36.212087 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:36.212058 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2c75691-4cf4-484b-96e2-3c2503b8d78c-tls-certs" pod "b2c75691-4cf4-484b-96e2-3c2503b8d78c" (UID: "b2c75691-4cf4-484b-96e2-3c2503b8d78c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 05:17:36.310890 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:36.310857 4376 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2c75691-4cf4-484b-96e2-3c2503b8d78c-dshm\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:17:36.310890 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:36.310888 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2bdk\" (UniqueName: \"kubernetes.io/projected/b2c75691-4cf4-484b-96e2-3c2503b8d78c-kube-api-access-k2bdk\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:17:36.311093 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:36.310902 4376 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2c75691-4cf4-484b-96e2-3c2503b8d78c-home\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:17:36.311093 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:36.310914 4376 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2c75691-4cf4-484b-96e2-3c2503b8d78c-tls-certs\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:17:36.791807 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:36.791769 4376 generic.go:356] "Generic (PLEG): container finished" podID="b2c75691-4cf4-484b-96e2-3c2503b8d78c" containerID="e601b65852ff9096dfc957fb7cdb571aef3670633546efe666ee6fc47a19f36b" exitCode=0 Oct 31 05:17:36.792041 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:36.791822 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-4c2387/distinct-w3-4c2387-kserve-768bdd84c7-6jmtl" Oct 31 05:17:36.792041 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:36.791862 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-4c2387/distinct-w3-4c2387-kserve-768bdd84c7-6jmtl" event={"ID":"b2c75691-4cf4-484b-96e2-3c2503b8d78c","Type":"ContainerDied","Data":"e601b65852ff9096dfc957fb7cdb571aef3670633546efe666ee6fc47a19f36b"} Oct 31 05:17:36.792041 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:36.791911 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-4c2387/distinct-w3-4c2387-kserve-768bdd84c7-6jmtl" event={"ID":"b2c75691-4cf4-484b-96e2-3c2503b8d78c","Type":"ContainerDied","Data":"bae544aa0168a3a9fe9bbb08132d74cf32cec489df228faf668e547ac3e15844"} Oct 31 05:17:36.792041 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:36.791929 4376 scope.go:122] "RemoveContainer" containerID="e601b65852ff9096dfc957fb7cdb571aef3670633546efe666ee6fc47a19f36b" Oct 31 05:17:36.800517 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:36.800491 4376 scope.go:122] "RemoveContainer" containerID="e601b65852ff9096dfc957fb7cdb571aef3670633546efe666ee6fc47a19f36b" Oct 31 05:17:36.800814 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:17:36.800793 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e601b65852ff9096dfc957fb7cdb571aef3670633546efe666ee6fc47a19f36b\": container with ID starting with e601b65852ff9096dfc957fb7cdb571aef3670633546efe666ee6fc47a19f36b not found: ID does not exist" containerID="e601b65852ff9096dfc957fb7cdb571aef3670633546efe666ee6fc47a19f36b" Oct 31 05:17:36.800863 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:36.800823 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e601b65852ff9096dfc957fb7cdb571aef3670633546efe666ee6fc47a19f36b"} err="failed to get container status \"e601b65852ff9096dfc957fb7cdb571aef3670633546efe666ee6fc47a19f36b\": rpc error: code = NotFound desc = could not find container \"e601b65852ff9096dfc957fb7cdb571aef3670633546efe666ee6fc47a19f36b\": container with ID starting with e601b65852ff9096dfc957fb7cdb571aef3670633546efe666ee6fc47a19f36b not found: ID does not exist" Oct 31 05:17:36.849818 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:36.849784 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-4c2387/distinct-w3-4c2387-kserve-768bdd84c7-6jmtl"] Oct 31 05:17:36.863842 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:36.863806 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-4c2387/distinct-w3-4c2387-kserve-768bdd84c7-6jmtl"] Oct 31 05:17:36.904881 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:36.904839 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-965f61-576d68cfbf-ffkvw" podUID="d2870b2c-9c85-43d1-a726-38edfcf6ae2b" containerName="payload-processing" containerID="cri-o://a8d41f023bf21e1e86500f2f77165da0241e08a284c500d343e05fcfede33883" gracePeriod=25 Oct 31 05:17:36.939554 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:36.939505 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-965f61-84cf84d479-gq7q5" podUID="2219d6d2-75e4-4fd8-ae05-901e3ac891a1" containerName="payload-pre-processing" containerID="cri-o://8c2df74c6a9dad654e97f0509fd434a7aacb58e86933be659e7592df76eed988" gracePeriod=25 Oct 31 05:17:38.441902 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:38.441866 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2c75691-4cf4-484b-96e2-3c2503b8d78c" path="/var/lib/kubelet/pods/b2c75691-4cf4-484b-96e2-3c2503b8d78c/volumes" Oct 31 05:17:39.134209 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:39.134172 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 05:17:40.999586 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:17:40.999550 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 05:17:41.178390 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:41.178356 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-965f61-84cf84d479-gq7q5" podUID="2219d6d2-75e4-4fd8-ae05-901e3ac891a1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 05:17:41.178569 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:41.178365 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-965f61-576d68cfbf-ffkvw" podUID="d2870b2c-9c85-43d1-a726-38edfcf6ae2b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 05:17:51.177994 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:51.177950 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-965f61-84cf84d479-gq7q5" podUID="2219d6d2-75e4-4fd8-ae05-901e3ac891a1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 05:17:51.177994 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:51.177966 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-965f61-576d68cfbf-ffkvw" podUID="d2870b2c-9c85-43d1-a726-38edfcf6ae2b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 05:17:55.430764 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:55.430722 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 05:17:57.192891 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:57.192859 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-965f61-576d68cfbf-ffkvw" Oct 31 05:17:57.244180 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:57.244152 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-965f61-84cf84d479-gq7q5" Oct 31 05:17:57.331001 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:57.330965 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2870b2c-9c85-43d1-a726-38edfcf6ae2b-kube-api-access-w5z6w\" (UniqueName: \"kubernetes.io/projected/d2870b2c-9c85-43d1-a726-38edfcf6ae2b-kube-api-access-w5z6w\") pod \"d2870b2c-9c85-43d1-a726-38edfcf6ae2b\" (UID: \"d2870b2c-9c85-43d1-a726-38edfcf6ae2b\") " Oct 31 05:17:57.331185 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:57.331083 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2870b2c-9c85-43d1-a726-38edfcf6ae2b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2870b2c-9c85-43d1-a726-38edfcf6ae2b-plugins-config-volume\") pod \"d2870b2c-9c85-43d1-a726-38edfcf6ae2b\" (UID: \"d2870b2c-9c85-43d1-a726-38edfcf6ae2b\") " Oct 31 05:17:57.331504 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:57.331479 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2870b2c-9c85-43d1-a726-38edfcf6ae2b-plugins-config-volume" pod "d2870b2c-9c85-43d1-a726-38edfcf6ae2b" (UID: "d2870b2c-9c85-43d1-a726-38edfcf6ae2b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:17:57.333181 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:57.333156 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2870b2c-9c85-43d1-a726-38edfcf6ae2b-kube-api-access-w5z6w" pod "d2870b2c-9c85-43d1-a726-38edfcf6ae2b" (UID: "d2870b2c-9c85-43d1-a726-38edfcf6ae2b"). InnerVolumeSpecName "kube-api-access-w5z6w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:17:57.432243 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:57.432146 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2219d6d2-75e4-4fd8-ae05-901e3ac891a1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2219d6d2-75e4-4fd8-ae05-901e3ac891a1-plugins-config-volume\") pod \"2219d6d2-75e4-4fd8-ae05-901e3ac891a1\" (UID: \"2219d6d2-75e4-4fd8-ae05-901e3ac891a1\") " Oct 31 05:17:57.432243 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:57.432205 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2219d6d2-75e4-4fd8-ae05-901e3ac891a1-kube-api-access-7blqt\" (UniqueName: \"kubernetes.io/projected/2219d6d2-75e4-4fd8-ae05-901e3ac891a1-kube-api-access-7blqt\") pod \"2219d6d2-75e4-4fd8-ae05-901e3ac891a1\" (UID: \"2219d6d2-75e4-4fd8-ae05-901e3ac891a1\") " Oct 31 05:17:57.432471 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:57.432439 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5z6w\" (UniqueName: \"kubernetes.io/projected/d2870b2c-9c85-43d1-a726-38edfcf6ae2b-kube-api-access-w5z6w\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:17:57.432471 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:57.432453 4376 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2870b2c-9c85-43d1-a726-38edfcf6ae2b-plugins-config-volume\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:17:57.432581 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:57.432558 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2219d6d2-75e4-4fd8-ae05-901e3ac891a1-plugins-config-volume" pod "2219d6d2-75e4-4fd8-ae05-901e3ac891a1" (UID: "2219d6d2-75e4-4fd8-ae05-901e3ac891a1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:17:57.434361 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:57.434317 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2219d6d2-75e4-4fd8-ae05-901e3ac891a1-kube-api-access-7blqt" pod "2219d6d2-75e4-4fd8-ae05-901e3ac891a1" (UID: "2219d6d2-75e4-4fd8-ae05-901e3ac891a1"). InnerVolumeSpecName "kube-api-access-7blqt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:17:57.534004 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:57.533962 4376 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2219d6d2-75e4-4fd8-ae05-901e3ac891a1-plugins-config-volume\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:17:57.534004 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:57.533999 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7blqt\" (UniqueName: \"kubernetes.io/projected/2219d6d2-75e4-4fd8-ae05-901e3ac891a1-kube-api-access-7blqt\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:17:57.896453 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:57.896414 4376 generic.go:356] "Generic (PLEG): container finished" podID="2219d6d2-75e4-4fd8-ae05-901e3ac891a1" containerID="8c2df74c6a9dad654e97f0509fd434a7aacb58e86933be659e7592df76eed988" exitCode=0 Oct 31 05:17:57.896666 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:57.896462 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-965f61-84cf84d479-gq7q5" Oct 31 05:17:57.896666 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:57.896503 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-965f61-84cf84d479-gq7q5" event={"ID":"2219d6d2-75e4-4fd8-ae05-901e3ac891a1","Type":"ContainerDied","Data":"8c2df74c6a9dad654e97f0509fd434a7aacb58e86933be659e7592df76eed988"} Oct 31 05:17:57.896666 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:57.896563 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-965f61-84cf84d479-gq7q5" event={"ID":"2219d6d2-75e4-4fd8-ae05-901e3ac891a1","Type":"ContainerDied","Data":"ff435e3dfb7eb0ddcd685d214c5d930d01f1dff36886705124784852c0d3a58a"} Oct 31 05:17:57.896666 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:57.896595 4376 scope.go:122] "RemoveContainer" containerID="8c2df74c6a9dad654e97f0509fd434a7aacb58e86933be659e7592df76eed988" Oct 31 05:17:57.898166 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:57.898144 4376 generic.go:356] "Generic (PLEG): container finished" podID="d2870b2c-9c85-43d1-a726-38edfcf6ae2b" containerID="a8d41f023bf21e1e86500f2f77165da0241e08a284c500d343e05fcfede33883" exitCode=0 Oct 31 05:17:57.898294 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:57.898191 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-965f61-576d68cfbf-ffkvw" Oct 31 05:17:57.898294 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:57.898239 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-965f61-576d68cfbf-ffkvw" event={"ID":"d2870b2c-9c85-43d1-a726-38edfcf6ae2b","Type":"ContainerDied","Data":"a8d41f023bf21e1e86500f2f77165da0241e08a284c500d343e05fcfede33883"} Oct 31 05:17:57.898294 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:57.898281 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-965f61-576d68cfbf-ffkvw" event={"ID":"d2870b2c-9c85-43d1-a726-38edfcf6ae2b","Type":"ContainerDied","Data":"30aee2111452a27e7e496737b3d4a48a2fdfbbf354154af2bdd40416f66ac000"} Oct 31 05:17:57.906570 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:57.906544 4376 scope.go:122] "RemoveContainer" containerID="8c2df74c6a9dad654e97f0509fd434a7aacb58e86933be659e7592df76eed988" Oct 31 05:17:57.907672 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:17:57.907635 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c2df74c6a9dad654e97f0509fd434a7aacb58e86933be659e7592df76eed988\": container with ID starting with 8c2df74c6a9dad654e97f0509fd434a7aacb58e86933be659e7592df76eed988 not found: ID does not exist" containerID="8c2df74c6a9dad654e97f0509fd434a7aacb58e86933be659e7592df76eed988" Oct 31 05:17:57.907808 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:57.907680 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c2df74c6a9dad654e97f0509fd434a7aacb58e86933be659e7592df76eed988"} err="failed to get container status \"8c2df74c6a9dad654e97f0509fd434a7aacb58e86933be659e7592df76eed988\": rpc error: code = NotFound desc = could not find container \"8c2df74c6a9dad654e97f0509fd434a7aacb58e86933be659e7592df76eed988\": container with ID starting with 8c2df74c6a9dad654e97f0509fd434a7aacb58e86933be659e7592df76eed988 not found: ID does not exist" Oct 31 05:17:57.907808 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:57.907698 4376 scope.go:122] "RemoveContainer" containerID="a8d41f023bf21e1e86500f2f77165da0241e08a284c500d343e05fcfede33883" Oct 31 05:17:57.924583 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:57.924562 4376 scope.go:122] "RemoveContainer" containerID="a8d41f023bf21e1e86500f2f77165da0241e08a284c500d343e05fcfede33883" Oct 31 05:17:57.924929 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:17:57.924880 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8d41f023bf21e1e86500f2f77165da0241e08a284c500d343e05fcfede33883\": container with ID starting with a8d41f023bf21e1e86500f2f77165da0241e08a284c500d343e05fcfede33883 not found: ID does not exist" containerID="a8d41f023bf21e1e86500f2f77165da0241e08a284c500d343e05fcfede33883" Oct 31 05:17:57.924929 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:57.924911 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8d41f023bf21e1e86500f2f77165da0241e08a284c500d343e05fcfede33883"} err="failed to get container status \"a8d41f023bf21e1e86500f2f77165da0241e08a284c500d343e05fcfede33883\": rpc error: code = NotFound desc = could not find container \"a8d41f023bf21e1e86500f2f77165da0241e08a284c500d343e05fcfede33883\": container with ID starting with a8d41f023bf21e1e86500f2f77165da0241e08a284c500d343e05fcfede33883 not found: ID does not exist" Oct 31 05:17:57.953230 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:57.953200 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-965f61-84cf84d479-gq7q5"] Oct 31 05:17:57.969424 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:57.969388 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-965f61-84cf84d479-gq7q5"] Oct 31 05:17:57.997556 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:57.997527 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-965f61-576d68cfbf-ffkvw"] Oct 31 05:17:58.005981 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:58.005953 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-965f61-576d68cfbf-ffkvw"] Oct 31 05:17:58.442226 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:58.442195 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2219d6d2-75e4-4fd8-ae05-901e3ac891a1" path="/var/lib/kubelet/pods/2219d6d2-75e4-4fd8-ae05-901e3ac891a1/volumes" Oct 31 05:17:58.442638 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:58.442599 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2870b2c-9c85-43d1-a726-38edfcf6ae2b" path="/var/lib/kubelet/pods/d2870b2c-9c85-43d1-a726-38edfcf6ae2b/volumes" Oct 31 05:17:59.820481 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:59.820441 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-d7frf"] Oct 31 05:17:59.821266 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:17:59.820744 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-d7frf" podUID="dfcd34ac-2ed4-4f1e-bd06-9cca74916956" containerName="istio-proxy" containerID="cri-o://3ca2c47646dda985888ec2e91896aceb1af3f7d1b0c94def3e80def15de0862a" gracePeriod=30 Oct 31 05:18:00.726176 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:00.726134 4376 patch_prober.go:29] interesting pod/e2e-worker-w3-4c2387-openshift-default-5b68d44445-d7frf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 05:18:00.726371 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:00.726206 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-d7frf" podUID="dfcd34ac-2ed4-4f1e-bd06-9cca74916956" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 05:18:04.938953 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:04.938916 4376 generic.go:356] "Generic (PLEG): container finished" podID="dfcd34ac-2ed4-4f1e-bd06-9cca74916956" containerID="3ca2c47646dda985888ec2e91896aceb1af3f7d1b0c94def3e80def15de0862a" exitCode=0 Oct 31 05:18:04.939417 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:04.938991 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-d7frf" event={"ID":"dfcd34ac-2ed4-4f1e-bd06-9cca74916956","Type":"ContainerDied","Data":"3ca2c47646dda985888ec2e91896aceb1af3f7d1b0c94def3e80def15de0862a"} Oct 31 05:18:05.120901 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:05.120867 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-d7frf" Oct 31 05:18:05.216272 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:05.216238 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfcd34ac-2ed4-4f1e-bd06-9cca74916956-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfcd34ac-2ed4-4f1e-bd06-9cca74916956-workload-socket\") pod \"dfcd34ac-2ed4-4f1e-bd06-9cca74916956\" (UID: \"dfcd34ac-2ed4-4f1e-bd06-9cca74916956\") " Oct 31 05:18:05.216489 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:05.216296 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfcd34ac-2ed4-4f1e-bd06-9cca74916956-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfcd34ac-2ed4-4f1e-bd06-9cca74916956-workload-certs\") pod \"dfcd34ac-2ed4-4f1e-bd06-9cca74916956\" (UID: \"dfcd34ac-2ed4-4f1e-bd06-9cca74916956\") " Oct 31 05:18:05.216489 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:05.216319 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dfcd34ac-2ed4-4f1e-bd06-9cca74916956-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfcd34ac-2ed4-4f1e-bd06-9cca74916956-istiod-ca-cert\") pod \"dfcd34ac-2ed4-4f1e-bd06-9cca74916956\" (UID: \"dfcd34ac-2ed4-4f1e-bd06-9cca74916956\") " Oct 31 05:18:05.216489 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:05.216361 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfcd34ac-2ed4-4f1e-bd06-9cca74916956-kube-api-access-ms878\" (UniqueName: \"kubernetes.io/projected/dfcd34ac-2ed4-4f1e-bd06-9cca74916956-kube-api-access-ms878\") pod \"dfcd34ac-2ed4-4f1e-bd06-9cca74916956\" (UID: \"dfcd34ac-2ed4-4f1e-bd06-9cca74916956\") " Oct 31 05:18:05.216489 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:05.216393 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfcd34ac-2ed4-4f1e-bd06-9cca74916956-istio-token\" (UniqueName: \"kubernetes.io/projected/dfcd34ac-2ed4-4f1e-bd06-9cca74916956-istio-token\") pod \"dfcd34ac-2ed4-4f1e-bd06-9cca74916956\" (UID: \"dfcd34ac-2ed4-4f1e-bd06-9cca74916956\") " Oct 31 05:18:05.216489 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:05.216416 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfcd34ac-2ed4-4f1e-bd06-9cca74916956-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfcd34ac-2ed4-4f1e-bd06-9cca74916956-istio-data\") pod \"dfcd34ac-2ed4-4f1e-bd06-9cca74916956\" (UID: \"dfcd34ac-2ed4-4f1e-bd06-9cca74916956\") " Oct 31 05:18:05.216489 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:05.216476 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfcd34ac-2ed4-4f1e-bd06-9cca74916956-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfcd34ac-2ed4-4f1e-bd06-9cca74916956-istio-envoy\") pod \"dfcd34ac-2ed4-4f1e-bd06-9cca74916956\" (UID: \"dfcd34ac-2ed4-4f1e-bd06-9cca74916956\") " Oct 31 05:18:05.216854 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:05.216495 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dfcd34ac-2ed4-4f1e-bd06-9cca74916956-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfcd34ac-2ed4-4f1e-bd06-9cca74916956-istio-podinfo\") pod \"dfcd34ac-2ed4-4f1e-bd06-9cca74916956\" (UID: \"dfcd34ac-2ed4-4f1e-bd06-9cca74916956\") " Oct 31 05:18:05.216854 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:05.216523 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfcd34ac-2ed4-4f1e-bd06-9cca74916956-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfcd34ac-2ed4-4f1e-bd06-9cca74916956-credential-socket\") pod \"dfcd34ac-2ed4-4f1e-bd06-9cca74916956\" (UID: \"dfcd34ac-2ed4-4f1e-bd06-9cca74916956\") " Oct 31 05:18:05.216854 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:05.216588 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfcd34ac-2ed4-4f1e-bd06-9cca74916956-workload-certs" pod "dfcd34ac-2ed4-4f1e-bd06-9cca74916956" (UID: "dfcd34ac-2ed4-4f1e-bd06-9cca74916956"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:18:05.216854 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:05.216600 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfcd34ac-2ed4-4f1e-bd06-9cca74916956-workload-socket" pod "dfcd34ac-2ed4-4f1e-bd06-9cca74916956" (UID: "dfcd34ac-2ed4-4f1e-bd06-9cca74916956"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:18:05.216854 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:05.216745 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfcd34ac-2ed4-4f1e-bd06-9cca74916956-istio-data" pod "dfcd34ac-2ed4-4f1e-bd06-9cca74916956" (UID: "dfcd34ac-2ed4-4f1e-bd06-9cca74916956"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:18:05.217113 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:05.216867 4376 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfcd34ac-2ed4-4f1e-bd06-9cca74916956-workload-socket\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:18:05.217113 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:05.216894 4376 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfcd34ac-2ed4-4f1e-bd06-9cca74916956-workload-certs\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:18:05.217113 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:05.216965 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfcd34ac-2ed4-4f1e-bd06-9cca74916956-istiod-ca-cert" pod "dfcd34ac-2ed4-4f1e-bd06-9cca74916956" (UID: "dfcd34ac-2ed4-4f1e-bd06-9cca74916956"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:18:05.217113 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:05.216973 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfcd34ac-2ed4-4f1e-bd06-9cca74916956-credential-socket" pod "dfcd34ac-2ed4-4f1e-bd06-9cca74916956" (UID: "dfcd34ac-2ed4-4f1e-bd06-9cca74916956"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:18:05.217113 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:05.216982 4376 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfcd34ac-2ed4-4f1e-bd06-9cca74916956-istio-data\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:18:05.218725 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:05.218699 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfcd34ac-2ed4-4f1e-bd06-9cca74916956-istio-envoy" pod "dfcd34ac-2ed4-4f1e-bd06-9cca74916956" (UID: "dfcd34ac-2ed4-4f1e-bd06-9cca74916956"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:18:05.218831 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:05.218724 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfcd34ac-2ed4-4f1e-bd06-9cca74916956-kube-api-access-ms878" pod "dfcd34ac-2ed4-4f1e-bd06-9cca74916956" (UID: "dfcd34ac-2ed4-4f1e-bd06-9cca74916956"). InnerVolumeSpecName "kube-api-access-ms878". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:18:05.218990 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:05.218969 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfcd34ac-2ed4-4f1e-bd06-9cca74916956-istio-token" pod "dfcd34ac-2ed4-4f1e-bd06-9cca74916956" (UID: "dfcd34ac-2ed4-4f1e-bd06-9cca74916956"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:18:05.219090 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:05.219077 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dfcd34ac-2ed4-4f1e-bd06-9cca74916956-istio-podinfo" pod "dfcd34ac-2ed4-4f1e-bd06-9cca74916956" (UID: "dfcd34ac-2ed4-4f1e-bd06-9cca74916956"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 05:18:05.317820 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:05.317781 4376 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfcd34ac-2ed4-4f1e-bd06-9cca74916956-istiod-ca-cert\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:18:05.317820 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:05.317816 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ms878\" (UniqueName: \"kubernetes.io/projected/dfcd34ac-2ed4-4f1e-bd06-9cca74916956-kube-api-access-ms878\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:18:05.317820 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:05.317827 4376 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfcd34ac-2ed4-4f1e-bd06-9cca74916956-istio-token\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:18:05.318064 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:05.317836 4376 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfcd34ac-2ed4-4f1e-bd06-9cca74916956-istio-envoy\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:18:05.318064 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:05.317844 4376 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfcd34ac-2ed4-4f1e-bd06-9cca74916956-istio-podinfo\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:18:05.318064 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:05.317851 4376 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfcd34ac-2ed4-4f1e-bd06-9cca74916956-credential-socket\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:18:05.945557 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:05.945517 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-d7frf" event={"ID":"dfcd34ac-2ed4-4f1e-bd06-9cca74916956","Type":"ContainerDied","Data":"b0e21e852e20c9bf939dbb22b350e8b0cb978f75b9f23ee67bdc6d46fa04becb"} Oct 31 05:18:05.945557 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:05.945534 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-d7frf" Oct 31 05:18:05.946099 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:05.945579 4376 scope.go:122] "RemoveContainer" containerID="3ca2c47646dda985888ec2e91896aceb1af3f7d1b0c94def3e80def15de0862a" Oct 31 05:18:06.019468 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:06.019433 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-d7frf"] Oct 31 05:18:06.024662 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:06.024637 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-d7frf"] Oct 31 05:18:06.443619 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:06.443575 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dfcd34ac-2ed4-4f1e-bd06-9cca74916956" path="/var/lib/kubelet/pods/dfcd34ac-2ed4-4f1e-bd06-9cca74916956/volumes" Oct 31 05:18:07.688871 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:07.688842 4376 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-54z7t_d9f625f9-0628-476d-8235-9afc0a939cc1/limitador/0.log" Oct 31 05:18:09.252200 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:09.252167 4376 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-mjffb_c3ae653d-3b01-495e-bc64-3466a1f9f14f/manager/0.log" Oct 31 05:18:10.026053 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:10.026020 4376 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-85c4f7957c-mh7qn_6e27a6c1-bb5d-44e4-80ab-5617e8046bbf/manager/0.log" Oct 31 05:18:10.653608 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:10.653575 4376 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-zzvpx_c8c856e4-7a53-4077-8f5e-4faf46ec9d2a/proxy/0.log" Oct 31 05:18:10.999600 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:18:10.999514 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 05:18:19.881500 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:19.881461 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8d386766-6cd6694b69-wqhpj"] Oct 31 05:18:19.881897 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:19.881815 4376 cpu_manager.go:414] "containerMap: removing container" podUID="72af9c95-36f4-467b-ba78-ec84c1d1d51c" containerName="cleanup" Oct 31 05:18:19.881897 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:19.881825 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72af9c95-36f4-467b-ba78-ec84c1d1d51c" containerName="cleanup" Oct 31 05:18:19.881897 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:19.881835 4376 cpu_manager.go:414] "containerMap: removing container" podUID="80cdb0ed-7002-415c-9916-7e1650f0481f" containerName="istio-proxy" Oct 31 05:18:19.881897 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:19.881840 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80cdb0ed-7002-415c-9916-7e1650f0481f" containerName="istio-proxy" Oct 31 05:18:19.881897 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:19.881846 4376 cpu_manager.go:414] "containerMap: removing container" podUID="80cdb0ed-7002-415c-9916-7e1650f0481f" containerName="istio-proxy" Oct 31 05:18:19.881897 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:19.881851 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80cdb0ed-7002-415c-9916-7e1650f0481f" containerName="istio-proxy" Oct 31 05:18:19.881897 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:19.881865 4376 cpu_manager.go:414] "containerMap: removing container" podUID="caf8772a-4894-4924-9f18-f2b191aadad7" containerName="istio-proxy" Oct 31 05:18:19.881897 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:19.881870 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="caf8772a-4894-4924-9f18-f2b191aadad7" containerName="istio-proxy" Oct 31 05:18:19.881897 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:19.881877 4376 cpu_manager.go:414] "containerMap: removing container" podUID="994761dc-71d9-4334-a0aa-891d7287a690" containerName="maas-api" Oct 31 05:18:19.881897 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:19.881882 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="994761dc-71d9-4334-a0aa-891d7287a690" containerName="maas-api" Oct 31 05:18:19.881897 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:19.881893 4376 cpu_manager.go:414] "containerMap: removing container" podUID="07764cf3-3c73-4f06-82c6-7df4d5e9955a" containerName="payload-pre-processing" Oct 31 05:18:19.881897 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:19.881897 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07764cf3-3c73-4f06-82c6-7df4d5e9955a" containerName="payload-pre-processing" Oct 31 05:18:19.881897 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:19.881905 4376 cpu_manager.go:414] "containerMap: removing container" podUID="86377cdd-c75b-496b-8fea-1d8bcd541be8" containerName="payload-processing" Oct 31 05:18:19.882468 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:19.881911 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86377cdd-c75b-496b-8fea-1d8bcd541be8" containerName="payload-processing" Oct 31 05:18:19.882468 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:19.881921 4376 cpu_manager.go:414] "containerMap: removing container" podUID="72af9c95-36f4-467b-ba78-ec84c1d1d51c" containerName="cleanup" Oct 31 05:18:19.882468 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:19.881928 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72af9c95-36f4-467b-ba78-ec84c1d1d51c" containerName="cleanup" Oct 31 05:18:19.882468 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:19.881944 4376 cpu_manager.go:414] "containerMap: removing container" podUID="7b65451e-2626-4b42-9340-d6db06b74760" containerName="maas-api" Oct 31 05:18:19.882468 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:19.881951 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b65451e-2626-4b42-9340-d6db06b74760" containerName="maas-api" Oct 31 05:18:19.882468 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:19.881960 4376 cpu_manager.go:414] "containerMap: removing container" podUID="69d5f412-aaf9-4d2a-afb0-3f76f0861259" containerName="payload-pre-processing" Oct 31 05:18:19.882468 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:19.881968 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69d5f412-aaf9-4d2a-afb0-3f76f0861259" containerName="payload-pre-processing" Oct 31 05:18:19.882468 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:19.881981 4376 cpu_manager.go:414] "containerMap: removing container" podUID="f05ba444-d067-4e81-a4ca-cb9ef426c2a1" containerName="payload-processing" Oct 31 05:18:19.882468 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:19.881987 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f05ba444-d067-4e81-a4ca-cb9ef426c2a1" containerName="payload-processing" Oct 31 05:18:19.882468 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:19.881997 4376 cpu_manager.go:414] "containerMap: removing container" podUID="6957320f-9d86-4430-bca7-85db00d7a099" containerName="istio-proxy" Oct 31 05:18:19.882468 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:19.882004 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6957320f-9d86-4430-bca7-85db00d7a099" containerName="istio-proxy" Oct 31 05:18:19.882468 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:19.882012 4376 cpu_manager.go:414] "containerMap: removing container" podUID="8414bff7-0d22-4ccd-8c0d-e6c2dd294022" containerName="payload-pre-processing" Oct 31 05:18:19.882468 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:19.882016 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8414bff7-0d22-4ccd-8c0d-e6c2dd294022" containerName="payload-pre-processing" Oct 31 05:18:19.882468 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:19.882023 4376 cpu_manager.go:414] "containerMap: removing container" podUID="2219d6d2-75e4-4fd8-ae05-901e3ac891a1" containerName="payload-pre-processing" Oct 31 05:18:19.882468 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:19.882028 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2219d6d2-75e4-4fd8-ae05-901e3ac891a1" containerName="payload-pre-processing" Oct 31 05:18:19.882468 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:19.882036 4376 cpu_manager.go:414] "containerMap: removing container" podUID="d057ba0f-e65d-44de-949f-bc94f3c20550" containerName="istio-proxy" Oct 31 05:18:19.882468 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:19.882042 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d057ba0f-e65d-44de-949f-bc94f3c20550" containerName="istio-proxy" Oct 31 05:18:19.882468 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:19.882051 4376 cpu_manager.go:414] "containerMap: removing container" podUID="dfcd34ac-2ed4-4f1e-bd06-9cca74916956" containerName="istio-proxy" Oct 31 05:18:19.882468 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:19.882060 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfcd34ac-2ed4-4f1e-bd06-9cca74916956" containerName="istio-proxy" Oct 31 05:18:19.882468 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:19.882072 4376 cpu_manager.go:414] "containerMap: removing container" podUID="caf8772a-4894-4924-9f18-f2b191aadad7" containerName="istio-proxy" Oct 31 05:18:19.882468 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:19.882080 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="caf8772a-4894-4924-9f18-f2b191aadad7" containerName="istio-proxy" Oct 31 05:18:19.882468 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:19.882090 4376 cpu_manager.go:414] "containerMap: removing container" podUID="775b25ed-d591-4102-98b8-59ce485f9f52" containerName="maas-api" Oct 31 05:18:19.882468 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:19.882097 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="775b25ed-d591-4102-98b8-59ce485f9f52" containerName="maas-api" Oct 31 05:18:19.882468 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:19.882110 4376 cpu_manager.go:414] "containerMap: removing container" podUID="b9faf32d-45c1-4167-bb23-663987f3e602" containerName="payload-pre-processing" Oct 31 05:18:19.882468 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:19.882118 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9faf32d-45c1-4167-bb23-663987f3e602" containerName="payload-pre-processing" Oct 31 05:18:19.882468 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:19.882139 4376 cpu_manager.go:414] "containerMap: removing container" podUID="d2870b2c-9c85-43d1-a726-38edfcf6ae2b" containerName="payload-processing" Oct 31 05:18:19.882468 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:19.882146 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2870b2c-9c85-43d1-a726-38edfcf6ae2b" containerName="payload-processing" Oct 31 05:18:19.882468 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:19.882156 4376 cpu_manager.go:414] "containerMap: removing container" podUID="b2c75691-4cf4-484b-96e2-3c2503b8d78c" containerName="main" Oct 31 05:18:19.882468 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:19.882162 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2c75691-4cf4-484b-96e2-3c2503b8d78c" containerName="main" Oct 31 05:18:19.882468 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:19.882169 4376 cpu_manager.go:414] "containerMap: removing container" podUID="8e0104d4-9c91-4dc6-93ae-a7844c9d89c8" containerName="payload-processing" Oct 31 05:18:19.882468 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:19.882176 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e0104d4-9c91-4dc6-93ae-a7844c9d89c8" containerName="payload-processing" Oct 31 05:18:19.882468 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:19.882186 4376 cpu_manager.go:414] "containerMap: removing container" podUID="e90146f8-ed41-49b8-bf8b-c92bee5f8cdd" containerName="payload-processing" Oct 31 05:18:19.882468 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:19.882193 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e90146f8-ed41-49b8-bf8b-c92bee5f8cdd" containerName="payload-processing" Oct 31 05:18:19.882468 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:19.882204 4376 cpu_manager.go:414] "containerMap: removing container" podUID="a6c79800-45c4-4778-91ef-027ceba7bb8a" containerName="istio-proxy" Oct 31 05:18:19.882468 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:19.882211 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6c79800-45c4-4778-91ef-027ceba7bb8a" containerName="istio-proxy" Oct 31 05:18:19.882468 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:19.882222 4376 cpu_manager.go:414] "containerMap: removing container" podUID="b3e4def5-df8d-42d2-810c-8fa6764d338b" containerName="maas-api" Oct 31 05:18:19.882468 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:19.882229 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3e4def5-df8d-42d2-810c-8fa6764d338b" containerName="maas-api" Oct 31 05:18:19.882468 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:19.882239 4376 cpu_manager.go:414] "containerMap: removing container" podUID="6957320f-9d86-4430-bca7-85db00d7a099" containerName="istio-proxy" Oct 31 05:18:19.882468 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:19.882246 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6957320f-9d86-4430-bca7-85db00d7a099" containerName="istio-proxy" Oct 31 05:18:19.882468 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:19.882253 4376 cpu_manager.go:414] "containerMap: removing container" podUID="43d89e12-3149-41df-825f-40801de7576a" containerName="payload-processing" Oct 31 05:18:19.882468 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:19.882258 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43d89e12-3149-41df-825f-40801de7576a" containerName="payload-processing" Oct 31 05:18:19.882468 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:19.882264 4376 cpu_manager.go:414] "containerMap: removing container" podUID="2a9e4834-09d5-4add-9004-04c30da0527c" containerName="payload-processing" Oct 31 05:18:19.882468 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:19.882269 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a9e4834-09d5-4add-9004-04c30da0527c" containerName="payload-processing" Oct 31 05:18:19.882468 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:19.882280 4376 cpu_manager.go:414] "containerMap: removing container" podUID="dc53083d-eadd-4cb8-8633-35e02d400b9c" containerName="main" Oct 31 05:18:19.882468 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:19.882287 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc53083d-eadd-4cb8-8633-35e02d400b9c" containerName="main" Oct 31 05:18:19.882468 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:19.882298 4376 cpu_manager.go:414] "containerMap: removing container" podUID="7cffa2e2-93ef-4f2b-a512-107f5d14b698" containerName="maas-api" Oct 31 05:18:19.882468 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:19.882303 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7cffa2e2-93ef-4f2b-a512-107f5d14b698" containerName="maas-api" Oct 31 05:18:19.882468 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:19.882310 4376 cpu_manager.go:414] "containerMap: removing container" podUID="576e2c1f-1c80-43a9-a136-3692776642ae" containerName="main" Oct 31 05:18:19.882468 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:19.882314 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="576e2c1f-1c80-43a9-a136-3692776642ae" containerName="main" Oct 31 05:18:19.882468 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:19.882320 4376 cpu_manager.go:414] "containerMap: removing container" podUID="84d95c12-4dca-44d2-a2cb-fdbfe919135b" containerName="payload-pre-processing" Oct 31 05:18:19.882468 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:19.882350 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84d95c12-4dca-44d2-a2cb-fdbfe919135b" containerName="payload-pre-processing" Oct 31 05:18:19.882468 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:19.882364 4376 cpu_manager.go:414] "containerMap: removing container" podUID="ba6f1fd8-1985-4ea5-9097-36263d48aa13" containerName="maas-api" Oct 31 05:18:19.882468 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:19.882369 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba6f1fd8-1985-4ea5-9097-36263d48aa13" containerName="maas-api" Oct 31 05:18:19.882468 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:19.882381 4376 cpu_manager.go:414] "containerMap: removing container" podUID="a4beb0fb-a6b6-4864-a65a-38e9e3a9c05f" containerName="payload-processing" Oct 31 05:18:19.882468 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:19.882386 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4beb0fb-a6b6-4864-a65a-38e9e3a9c05f" containerName="payload-processing" Oct 31 05:18:19.882468 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:19.882470 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="72af9c95-36f4-467b-ba78-ec84c1d1d51c" containerName="cleanup" Oct 31 05:18:19.882468 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:19.882479 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="80cdb0ed-7002-415c-9916-7e1650f0481f" containerName="istio-proxy" Oct 31 05:18:19.882468 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:19.882492 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="80cdb0ed-7002-415c-9916-7e1650f0481f" containerName="istio-proxy" Oct 31 05:18:19.884393 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:19.882502 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="caf8772a-4894-4924-9f18-f2b191aadad7" containerName="istio-proxy" Oct 31 05:18:19.884393 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:19.882513 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="994761dc-71d9-4334-a0aa-891d7287a690" containerName="maas-api" Oct 31 05:18:19.884393 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:19.882522 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="07764cf3-3c73-4f06-82c6-7df4d5e9955a" containerName="payload-pre-processing" Oct 31 05:18:19.884393 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:19.882530 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="86377cdd-c75b-496b-8fea-1d8bcd541be8" containerName="payload-processing" Oct 31 05:18:19.884393 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:19.882538 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="72af9c95-36f4-467b-ba78-ec84c1d1d51c" containerName="cleanup" Oct 31 05:18:19.884393 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:19.882545 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b65451e-2626-4b42-9340-d6db06b74760" containerName="maas-api" Oct 31 05:18:19.884393 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:19.882555 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="69d5f412-aaf9-4d2a-afb0-3f76f0861259" containerName="payload-pre-processing" Oct 31 05:18:19.884393 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:19.882562 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="f05ba444-d067-4e81-a4ca-cb9ef426c2a1" containerName="payload-processing" Oct 31 05:18:19.884393 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:19.882567 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="6957320f-9d86-4430-bca7-85db00d7a099" containerName="istio-proxy" Oct 31 05:18:19.884393 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:19.882575 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="8414bff7-0d22-4ccd-8c0d-e6c2dd294022" containerName="payload-pre-processing" Oct 31 05:18:19.884393 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:19.882582 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="2219d6d2-75e4-4fd8-ae05-901e3ac891a1" containerName="payload-pre-processing" Oct 31 05:18:19.884393 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:19.882590 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="d057ba0f-e65d-44de-949f-bc94f3c20550" containerName="istio-proxy" Oct 31 05:18:19.884393 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:19.882599 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfcd34ac-2ed4-4f1e-bd06-9cca74916956" containerName="istio-proxy" Oct 31 05:18:19.884393 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:19.882607 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="caf8772a-4894-4924-9f18-f2b191aadad7" containerName="istio-proxy" Oct 31 05:18:19.884393 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:19.882615 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="775b25ed-d591-4102-98b8-59ce485f9f52" containerName="maas-api" Oct 31 05:18:19.884393 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:19.882621 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9faf32d-45c1-4167-bb23-663987f3e602" containerName="payload-pre-processing" Oct 31 05:18:19.884393 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:19.882631 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2870b2c-9c85-43d1-a726-38edfcf6ae2b" containerName="payload-processing" Oct 31 05:18:19.884393 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:19.882639 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2c75691-4cf4-484b-96e2-3c2503b8d78c" containerName="main" Oct 31 05:18:19.884393 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:19.882645 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e0104d4-9c91-4dc6-93ae-a7844c9d89c8" containerName="payload-processing" Oct 31 05:18:19.884393 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:19.882651 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="e90146f8-ed41-49b8-bf8b-c92bee5f8cdd" containerName="payload-processing" Oct 31 05:18:19.884393 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:19.882660 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3e4def5-df8d-42d2-810c-8fa6764d338b" containerName="maas-api" Oct 31 05:18:19.884393 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:19.882668 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="6957320f-9d86-4430-bca7-85db00d7a099" containerName="istio-proxy" Oct 31 05:18:19.884393 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:19.882676 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="43d89e12-3149-41df-825f-40801de7576a" containerName="payload-processing" Oct 31 05:18:19.884393 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:19.882685 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a9e4834-09d5-4add-9004-04c30da0527c" containerName="payload-processing" Oct 31 05:18:19.884393 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:19.882692 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc53083d-eadd-4cb8-8633-35e02d400b9c" containerName="main" Oct 31 05:18:19.884393 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:19.882700 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="7cffa2e2-93ef-4f2b-a512-107f5d14b698" containerName="maas-api" Oct 31 05:18:19.884393 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:19.882704 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="576e2c1f-1c80-43a9-a136-3692776642ae" containerName="main" Oct 31 05:18:19.884393 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:19.882709 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="84d95c12-4dca-44d2-a2cb-fdbfe919135b" containerName="payload-pre-processing" Oct 31 05:18:19.884393 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:19.882715 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba6f1fd8-1985-4ea5-9097-36263d48aa13" containerName="maas-api" Oct 31 05:18:19.884393 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:19.882720 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4beb0fb-a6b6-4864-a65a-38e9e3a9c05f" containerName="payload-processing" Oct 31 05:18:19.887596 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:19.887578 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8d386766-6cd6694b69-wqhpj" Oct 31 05:18:19.906247 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:19.906219 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8d386766-6cd6694b69-wqhpj"] Oct 31 05:18:19.919039 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:19.919008 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-8d386766-serving-cert\"" Oct 31 05:18:19.963191 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:19.963153 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01ec4d78-9bd3-4762-a7c6-e277a212f2a6-openshift-service-ca\") pod \"maas-api-e2e-ait-8d386766-6cd6694b69-wqhpj\" (UID: \"01ec4d78-9bd3-4762-a7c6-e277a212f2a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8d386766-6cd6694b69-wqhpj" Oct 31 05:18:19.963421 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:19.963299 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01ec4d78-9bd3-4762-a7c6-e277a212f2a6-maas-api-tls\") pod \"maas-api-e2e-ait-8d386766-6cd6694b69-wqhpj\" (UID: \"01ec4d78-9bd3-4762-a7c6-e277a212f2a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8d386766-6cd6694b69-wqhpj" Oct 31 05:18:19.963421 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:19.963351 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/01ec4d78-9bd3-4762-a7c6-e277a212f2a6-metrics-certs\") pod \"maas-api-e2e-ait-8d386766-6cd6694b69-wqhpj\" (UID: \"01ec4d78-9bd3-4762-a7c6-e277a212f2a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8d386766-6cd6694b69-wqhpj" Oct 31 05:18:19.963550 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:19.963426 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtpks\" (UniqueName: \"kubernetes.io/projected/01ec4d78-9bd3-4762-a7c6-e277a212f2a6-kube-api-access-mtpks\") pod \"maas-api-e2e-ait-8d386766-6cd6694b69-wqhpj\" (UID: \"01ec4d78-9bd3-4762-a7c6-e277a212f2a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8d386766-6cd6694b69-wqhpj" Oct 31 05:18:20.064712 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:20.064677 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01ec4d78-9bd3-4762-a7c6-e277a212f2a6-openshift-service-ca\") pod \"maas-api-e2e-ait-8d386766-6cd6694b69-wqhpj\" (UID: \"01ec4d78-9bd3-4762-a7c6-e277a212f2a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8d386766-6cd6694b69-wqhpj" Oct 31 05:18:20.064915 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:20.064768 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01ec4d78-9bd3-4762-a7c6-e277a212f2a6-maas-api-tls\") pod \"maas-api-e2e-ait-8d386766-6cd6694b69-wqhpj\" (UID: \"01ec4d78-9bd3-4762-a7c6-e277a212f2a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8d386766-6cd6694b69-wqhpj" Oct 31 05:18:20.064915 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:20.064800 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/01ec4d78-9bd3-4762-a7c6-e277a212f2a6-metrics-certs\") pod \"maas-api-e2e-ait-8d386766-6cd6694b69-wqhpj\" (UID: \"01ec4d78-9bd3-4762-a7c6-e277a212f2a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8d386766-6cd6694b69-wqhpj" Oct 31 05:18:20.064915 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:20.064826 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtpks\" (UniqueName: \"kubernetes.io/projected/01ec4d78-9bd3-4762-a7c6-e277a212f2a6-kube-api-access-mtpks\") pod \"maas-api-e2e-ait-8d386766-6cd6694b69-wqhpj\" (UID: \"01ec4d78-9bd3-4762-a7c6-e277a212f2a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8d386766-6cd6694b69-wqhpj" Oct 31 05:18:20.065746 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:20.065716 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01ec4d78-9bd3-4762-a7c6-e277a212f2a6-openshift-service-ca\") pod \"maas-api-e2e-ait-8d386766-6cd6694b69-wqhpj\" (UID: \"01ec4d78-9bd3-4762-a7c6-e277a212f2a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8d386766-6cd6694b69-wqhpj" Oct 31 05:18:20.067412 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:20.067389 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/01ec4d78-9bd3-4762-a7c6-e277a212f2a6-metrics-certs\") pod \"maas-api-e2e-ait-8d386766-6cd6694b69-wqhpj\" (UID: \"01ec4d78-9bd3-4762-a7c6-e277a212f2a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8d386766-6cd6694b69-wqhpj" Oct 31 05:18:20.067610 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:20.067587 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01ec4d78-9bd3-4762-a7c6-e277a212f2a6-maas-api-tls\") pod \"maas-api-e2e-ait-8d386766-6cd6694b69-wqhpj\" (UID: \"01ec4d78-9bd3-4762-a7c6-e277a212f2a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8d386766-6cd6694b69-wqhpj" Oct 31 05:18:20.094508 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:20.094463 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtpks\" (UniqueName: \"kubernetes.io/projected/01ec4d78-9bd3-4762-a7c6-e277a212f2a6-kube-api-access-mtpks\") pod \"maas-api-e2e-ait-8d386766-6cd6694b69-wqhpj\" (UID: \"01ec4d78-9bd3-4762-a7c6-e277a212f2a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8d386766-6cd6694b69-wqhpj" Oct 31 05:18:20.202513 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:20.202397 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8d386766-6cd6694b69-wqhpj" Oct 31 05:18:20.522708 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:20.522677 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8d386766-6cd6694b69-wqhpj"] Oct 31 05:18:20.522821 ip-10-0-134-155 kubenswrapper[4376]: W1031 05:18:20.522706 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01ec4d78_9bd3_4762_a7c6_e277a212f2a6.slice/crio-861a198d461e2a5c9cce2795a65c4f0665a07ebdea5ce67304d946776d812149 WatchSource:0}: Error finding container 861a198d461e2a5c9cce2795a65c4f0665a07ebdea5ce67304d946776d812149: Status 404 returned error can't find the container with id 861a198d461e2a5c9cce2795a65c4f0665a07ebdea5ce67304d946776d812149 Oct 31 05:18:20.524019 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:20.524004 4376 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 05:18:21.019686 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:21.019649 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8d386766-6cd6694b69-wqhpj" event={"ID":"01ec4d78-9bd3-4762-a7c6-e277a212f2a6","Type":"ContainerStarted","Data":"6ef654fd65002da428bcb42cc002e4b867875c4b8a6d6dfeb3d3038e4bee0206"} Oct 31 05:18:21.020109 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:21.019712 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8d386766-6cd6694b69-wqhpj" event={"ID":"01ec4d78-9bd3-4762-a7c6-e277a212f2a6","Type":"ContainerStarted","Data":"861a198d461e2a5c9cce2795a65c4f0665a07ebdea5ce67304d946776d812149"} Oct 31 05:18:21.020109 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:21.019734 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8d386766-6cd6694b69-wqhpj" Oct 31 05:18:21.066847 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:21.066789 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8d386766-6cd6694b69-wqhpj" podStartSLOduration=1.6667724640000001 podStartE2EDuration="2.06677667s" totalImagesPullingTime="400.004206ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 05:18:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 05:18:21.065754609 +0000 UTC m=+4448.780089375" watchObservedRunningTime="2026-10-31 05:18:21.06677667 +0000 UTC m=+4448.781111288" Oct 31 05:18:23.845822 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:23.845784 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 05:18:27.029222 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:27.029142 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8d386766-6cd6694b69-wqhpj" Oct 31 05:18:40.997716 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:18:40.997674 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 05:18:45.129413 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:45.129374 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8d386766-6cd6694b69-wqhpj"] Oct 31 05:18:45.129881 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:45.129599 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8d386766-6cd6694b69-wqhpj" podUID="01ec4d78-9bd3-4762-a7c6-e277a212f2a6" containerName="maas-api" containerID="cri-o://6ef654fd65002da428bcb42cc002e4b867875c4b8a6d6dfeb3d3038e4bee0206" gracePeriod=30 Oct 31 05:18:45.383226 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:45.383134 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8d386766-6cd6694b69-wqhpj" Oct 31 05:18:45.443591 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:45.443559 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01ec4d78-9bd3-4762-a7c6-e277a212f2a6-kube-api-access-mtpks\" (UniqueName: \"kubernetes.io/projected/01ec4d78-9bd3-4762-a7c6-e277a212f2a6-kube-api-access-mtpks\") pod \"01ec4d78-9bd3-4762-a7c6-e277a212f2a6\" (UID: \"01ec4d78-9bd3-4762-a7c6-e277a212f2a6\") " Oct 31 05:18:45.443770 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:45.443600 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01ec4d78-9bd3-4762-a7c6-e277a212f2a6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01ec4d78-9bd3-4762-a7c6-e277a212f2a6-openshift-service-ca\") pod \"01ec4d78-9bd3-4762-a7c6-e277a212f2a6\" (UID: \"01ec4d78-9bd3-4762-a7c6-e277a212f2a6\") " Oct 31 05:18:45.443770 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:45.443644 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/01ec4d78-9bd3-4762-a7c6-e277a212f2a6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/01ec4d78-9bd3-4762-a7c6-e277a212f2a6-metrics-certs\") pod \"01ec4d78-9bd3-4762-a7c6-e277a212f2a6\" (UID: \"01ec4d78-9bd3-4762-a7c6-e277a212f2a6\") " Oct 31 05:18:45.443861 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:45.443765 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/01ec4d78-9bd3-4762-a7c6-e277a212f2a6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01ec4d78-9bd3-4762-a7c6-e277a212f2a6-maas-api-tls\") pod \"01ec4d78-9bd3-4762-a7c6-e277a212f2a6\" (UID: \"01ec4d78-9bd3-4762-a7c6-e277a212f2a6\") " Oct 31 05:18:45.444061 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:45.444035 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01ec4d78-9bd3-4762-a7c6-e277a212f2a6-openshift-service-ca" pod "01ec4d78-9bd3-4762-a7c6-e277a212f2a6" (UID: "01ec4d78-9bd3-4762-a7c6-e277a212f2a6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:18:45.445801 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:45.445769 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01ec4d78-9bd3-4762-a7c6-e277a212f2a6-metrics-certs" pod "01ec4d78-9bd3-4762-a7c6-e277a212f2a6" (UID: "01ec4d78-9bd3-4762-a7c6-e277a212f2a6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 05:18:45.445915 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:45.445813 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01ec4d78-9bd3-4762-a7c6-e277a212f2a6-kube-api-access-mtpks" pod "01ec4d78-9bd3-4762-a7c6-e277a212f2a6" (UID: "01ec4d78-9bd3-4762-a7c6-e277a212f2a6"). InnerVolumeSpecName "kube-api-access-mtpks". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:18:45.445957 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:45.445910 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01ec4d78-9bd3-4762-a7c6-e277a212f2a6-maas-api-tls" pod "01ec4d78-9bd3-4762-a7c6-e277a212f2a6" (UID: "01ec4d78-9bd3-4762-a7c6-e277a212f2a6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 05:18:45.545719 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:45.545683 4376 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01ec4d78-9bd3-4762-a7c6-e277a212f2a6-maas-api-tls\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:18:45.545719 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:45.545721 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mtpks\" (UniqueName: \"kubernetes.io/projected/01ec4d78-9bd3-4762-a7c6-e277a212f2a6-kube-api-access-mtpks\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:18:45.545960 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:45.545736 4376 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01ec4d78-9bd3-4762-a7c6-e277a212f2a6-openshift-service-ca\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:18:45.545960 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:45.545750 4376 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/01ec4d78-9bd3-4762-a7c6-e277a212f2a6-metrics-certs\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:18:46.131169 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:46.131135 4376 generic.go:356] "Generic (PLEG): container finished" podID="01ec4d78-9bd3-4762-a7c6-e277a212f2a6" containerID="6ef654fd65002da428bcb42cc002e4b867875c4b8a6d6dfeb3d3038e4bee0206" exitCode=0 Oct 31 05:18:46.131718 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:46.131189 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8d386766-6cd6694b69-wqhpj" event={"ID":"01ec4d78-9bd3-4762-a7c6-e277a212f2a6","Type":"ContainerDied","Data":"6ef654fd65002da428bcb42cc002e4b867875c4b8a6d6dfeb3d3038e4bee0206"} Oct 31 05:18:46.131718 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:46.131229 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8d386766-6cd6694b69-wqhpj" event={"ID":"01ec4d78-9bd3-4762-a7c6-e277a212f2a6","Type":"ContainerDied","Data":"861a198d461e2a5c9cce2795a65c4f0665a07ebdea5ce67304d946776d812149"} Oct 31 05:18:46.131718 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:46.131252 4376 scope.go:122] "RemoveContainer" containerID="6ef654fd65002da428bcb42cc002e4b867875c4b8a6d6dfeb3d3038e4bee0206" Oct 31 05:18:46.131718 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:46.131190 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8d386766-6cd6694b69-wqhpj" Oct 31 05:18:46.140438 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:46.140422 4376 scope.go:122] "RemoveContainer" containerID="6ef654fd65002da428bcb42cc002e4b867875c4b8a6d6dfeb3d3038e4bee0206" Oct 31 05:18:46.140730 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:18:46.140712 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ef654fd65002da428bcb42cc002e4b867875c4b8a6d6dfeb3d3038e4bee0206\": container with ID starting with 6ef654fd65002da428bcb42cc002e4b867875c4b8a6d6dfeb3d3038e4bee0206 not found: ID does not exist" containerID="6ef654fd65002da428bcb42cc002e4b867875c4b8a6d6dfeb3d3038e4bee0206" Oct 31 05:18:46.140806 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:46.140739 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ef654fd65002da428bcb42cc002e4b867875c4b8a6d6dfeb3d3038e4bee0206"} err="failed to get container status \"6ef654fd65002da428bcb42cc002e4b867875c4b8a6d6dfeb3d3038e4bee0206\": rpc error: code = NotFound desc = could not find container \"6ef654fd65002da428bcb42cc002e4b867875c4b8a6d6dfeb3d3038e4bee0206\": container with ID starting with 6ef654fd65002da428bcb42cc002e4b867875c4b8a6d6dfeb3d3038e4bee0206 not found: ID does not exist" Oct 31 05:18:46.187608 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:46.187569 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8d386766-6cd6694b69-wqhpj"] Oct 31 05:18:46.201754 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:46.201726 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8d386766-6cd6694b69-wqhpj"] Oct 31 05:18:46.441163 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:46.441081 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01ec4d78-9bd3-4762-a7c6-e277a212f2a6" path="/var/lib/kubelet/pods/01ec4d78-9bd3-4762-a7c6-e277a212f2a6/volumes" Oct 31 05:18:47.441639 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:47.441605 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 05:18:54.500358 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:54.500310 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-85c4f7957c-mh7qn"] Oct 31 05:18:54.500929 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:54.500532 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-85c4f7957c-mh7qn" podUID="6e27a6c1-bb5d-44e4-80ab-5617e8046bbf" containerName="manager" containerID="cri-o://55a11a9eb2adedcb1f5c82fa468906d6931229faae3b1a459950a1898a152131" gracePeriod=30 Oct 31 05:18:54.756790 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:54.756725 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-85c4f7957c-mh7qn" Oct 31 05:18:54.835852 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:54.835818 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e27a6c1-bb5d-44e4-80ab-5617e8046bbf-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e27a6c1-bb5d-44e4-80ab-5617e8046bbf-metrics-certs\") pod \"6e27a6c1-bb5d-44e4-80ab-5617e8046bbf\" (UID: \"6e27a6c1-bb5d-44e4-80ab-5617e8046bbf\") " Oct 31 05:18:54.836058 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:54.835888 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e27a6c1-bb5d-44e4-80ab-5617e8046bbf-kube-api-access-gh8bz\" (UniqueName: \"kubernetes.io/projected/6e27a6c1-bb5d-44e4-80ab-5617e8046bbf-kube-api-access-gh8bz\") pod \"6e27a6c1-bb5d-44e4-80ab-5617e8046bbf\" (UID: \"6e27a6c1-bb5d-44e4-80ab-5617e8046bbf\") " Oct 31 05:18:54.836058 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:54.835924 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e27a6c1-bb5d-44e4-80ab-5617e8046bbf-webhook-cert\" (UniqueName: \"kubernetes.io/secret/6e27a6c1-bb5d-44e4-80ab-5617e8046bbf-webhook-cert\") pod \"6e27a6c1-bb5d-44e4-80ab-5617e8046bbf\" (UID: \"6e27a6c1-bb5d-44e4-80ab-5617e8046bbf\") " Oct 31 05:18:54.838579 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:54.838532 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e27a6c1-bb5d-44e4-80ab-5617e8046bbf-metrics-certs" pod "6e27a6c1-bb5d-44e4-80ab-5617e8046bbf" (UID: "6e27a6c1-bb5d-44e4-80ab-5617e8046bbf"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 05:18:54.838579 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:54.838549 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e27a6c1-bb5d-44e4-80ab-5617e8046bbf-webhook-cert" pod "6e27a6c1-bb5d-44e4-80ab-5617e8046bbf" (UID: "6e27a6c1-bb5d-44e4-80ab-5617e8046bbf"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 05:18:54.838785 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:54.838677 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e27a6c1-bb5d-44e4-80ab-5617e8046bbf-kube-api-access-gh8bz" pod "6e27a6c1-bb5d-44e4-80ab-5617e8046bbf" (UID: "6e27a6c1-bb5d-44e4-80ab-5617e8046bbf"). InnerVolumeSpecName "kube-api-access-gh8bz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:18:54.937082 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:54.937045 4376 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e27a6c1-bb5d-44e4-80ab-5617e8046bbf-metrics-certs\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:18:54.937082 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:54.937082 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gh8bz\" (UniqueName: \"kubernetes.io/projected/6e27a6c1-bb5d-44e4-80ab-5617e8046bbf-kube-api-access-gh8bz\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:18:54.937082 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:54.937095 4376 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6e27a6c1-bb5d-44e4-80ab-5617e8046bbf-webhook-cert\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:18:55.175693 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:55.175660 4376 generic.go:356] "Generic (PLEG): container finished" podID="6e27a6c1-bb5d-44e4-80ab-5617e8046bbf" containerID="55a11a9eb2adedcb1f5c82fa468906d6931229faae3b1a459950a1898a152131" exitCode=0 Oct 31 05:18:55.175925 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:55.175722 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-85c4f7957c-mh7qn" Oct 31 05:18:55.175925 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:55.175761 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-85c4f7957c-mh7qn" event={"ID":"6e27a6c1-bb5d-44e4-80ab-5617e8046bbf","Type":"ContainerDied","Data":"55a11a9eb2adedcb1f5c82fa468906d6931229faae3b1a459950a1898a152131"} Oct 31 05:18:55.175925 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:55.175837 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-85c4f7957c-mh7qn" event={"ID":"6e27a6c1-bb5d-44e4-80ab-5617e8046bbf","Type":"ContainerDied","Data":"d9d86c3f840a5497963162d48fe6e163957d2ee9e3f6af3ba9e65bf717bcc439"} Oct 31 05:18:55.175925 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:55.175873 4376 scope.go:122] "RemoveContainer" containerID="55a11a9eb2adedcb1f5c82fa468906d6931229faae3b1a459950a1898a152131" Oct 31 05:18:55.187432 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:55.187406 4376 scope.go:122] "RemoveContainer" containerID="55a11a9eb2adedcb1f5c82fa468906d6931229faae3b1a459950a1898a152131" Oct 31 05:18:55.194202 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:18:55.194165 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55a11a9eb2adedcb1f5c82fa468906d6931229faae3b1a459950a1898a152131\": container with ID starting with 55a11a9eb2adedcb1f5c82fa468906d6931229faae3b1a459950a1898a152131 not found: ID does not exist" containerID="55a11a9eb2adedcb1f5c82fa468906d6931229faae3b1a459950a1898a152131" Oct 31 05:18:55.196944 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:55.196899 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55a11a9eb2adedcb1f5c82fa468906d6931229faae3b1a459950a1898a152131"} err="failed to get container status \"55a11a9eb2adedcb1f5c82fa468906d6931229faae3b1a459950a1898a152131\": rpc error: code = NotFound desc = could not find container \"55a11a9eb2adedcb1f5c82fa468906d6931229faae3b1a459950a1898a152131\": container with ID starting with 55a11a9eb2adedcb1f5c82fa468906d6931229faae3b1a459950a1898a152131 not found: ID does not exist" Oct 31 05:18:55.246474 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:55.246424 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-85c4f7957c-mh7qn"] Oct 31 05:18:55.260082 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:55.260036 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-85c4f7957c-mh7qn"] Oct 31 05:18:56.474022 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:18:56.473983 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e27a6c1-bb5d-44e4-80ab-5617e8046bbf" path="/var/lib/kubelet/pods/6e27a6c1-bb5d-44e4-80ab-5617e8046bbf/volumes" Oct 31 05:19:04.899732 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:19:04.899694 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-n5dnl"] Oct 31 05:19:04.900246 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:19:04.899894 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-n5dnl" podUID="577f4375-87fc-4850-a066-7f10cf8307b6" containerName="istio-proxy" containerID="cri-o://d3898aa45660ba8b16051cfca62058251c0de44e0fe8704e85516ef92a67b26a" gracePeriod=30 Oct 31 05:19:10.150801 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:19:10.150774 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-n5dnl" Oct 31 05:19:10.210561 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:19:10.210476 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/577f4375-87fc-4850-a066-7f10cf8307b6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/577f4375-87fc-4850-a066-7f10cf8307b6-workload-socket\") pod \"577f4375-87fc-4850-a066-7f10cf8307b6\" (UID: \"577f4375-87fc-4850-a066-7f10cf8307b6\") " Oct 31 05:19:10.210561 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:19:10.210546 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/577f4375-87fc-4850-a066-7f10cf8307b6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/577f4375-87fc-4850-a066-7f10cf8307b6-istio-data\") pod \"577f4375-87fc-4850-a066-7f10cf8307b6\" (UID: \"577f4375-87fc-4850-a066-7f10cf8307b6\") " Oct 31 05:19:10.210761 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:19:10.210580 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/577f4375-87fc-4850-a066-7f10cf8307b6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/577f4375-87fc-4850-a066-7f10cf8307b6-istio-envoy\") pod \"577f4375-87fc-4850-a066-7f10cf8307b6\" (UID: \"577f4375-87fc-4850-a066-7f10cf8307b6\") " Oct 31 05:19:10.210761 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:19:10.210606 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/577f4375-87fc-4850-a066-7f10cf8307b6-kube-api-access-8dczl\" (UniqueName: \"kubernetes.io/projected/577f4375-87fc-4850-a066-7f10cf8307b6-kube-api-access-8dczl\") pod \"577f4375-87fc-4850-a066-7f10cf8307b6\" (UID: \"577f4375-87fc-4850-a066-7f10cf8307b6\") " Oct 31 05:19:10.210761 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:19:10.210662 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/577f4375-87fc-4850-a066-7f10cf8307b6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/577f4375-87fc-4850-a066-7f10cf8307b6-istio-podinfo\") pod \"577f4375-87fc-4850-a066-7f10cf8307b6\" (UID: \"577f4375-87fc-4850-a066-7f10cf8307b6\") " Oct 31 05:19:10.210761 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:19:10.210685 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/577f4375-87fc-4850-a066-7f10cf8307b6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/577f4375-87fc-4850-a066-7f10cf8307b6-workload-certs\") pod \"577f4375-87fc-4850-a066-7f10cf8307b6\" (UID: \"577f4375-87fc-4850-a066-7f10cf8307b6\") " Oct 31 05:19:10.210991 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:19:10.210783 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/577f4375-87fc-4850-a066-7f10cf8307b6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/577f4375-87fc-4850-a066-7f10cf8307b6-istiod-ca-cert\") pod \"577f4375-87fc-4850-a066-7f10cf8307b6\" (UID: \"577f4375-87fc-4850-a066-7f10cf8307b6\") " Oct 31 05:19:10.210991 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:19:10.210810 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/577f4375-87fc-4850-a066-7f10cf8307b6-workload-socket" pod "577f4375-87fc-4850-a066-7f10cf8307b6" (UID: "577f4375-87fc-4850-a066-7f10cf8307b6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:19:10.210991 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:19:10.210823 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/577f4375-87fc-4850-a066-7f10cf8307b6-istio-data" pod "577f4375-87fc-4850-a066-7f10cf8307b6" (UID: "577f4375-87fc-4850-a066-7f10cf8307b6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:19:10.210991 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:19:10.210832 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/577f4375-87fc-4850-a066-7f10cf8307b6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/577f4375-87fc-4850-a066-7f10cf8307b6-credential-socket\") pod \"577f4375-87fc-4850-a066-7f10cf8307b6\" (UID: \"577f4375-87fc-4850-a066-7f10cf8307b6\") " Oct 31 05:19:10.210991 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:19:10.210946 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/577f4375-87fc-4850-a066-7f10cf8307b6-istio-token\" (UniqueName: \"kubernetes.io/projected/577f4375-87fc-4850-a066-7f10cf8307b6-istio-token\") pod \"577f4375-87fc-4850-a066-7f10cf8307b6\" (UID: \"577f4375-87fc-4850-a066-7f10cf8307b6\") " Oct 31 05:19:10.210991 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:19:10.210962 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/577f4375-87fc-4850-a066-7f10cf8307b6-workload-certs" pod "577f4375-87fc-4850-a066-7f10cf8307b6" (UID: "577f4375-87fc-4850-a066-7f10cf8307b6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:19:10.211383 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:19:10.211035 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/577f4375-87fc-4850-a066-7f10cf8307b6-credential-socket" pod "577f4375-87fc-4850-a066-7f10cf8307b6" (UID: "577f4375-87fc-4850-a066-7f10cf8307b6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:19:10.211383 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:19:10.211158 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/577f4375-87fc-4850-a066-7f10cf8307b6-istiod-ca-cert" pod "577f4375-87fc-4850-a066-7f10cf8307b6" (UID: "577f4375-87fc-4850-a066-7f10cf8307b6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:19:10.211383 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:19:10.211311 4376 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/577f4375-87fc-4850-a066-7f10cf8307b6-workload-certs\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:19:10.211383 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:19:10.211355 4376 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/577f4375-87fc-4850-a066-7f10cf8307b6-istiod-ca-cert\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:19:10.211383 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:19:10.211369 4376 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/577f4375-87fc-4850-a066-7f10cf8307b6-credential-socket\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:19:10.211383 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:19:10.211382 4376 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/577f4375-87fc-4850-a066-7f10cf8307b6-workload-socket\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:19:10.211738 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:19:10.211395 4376 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/577f4375-87fc-4850-a066-7f10cf8307b6-istio-data\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:19:10.213234 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:19:10.213202 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/577f4375-87fc-4850-a066-7f10cf8307b6-istio-podinfo" pod "577f4375-87fc-4850-a066-7f10cf8307b6" (UID: "577f4375-87fc-4850-a066-7f10cf8307b6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 05:19:10.213385 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:19:10.213292 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/577f4375-87fc-4850-a066-7f10cf8307b6-kube-api-access-8dczl" pod "577f4375-87fc-4850-a066-7f10cf8307b6" (UID: "577f4375-87fc-4850-a066-7f10cf8307b6"). InnerVolumeSpecName "kube-api-access-8dczl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:19:10.213385 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:19:10.213309 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/577f4375-87fc-4850-a066-7f10cf8307b6-istio-token" pod "577f4375-87fc-4850-a066-7f10cf8307b6" (UID: "577f4375-87fc-4850-a066-7f10cf8307b6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:19:10.213732 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:19:10.213706 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/577f4375-87fc-4850-a066-7f10cf8307b6-istio-envoy" pod "577f4375-87fc-4850-a066-7f10cf8307b6" (UID: "577f4375-87fc-4850-a066-7f10cf8307b6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:19:10.270023 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:19:10.269987 4376 generic.go:356] "Generic (PLEG): container finished" podID="577f4375-87fc-4850-a066-7f10cf8307b6" containerID="d3898aa45660ba8b16051cfca62058251c0de44e0fe8704e85516ef92a67b26a" exitCode=0 Oct 31 05:19:10.270233 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:19:10.270029 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-n5dnl" Oct 31 05:19:10.270233 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:19:10.270060 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-n5dnl" event={"ID":"577f4375-87fc-4850-a066-7f10cf8307b6","Type":"ContainerDied","Data":"d3898aa45660ba8b16051cfca62058251c0de44e0fe8704e85516ef92a67b26a"} Oct 31 05:19:10.270233 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:19:10.270116 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-n5dnl" event={"ID":"577f4375-87fc-4850-a066-7f10cf8307b6","Type":"ContainerDied","Data":"63d5d44fec161951bd1566338c230f954da0d30d72630c12c499b6e0582be07e"} Oct 31 05:19:10.270233 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:19:10.270144 4376 scope.go:122] "RemoveContainer" containerID="d3898aa45660ba8b16051cfca62058251c0de44e0fe8704e85516ef92a67b26a" Oct 31 05:19:10.280235 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:19:10.280218 4376 scope.go:122] "RemoveContainer" containerID="d3898aa45660ba8b16051cfca62058251c0de44e0fe8704e85516ef92a67b26a" Oct 31 05:19:10.280532 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:19:10.280513 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3898aa45660ba8b16051cfca62058251c0de44e0fe8704e85516ef92a67b26a\": container with ID starting with d3898aa45660ba8b16051cfca62058251c0de44e0fe8704e85516ef92a67b26a not found: ID does not exist" containerID="d3898aa45660ba8b16051cfca62058251c0de44e0fe8704e85516ef92a67b26a" Oct 31 05:19:10.280608 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:19:10.280546 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3898aa45660ba8b16051cfca62058251c0de44e0fe8704e85516ef92a67b26a"} err="failed to get container status \"d3898aa45660ba8b16051cfca62058251c0de44e0fe8704e85516ef92a67b26a\": rpc error: code = NotFound desc = could not find container \"d3898aa45660ba8b16051cfca62058251c0de44e0fe8704e85516ef92a67b26a\": container with ID starting with d3898aa45660ba8b16051cfca62058251c0de44e0fe8704e85516ef92a67b26a not found: ID does not exist" Oct 31 05:19:10.311840 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:19:10.311807 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8dczl\" (UniqueName: \"kubernetes.io/projected/577f4375-87fc-4850-a066-7f10cf8307b6-kube-api-access-8dczl\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:19:10.311840 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:19:10.311835 4376 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/577f4375-87fc-4850-a066-7f10cf8307b6-istio-podinfo\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:19:10.312018 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:19:10.311850 4376 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/577f4375-87fc-4850-a066-7f10cf8307b6-istio-token\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:19:10.312018 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:19:10.311863 4376 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/577f4375-87fc-4850-a066-7f10cf8307b6-istio-envoy\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:19:10.330636 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:19:10.330602 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-n5dnl"] Oct 31 05:19:10.334041 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:19:10.334019 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-n5dnl"] Oct 31 05:19:10.442345 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:19:10.442305 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="577f4375-87fc-4850-a066-7f10cf8307b6" path="/var/lib/kubelet/pods/577f4375-87fc-4850-a066-7f10cf8307b6/volumes" Oct 31 05:19:10.998216 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:19:10.998179 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 05:19:20.000843 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:19:20.000809 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ptb2g"] Oct 31 05:19:20.001426 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:19:20.001159 4376 cpu_manager.go:414] "containerMap: removing container" podUID="01ec4d78-9bd3-4762-a7c6-e277a212f2a6" containerName="maas-api" Oct 31 05:19:20.001426 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:19:20.001170 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01ec4d78-9bd3-4762-a7c6-e277a212f2a6" containerName="maas-api" Oct 31 05:19:20.001426 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:19:20.001189 4376 cpu_manager.go:414] "containerMap: removing container" podUID="6e27a6c1-bb5d-44e4-80ab-5617e8046bbf" containerName="manager" Oct 31 05:19:20.001426 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:19:20.001194 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e27a6c1-bb5d-44e4-80ab-5617e8046bbf" containerName="manager" Oct 31 05:19:20.001426 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:19:20.001205 4376 cpu_manager.go:414] "containerMap: removing container" podUID="577f4375-87fc-4850-a066-7f10cf8307b6" containerName="istio-proxy" Oct 31 05:19:20.001426 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:19:20.001211 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="577f4375-87fc-4850-a066-7f10cf8307b6" containerName="istio-proxy" Oct 31 05:19:20.001426 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:19:20.001218 4376 cpu_manager.go:414] "containerMap: removing container" podUID="72af9c95-36f4-467b-ba78-ec84c1d1d51c" containerName="cleanup" Oct 31 05:19:20.001426 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:19:20.001223 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72af9c95-36f4-467b-ba78-ec84c1d1d51c" containerName="cleanup" Oct 31 05:19:20.001426 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:19:20.001287 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="01ec4d78-9bd3-4762-a7c6-e277a212f2a6" containerName="maas-api" Oct 31 05:19:20.001426 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:19:20.001296 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e27a6c1-bb5d-44e4-80ab-5617e8046bbf" containerName="manager" Oct 31 05:19:20.001426 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:19:20.001302 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="577f4375-87fc-4850-a066-7f10cf8307b6" containerName="istio-proxy" Oct 31 05:19:20.001426 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:19:20.001307 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="72af9c95-36f4-467b-ba78-ec84c1d1d51c" containerName="cleanup" Oct 31 05:19:20.006864 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:19:20.006835 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ptb2g" Oct 31 05:19:20.026612 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:19:20.026580 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ptb2g"] Oct 31 05:19:20.101401 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:19:20.101359 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3f2eb40-6a45-42f1-9f44-71784631f635-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-ptb2g\" (UID: \"f3f2eb40-6a45-42f1-9f44-71784631f635\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ptb2g" Oct 31 05:19:20.101401 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:19:20.101401 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3f2eb40-6a45-42f1-9f44-71784631f635-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-ptb2g\" (UID: \"f3f2eb40-6a45-42f1-9f44-71784631f635\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ptb2g" Oct 31 05:19:20.101644 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:19:20.101482 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3f2eb40-6a45-42f1-9f44-71784631f635-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-ptb2g\" (UID: \"f3f2eb40-6a45-42f1-9f44-71784631f635\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ptb2g" Oct 31 05:19:20.101644 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:19:20.101523 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3f2eb40-6a45-42f1-9f44-71784631f635-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-ptb2g\" (UID: \"f3f2eb40-6a45-42f1-9f44-71784631f635\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ptb2g" Oct 31 05:19:20.101644 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:19:20.101619 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3f2eb40-6a45-42f1-9f44-71784631f635-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-ptb2g\" (UID: \"f3f2eb40-6a45-42f1-9f44-71784631f635\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ptb2g" Oct 31 05:19:20.101762 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:19:20.101654 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8k2qv\" (UniqueName: \"kubernetes.io/projected/f3f2eb40-6a45-42f1-9f44-71784631f635-kube-api-access-8k2qv\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-ptb2g\" (UID: \"f3f2eb40-6a45-42f1-9f44-71784631f635\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ptb2g" Oct 31 05:19:20.101762 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:19:20.101681 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3f2eb40-6a45-42f1-9f44-71784631f635-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-ptb2g\" (UID: \"f3f2eb40-6a45-42f1-9f44-71784631f635\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ptb2g" Oct 31 05:19:20.101842 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:19:20.101782 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3f2eb40-6a45-42f1-9f44-71784631f635-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-ptb2g\" (UID: \"f3f2eb40-6a45-42f1-9f44-71784631f635\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ptb2g" Oct 31 05:19:20.101842 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:19:20.101833 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3f2eb40-6a45-42f1-9f44-71784631f635-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-ptb2g\" (UID: \"f3f2eb40-6a45-42f1-9f44-71784631f635\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ptb2g" Oct 31 05:19:20.202308 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:19:20.202271 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3f2eb40-6a45-42f1-9f44-71784631f635-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-ptb2g\" (UID: \"f3f2eb40-6a45-42f1-9f44-71784631f635\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ptb2g" Oct 31 05:19:20.202519 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:19:20.202348 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3f2eb40-6a45-42f1-9f44-71784631f635-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-ptb2g\" (UID: \"f3f2eb40-6a45-42f1-9f44-71784631f635\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ptb2g" Oct 31 05:19:20.202519 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:19:20.202373 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3f2eb40-6a45-42f1-9f44-71784631f635-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-ptb2g\" (UID: \"f3f2eb40-6a45-42f1-9f44-71784631f635\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ptb2g" Oct 31 05:19:20.202519 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:19:20.202393 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3f2eb40-6a45-42f1-9f44-71784631f635-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-ptb2g\" (UID: \"f3f2eb40-6a45-42f1-9f44-71784631f635\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ptb2g" Oct 31 05:19:20.202519 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:19:20.202411 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3f2eb40-6a45-42f1-9f44-71784631f635-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-ptb2g\" (UID: \"f3f2eb40-6a45-42f1-9f44-71784631f635\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ptb2g" Oct 31 05:19:20.202519 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:19:20.202496 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3f2eb40-6a45-42f1-9f44-71784631f635-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-ptb2g\" (UID: \"f3f2eb40-6a45-42f1-9f44-71784631f635\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ptb2g" Oct 31 05:19:20.202800 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:19:20.202539 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8k2qv\" (UniqueName: \"kubernetes.io/projected/f3f2eb40-6a45-42f1-9f44-71784631f635-kube-api-access-8k2qv\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-ptb2g\" (UID: \"f3f2eb40-6a45-42f1-9f44-71784631f635\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ptb2g" Oct 31 05:19:20.202800 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:19:20.202582 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3f2eb40-6a45-42f1-9f44-71784631f635-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-ptb2g\" (UID: \"f3f2eb40-6a45-42f1-9f44-71784631f635\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ptb2g" Oct 31 05:19:20.202800 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:19:20.202624 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3f2eb40-6a45-42f1-9f44-71784631f635-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-ptb2g\" (UID: \"f3f2eb40-6a45-42f1-9f44-71784631f635\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ptb2g" Oct 31 05:19:20.202977 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:19:20.202859 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3f2eb40-6a45-42f1-9f44-71784631f635-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-ptb2g\" (UID: \"f3f2eb40-6a45-42f1-9f44-71784631f635\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ptb2g" Oct 31 05:19:20.202977 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:19:20.202926 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3f2eb40-6a45-42f1-9f44-71784631f635-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-ptb2g\" (UID: \"f3f2eb40-6a45-42f1-9f44-71784631f635\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ptb2g" Oct 31 05:19:20.203070 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:19:20.203020 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3f2eb40-6a45-42f1-9f44-71784631f635-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-ptb2g\" (UID: \"f3f2eb40-6a45-42f1-9f44-71784631f635\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ptb2g" Oct 31 05:19:20.203108 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:19:20.203093 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3f2eb40-6a45-42f1-9f44-71784631f635-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-ptb2g\" (UID: \"f3f2eb40-6a45-42f1-9f44-71784631f635\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ptb2g" Oct 31 05:19:20.203187 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:19:20.203165 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3f2eb40-6a45-42f1-9f44-71784631f635-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-ptb2g\" (UID: \"f3f2eb40-6a45-42f1-9f44-71784631f635\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ptb2g" Oct 31 05:19:20.204915 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:19:20.204890 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3f2eb40-6a45-42f1-9f44-71784631f635-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-ptb2g\" (UID: \"f3f2eb40-6a45-42f1-9f44-71784631f635\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ptb2g" Oct 31 05:19:20.205034 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:19:20.204969 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3f2eb40-6a45-42f1-9f44-71784631f635-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-ptb2g\" (UID: \"f3f2eb40-6a45-42f1-9f44-71784631f635\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ptb2g" Oct 31 05:19:20.229053 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:19:20.229027 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3f2eb40-6a45-42f1-9f44-71784631f635-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-ptb2g\" (UID: \"f3f2eb40-6a45-42f1-9f44-71784631f635\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ptb2g" Oct 31 05:19:20.229361 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:19:20.229318 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8k2qv\" (UniqueName: \"kubernetes.io/projected/f3f2eb40-6a45-42f1-9f44-71784631f635-kube-api-access-8k2qv\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-ptb2g\" (UID: \"f3f2eb40-6a45-42f1-9f44-71784631f635\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ptb2g" Oct 31 05:19:20.324756 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:19:20.324676 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ptb2g" Oct 31 05:19:20.636794 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:19:20.636755 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ptb2g"] Oct 31 05:19:20.636939 ip-10-0-134-155 kubenswrapper[4376]: W1031 05:19:20.636759 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3f2eb40_6a45_42f1_9f44_71784631f635.slice/crio-9d8a83457c0afdca1ef00a878844d4efed6ddb4c507cd924102965f0f8008afd WatchSource:0}: Error finding container 9d8a83457c0afdca1ef00a878844d4efed6ddb4c507cd924102965f0f8008afd: Status 404 returned error can't find the container with id 9d8a83457c0afdca1ef00a878844d4efed6ddb4c507cd924102965f0f8008afd Oct 31 05:19:20.638883 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:19:20.638846 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 05:19:20.638985 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:19:20.638914 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 05:19:20.638985 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:19:20.638944 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 05:19:21.325635 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:19:21.325599 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ptb2g" event={"ID":"f3f2eb40-6a45-42f1-9f44-71784631f635","Type":"ContainerStarted","Data":"4919e384a4179af1f0d2e2e77193c6bc4fc04e49eb4042390d165fcd48b6224e"} Oct 31 05:19:21.326068 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:19:21.325657 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ptb2g" event={"ID":"f3f2eb40-6a45-42f1-9f44-71784631f635","Type":"ContainerStarted","Data":"9d8a83457c0afdca1ef00a878844d4efed6ddb4c507cd924102965f0f8008afd"} Oct 31 05:19:21.392948 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:19:21.392894 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ptb2g" podStartSLOduration=2.392877923 podStartE2EDuration="2.392877923s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 05:19:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 05:19:21.392389417 +0000 UTC m=+4509.106724045" watchObservedRunningTime="2026-10-31 05:19:21.392877923 +0000 UTC m=+4509.107212537" Oct 31 05:19:22.325348 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:19:22.325294 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ptb2g" Oct 31 05:19:22.327230 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:19:22.326978 4376 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-ptb2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.244:15021/healthz/ready\": dial tcp 10.133.0.244:15021: connect: connection refused" start-of-body= Oct 31 05:19:22.327230 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:19:22.327039 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ptb2g" podUID="f3f2eb40-6a45-42f1-9f44-71784631f635" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.244:15021/healthz/ready\": dial tcp 10.133.0.244:15021: connect: connection refused" Oct 31 05:19:23.325225 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:19:23.325191 4376 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-ptb2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.244:15021/healthz/ready\": dial tcp 10.133.0.244:15021: connect: connection refused" start-of-body= Oct 31 05:19:23.325453 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:19:23.325249 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ptb2g" podUID="f3f2eb40-6a45-42f1-9f44-71784631f635" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.244:15021/healthz/ready\": dial tcp 10.133.0.244:15021: connect: connection refused" Oct 31 05:19:24.325999 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:19:24.325963 4376 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-ptb2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.244:15021/healthz/ready\": dial tcp 10.133.0.244:15021: connect: connection refused" start-of-body= Oct 31 05:19:24.326439 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:19:24.326019 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ptb2g" podUID="f3f2eb40-6a45-42f1-9f44-71784631f635" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.244:15021/healthz/ready\": dial tcp 10.133.0.244:15021: connect: connection refused" Oct 31 05:19:25.325192 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:19:25.325150 4376 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-ptb2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.244:15021/healthz/ready\": dial tcp 10.133.0.244:15021: connect: connection refused" start-of-body= Oct 31 05:19:25.325420 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:19:25.325206 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ptb2g" podUID="f3f2eb40-6a45-42f1-9f44-71784631f635" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.244:15021/healthz/ready\": dial tcp 10.133.0.244:15021: connect: connection refused" Oct 31 05:19:26.325344 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:19:26.325298 4376 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-ptb2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.244:15021/healthz/ready\": dial tcp 10.133.0.244:15021: connect: connection refused" start-of-body= Oct 31 05:19:26.325744 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:19:26.325379 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ptb2g" podUID="f3f2eb40-6a45-42f1-9f44-71784631f635" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.244:15021/healthz/ready\": dial tcp 10.133.0.244:15021: connect: connection refused" Oct 31 05:19:27.325894 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:19:27.325847 4376 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-ptb2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.244:15021/healthz/ready\": dial tcp 10.133.0.244:15021: connect: connection refused" start-of-body= Oct 31 05:19:27.326303 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:19:27.325919 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ptb2g" podUID="f3f2eb40-6a45-42f1-9f44-71784631f635" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.244:15021/healthz/ready\": dial tcp 10.133.0.244:15021: connect: connection refused" Oct 31 05:19:28.885214 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:19:28.885156 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ptb2g" Oct 31 05:19:28.885910 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:19:28.885544 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ptb2g" Oct 31 05:19:28.886534 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:19:28.886514 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ptb2g" Oct 31 05:19:29.707566 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:19:29.707529 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-85c4f7957c-jtbfr"] Oct 31 05:19:29.710918 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:19:29.710897 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-85c4f7957c-jtbfr" Oct 31 05:19:29.744120 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:19:29.744093 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Oct 31 05:19:29.744299 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:19:29.744284 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Oct 31 05:19:29.744395 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:19:29.744359 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-gw86m\"" Oct 31 05:19:29.744395 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:19:29.744374 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Oct 31 05:19:29.746360 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:19:29.746318 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-85c4f7957c-jtbfr"] Oct 31 05:19:29.824351 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:19:29.824299 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q59p7\" (UniqueName: \"kubernetes.io/projected/74c20194-9a91-40b7-b8b9-3d502c5d90be-kube-api-access-q59p7\") pod \"maas-controller-85c4f7957c-jtbfr\" (UID: \"74c20194-9a91-40b7-b8b9-3d502c5d90be\") " pod="opendatahub/maas-controller-85c4f7957c-jtbfr" Oct 31 05:19:29.824570 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:19:29.824374 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/74c20194-9a91-40b7-b8b9-3d502c5d90be-metrics-certs\") pod \"maas-controller-85c4f7957c-jtbfr\" (UID: \"74c20194-9a91-40b7-b8b9-3d502c5d90be\") " pod="opendatahub/maas-controller-85c4f7957c-jtbfr" Oct 31 05:19:29.824570 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:19:29.824523 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/74c20194-9a91-40b7-b8b9-3d502c5d90be-webhook-cert\") pod \"maas-controller-85c4f7957c-jtbfr\" (UID: \"74c20194-9a91-40b7-b8b9-3d502c5d90be\") " pod="opendatahub/maas-controller-85c4f7957c-jtbfr" Oct 31 05:19:29.926090 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:19:29.926051 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q59p7\" (UniqueName: \"kubernetes.io/projected/74c20194-9a91-40b7-b8b9-3d502c5d90be-kube-api-access-q59p7\") pod \"maas-controller-85c4f7957c-jtbfr\" (UID: \"74c20194-9a91-40b7-b8b9-3d502c5d90be\") " pod="opendatahub/maas-controller-85c4f7957c-jtbfr" Oct 31 05:19:29.926548 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:19:29.926101 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/74c20194-9a91-40b7-b8b9-3d502c5d90be-metrics-certs\") pod \"maas-controller-85c4f7957c-jtbfr\" (UID: \"74c20194-9a91-40b7-b8b9-3d502c5d90be\") " pod="opendatahub/maas-controller-85c4f7957c-jtbfr" Oct 31 05:19:29.926548 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:19:29.926152 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/74c20194-9a91-40b7-b8b9-3d502c5d90be-webhook-cert\") pod \"maas-controller-85c4f7957c-jtbfr\" (UID: \"74c20194-9a91-40b7-b8b9-3d502c5d90be\") " pod="opendatahub/maas-controller-85c4f7957c-jtbfr" Oct 31 05:19:29.928647 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:19:29.928624 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/74c20194-9a91-40b7-b8b9-3d502c5d90be-metrics-certs\") pod \"maas-controller-85c4f7957c-jtbfr\" (UID: \"74c20194-9a91-40b7-b8b9-3d502c5d90be\") " pod="opendatahub/maas-controller-85c4f7957c-jtbfr" Oct 31 05:19:29.928775 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:19:29.928744 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/74c20194-9a91-40b7-b8b9-3d502c5d90be-webhook-cert\") pod \"maas-controller-85c4f7957c-jtbfr\" (UID: \"74c20194-9a91-40b7-b8b9-3d502c5d90be\") " pod="opendatahub/maas-controller-85c4f7957c-jtbfr" Oct 31 05:19:29.951257 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:19:29.951221 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q59p7\" (UniqueName: \"kubernetes.io/projected/74c20194-9a91-40b7-b8b9-3d502c5d90be-kube-api-access-q59p7\") pod \"maas-controller-85c4f7957c-jtbfr\" (UID: \"74c20194-9a91-40b7-b8b9-3d502c5d90be\") " pod="opendatahub/maas-controller-85c4f7957c-jtbfr" Oct 31 05:19:30.023777 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:19:30.023685 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-85c4f7957c-jtbfr" Oct 31 05:19:30.321376 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:19:30.321324 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-85c4f7957c-jtbfr"] Oct 31 05:19:30.321724 ip-10-0-134-155 kubenswrapper[4376]: W1031 05:19:30.321699 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74c20194_9a91_40b7_b8b9_3d502c5d90be.slice/crio-0548d535cb1fc6b252fce10574658388e31f098d7936078d18c75b248cc03cab WatchSource:0}: Error finding container 0548d535cb1fc6b252fce10574658388e31f098d7936078d18c75b248cc03cab: Status 404 returned error can't find the container with id 0548d535cb1fc6b252fce10574658388e31f098d7936078d18c75b248cc03cab Oct 31 05:19:30.394015 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:19:30.393976 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-85c4f7957c-jtbfr" event={"ID":"74c20194-9a91-40b7-b8b9-3d502c5d90be","Type":"ContainerStarted","Data":"0548d535cb1fc6b252fce10574658388e31f098d7936078d18c75b248cc03cab"} Oct 31 05:19:31.399777 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:19:31.399741 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-85c4f7957c-jtbfr" event={"ID":"74c20194-9a91-40b7-b8b9-3d502c5d90be","Type":"ContainerStarted","Data":"46573c93652491ba06b41ab651f665afd65dcc487792fcaddd613807d96a5727"} Oct 31 05:19:31.400190 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:19:31.399840 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-85c4f7957c-jtbfr" Oct 31 05:19:31.447884 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:19:31.447832 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-85c4f7957c-jtbfr" podStartSLOduration=2.107873911 podStartE2EDuration="2.447815921s" totalImagesPullingTime="339.94201ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 05:19:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 05:19:31.446359225 +0000 UTC m=+4519.160693859" watchObservedRunningTime="2026-10-31 05:19:31.447815921 +0000 UTC m=+4519.162150538" Oct 31 05:19:40.999037 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:19:40.998996 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 05:19:42.407455 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:19:42.407424 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-85c4f7957c-jtbfr" Oct 31 05:19:49.730559 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:19:49.730523 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 05:20:06.026564 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:20:06.026528 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 05:20:11.033019 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:20:11.032979 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 05:20:11.709353 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:20:11.699573 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 05:20:19.039629 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:20:19.039585 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 05:20:23.728316 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:20:23.728284 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 05:20:40.998657 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:20:40.998617 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 05:20:59.353422 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:20:59.353379 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 05:21:01.950516 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:21:01.950478 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-prio-pool-119c987e-kserve-router-scheduler-64ddb68fd7-gp6lm"] Oct 31 05:21:01.954062 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:21:01.954030 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-119c987e-kserve-router-scheduler-64ddb68fd7-gp6lm" Oct 31 05:21:01.973770 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:21:01.973736 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-119c987e-kserve-router-scheduler-64ddb68fd7-gp6lm"] Oct 31 05:21:01.985970 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:21:01.985942 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-prio-pool-119c987e-kserve-self-signed-certs\"" Oct 31 05:21:01.986127 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:21:01.986006 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-prio-pool-119c987e-epp-sa-dockercfg-pdj6j\"" Oct 31 05:21:02.043624 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:21:02.043587 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x597q\" (UniqueName: \"kubernetes.io/projected/0dc4a32d-09c2-4e55-b847-737642b9b660-kube-api-access-x597q\") pod \"e2e-prio-pool-119c987e-kserve-router-scheduler-64ddb68fd7-gp6lm\" (UID: \"0dc4a32d-09c2-4e55-b847-737642b9b660\") " pod="llm/e2e-prio-pool-119c987e-kserve-router-scheduler-64ddb68fd7-gp6lm" Oct 31 05:21:02.043839 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:21:02.043648 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0dc4a32d-09c2-4e55-b847-737642b9b660-tls-certs\") pod \"e2e-prio-pool-119c987e-kserve-router-scheduler-64ddb68fd7-gp6lm\" (UID: \"0dc4a32d-09c2-4e55-b847-737642b9b660\") " pod="llm/e2e-prio-pool-119c987e-kserve-router-scheduler-64ddb68fd7-gp6lm" Oct 31 05:21:02.145008 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:21:02.144968 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x597q\" (UniqueName: \"kubernetes.io/projected/0dc4a32d-09c2-4e55-b847-737642b9b660-kube-api-access-x597q\") pod \"e2e-prio-pool-119c987e-kserve-router-scheduler-64ddb68fd7-gp6lm\" (UID: \"0dc4a32d-09c2-4e55-b847-737642b9b660\") " pod="llm/e2e-prio-pool-119c987e-kserve-router-scheduler-64ddb68fd7-gp6lm" Oct 31 05:21:02.145215 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:21:02.145038 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0dc4a32d-09c2-4e55-b847-737642b9b660-tls-certs\") pod \"e2e-prio-pool-119c987e-kserve-router-scheduler-64ddb68fd7-gp6lm\" (UID: \"0dc4a32d-09c2-4e55-b847-737642b9b660\") " pod="llm/e2e-prio-pool-119c987e-kserve-router-scheduler-64ddb68fd7-gp6lm" Oct 31 05:21:02.147573 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:21:02.147553 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0dc4a32d-09c2-4e55-b847-737642b9b660-tls-certs\") pod \"e2e-prio-pool-119c987e-kserve-router-scheduler-64ddb68fd7-gp6lm\" (UID: \"0dc4a32d-09c2-4e55-b847-737642b9b660\") " pod="llm/e2e-prio-pool-119c987e-kserve-router-scheduler-64ddb68fd7-gp6lm" Oct 31 05:21:02.170761 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:21:02.170733 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x597q\" (UniqueName: \"kubernetes.io/projected/0dc4a32d-09c2-4e55-b847-737642b9b660-kube-api-access-x597q\") pod \"e2e-prio-pool-119c987e-kserve-router-scheduler-64ddb68fd7-gp6lm\" (UID: \"0dc4a32d-09c2-4e55-b847-737642b9b660\") " pod="llm/e2e-prio-pool-119c987e-kserve-router-scheduler-64ddb68fd7-gp6lm" Oct 31 05:21:02.269420 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:21:02.269314 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-119c987e-kserve-router-scheduler-64ddb68fd7-gp6lm" Oct 31 05:21:02.566080 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:21:02.566050 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-119c987e-kserve-router-scheduler-64ddb68fd7-gp6lm"] Oct 31 05:21:02.566340 ip-10-0-134-155 kubenswrapper[4376]: W1031 05:21:02.566306 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dc4a32d_09c2_4e55_b847_737642b9b660.slice/crio-dd83d5a7fd629e62ae76dc6394a7f01117b3a25ffdfe7952031759fb49227bb1 WatchSource:0}: Error finding container dd83d5a7fd629e62ae76dc6394a7f01117b3a25ffdfe7952031759fb49227bb1: Status 404 returned error can't find the container with id dd83d5a7fd629e62ae76dc6394a7f01117b3a25ffdfe7952031759fb49227bb1 Oct 31 05:21:02.845227 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:21:02.845113 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-119c987e-kserve-router-scheduler-64ddb68fd7-gp6lm" event={"ID":"0dc4a32d-09c2-4e55-b847-737642b9b660","Type":"ContainerStarted","Data":"dd83d5a7fd629e62ae76dc6394a7f01117b3a25ffdfe7952031759fb49227bb1"} Oct 31 05:21:04.858755 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:21:04.858717 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-119c987e-kserve-router-scheduler-64ddb68fd7-gp6lm" event={"ID":"0dc4a32d-09c2-4e55-b847-737642b9b660","Type":"ContainerStarted","Data":"41ca06c7b6f176b2a2765574679d5878492ec47f850b08ea78f64c1cb8c7bdc4"} Oct 31 05:21:04.859489 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:21:04.859468 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-prio-pool-119c987e-kserve-router-scheduler-64ddb68fd7-gp6lm" Oct 31 05:21:04.904575 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:21:04.904506 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-prio-pool-119c987e-kserve-router-scheduler-64ddb68fd7-gp6lm" podStartSLOduration=2.282694144 podStartE2EDuration="3.904487243s" totalImagesPullingTime="1.621793099s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 05:21:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 05:21:04.904151289 +0000 UTC m=+4612.618485907" watchObservedRunningTime="2026-10-31 05:21:04.904487243 +0000 UTC m=+4612.618821862" Oct 31 05:21:08.246655 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:21:08.246621 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 05:21:10.999519 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:21:10.999478 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 05:21:36.887204 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:21:36.887170 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-prio-pool-119c987e-kserve-router-scheduler-64ddb68fd7-gp6lm" Oct 31 05:21:40.999295 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:21:40.999257 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 05:21:43.827616 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:21:43.827577 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 05:22:03.653721 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:22:03.653679 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 05:22:05.041064 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:22:05.041021 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-119c987e-kserve-router-scheduler-64ddb68fd7-gp6lm"] Oct 31 05:22:06.887566 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:22:06.887530 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-119c987e-kserve-router-scheduler-64ddb68fd7-gp6lm" podUID="0dc4a32d-09c2-4e55-b847-737642b9b660" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 05:22:10.999148 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:22:10.999111 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 05:22:11.037421 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:22:11.037387 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 05:22:13.406569 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:22:13.406521 4376 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-l4jlq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.240:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 05:22:13.407235 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:22:13.407195 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-l4jlq" podUID="d3e2da41-2d03-4b11-8ef0-40e3848109d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.240:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 05:22:16.887001 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:22:16.886961 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-119c987e-kserve-router-scheduler-64ddb68fd7-gp6lm" podUID="0dc4a32d-09c2-4e55-b847-737642b9b660" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 05:22:20.066636 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:22:20.066557 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-prio-pool-119c987e-kserve-router-scheduler-64ddb68fd7-gp6lm" podUID="0dc4a32d-09c2-4e55-b847-737642b9b660" containerName="main" containerID="cri-o://41ca06c7b6f176b2a2765574679d5878492ec47f850b08ea78f64c1cb8c7bdc4" gracePeriod=45 Oct 31 05:22:26.887207 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:22:26.887169 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-119c987e-kserve-router-scheduler-64ddb68fd7-gp6lm" podUID="0dc4a32d-09c2-4e55-b847-737642b9b660" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 05:22:26.887708 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:22:26.887292 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-prio-pool-119c987e-kserve-router-scheduler-64ddb68fd7-gp6lm" Oct 31 05:22:31.056917 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:22:31.056882 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 05:22:36.887256 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:22:36.887216 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-119c987e-kserve-router-scheduler-64ddb68fd7-gp6lm" podUID="0dc4a32d-09c2-4e55-b847-737642b9b660" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 05:22:40.998915 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:22:40.998878 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 05:22:45.874847 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:22:45.874803 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 05:22:46.887541 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:22:46.887506 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-119c987e-kserve-router-scheduler-64ddb68fd7-gp6lm" podUID="0dc4a32d-09c2-4e55-b847-737642b9b660" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 05:22:50.331119 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:22:50.331058 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-119c987e-kserve-router-scheduler-64ddb68fd7-gp6lm" Oct 31 05:22:50.390988 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:22:50.390963 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0dc4a32d-09c2-4e55-b847-737642b9b660-kube-api-access-x597q\" (UniqueName: \"kubernetes.io/projected/0dc4a32d-09c2-4e55-b847-737642b9b660-kube-api-access-x597q\") pod \"0dc4a32d-09c2-4e55-b847-737642b9b660\" (UID: \"0dc4a32d-09c2-4e55-b847-737642b9b660\") " Oct 31 05:22:50.391184 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:22:50.391043 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0dc4a32d-09c2-4e55-b847-737642b9b660-tls-certs\" (UniqueName: \"kubernetes.io/secret/0dc4a32d-09c2-4e55-b847-737642b9b660-tls-certs\") pod \"0dc4a32d-09c2-4e55-b847-737642b9b660\" (UID: \"0dc4a32d-09c2-4e55-b847-737642b9b660\") " Oct 31 05:22:50.391278 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:22:50.391228 4376 generic.go:356] "Generic (PLEG): container finished" podID="0dc4a32d-09c2-4e55-b847-737642b9b660" containerID="41ca06c7b6f176b2a2765574679d5878492ec47f850b08ea78f64c1cb8c7bdc4" exitCode=0 Oct 31 05:22:50.391372 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:22:50.391298 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-119c987e-kserve-router-scheduler-64ddb68fd7-gp6lm" Oct 31 05:22:50.391445 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:22:50.391352 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-119c987e-kserve-router-scheduler-64ddb68fd7-gp6lm" event={"ID":"0dc4a32d-09c2-4e55-b847-737642b9b660","Type":"ContainerDied","Data":"41ca06c7b6f176b2a2765574679d5878492ec47f850b08ea78f64c1cb8c7bdc4"} Oct 31 05:22:50.391504 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:22:50.391472 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-119c987e-kserve-router-scheduler-64ddb68fd7-gp6lm" event={"ID":"0dc4a32d-09c2-4e55-b847-737642b9b660","Type":"ContainerDied","Data":"dd83d5a7fd629e62ae76dc6394a7f01117b3a25ffdfe7952031759fb49227bb1"} Oct 31 05:22:50.391564 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:22:50.391505 4376 scope.go:122] "RemoveContainer" containerID="41ca06c7b6f176b2a2765574679d5878492ec47f850b08ea78f64c1cb8c7bdc4" Oct 31 05:22:50.393455 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:22:50.393421 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0dc4a32d-09c2-4e55-b847-737642b9b660-tls-certs" pod "0dc4a32d-09c2-4e55-b847-737642b9b660" (UID: "0dc4a32d-09c2-4e55-b847-737642b9b660"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 05:22:50.393587 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:22:50.393527 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0dc4a32d-09c2-4e55-b847-737642b9b660-kube-api-access-x597q" pod "0dc4a32d-09c2-4e55-b847-737642b9b660" (UID: "0dc4a32d-09c2-4e55-b847-737642b9b660"). InnerVolumeSpecName "kube-api-access-x597q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:22:50.405214 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:22:50.405193 4376 scope.go:122] "RemoveContainer" containerID="41ca06c7b6f176b2a2765574679d5878492ec47f850b08ea78f64c1cb8c7bdc4" Oct 31 05:22:50.405574 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:22:50.405555 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41ca06c7b6f176b2a2765574679d5878492ec47f850b08ea78f64c1cb8c7bdc4\": container with ID starting with 41ca06c7b6f176b2a2765574679d5878492ec47f850b08ea78f64c1cb8c7bdc4 not found: ID does not exist" containerID="41ca06c7b6f176b2a2765574679d5878492ec47f850b08ea78f64c1cb8c7bdc4" Oct 31 05:22:50.405654 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:22:50.405588 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41ca06c7b6f176b2a2765574679d5878492ec47f850b08ea78f64c1cb8c7bdc4"} err="failed to get container status \"41ca06c7b6f176b2a2765574679d5878492ec47f850b08ea78f64c1cb8c7bdc4\": rpc error: code = NotFound desc = could not find container \"41ca06c7b6f176b2a2765574679d5878492ec47f850b08ea78f64c1cb8c7bdc4\": container with ID starting with 41ca06c7b6f176b2a2765574679d5878492ec47f850b08ea78f64c1cb8c7bdc4 not found: ID does not exist" Oct 31 05:22:50.492788 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:22:50.492751 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x597q\" (UniqueName: \"kubernetes.io/projected/0dc4a32d-09c2-4e55-b847-737642b9b660-kube-api-access-x597q\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:22:50.492788 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:22:50.492787 4376 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0dc4a32d-09c2-4e55-b847-737642b9b660-tls-certs\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:22:50.734191 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:22:50.734158 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-119c987e-kserve-router-scheduler-64ddb68fd7-gp6lm"] Oct 31 05:22:50.748618 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:22:50.748590 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-prio-pool-119c987e-kserve-router-scheduler-64ddb68fd7-gp6lm"] Oct 31 05:22:52.441975 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:22:52.441941 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0dc4a32d-09c2-4e55-b847-737642b9b660" path="/var/lib/kubelet/pods/0dc4a32d-09c2-4e55-b847-737642b9b660/volumes" Oct 31 05:23:01.533728 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:23:01.533692 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 05:23:10.998280 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:23:10.998241 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 05:23:25.309965 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:23:25.309929 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 05:23:36.334430 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:23:36.334382 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 05:23:41.004515 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:23:41.004477 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 05:24:05.773484 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:24:05.773450 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5534f0-7dfb7cd94f-xsv4c"] Oct 31 05:24:05.773955 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:24:05.773815 4376 cpu_manager.go:414] "containerMap: removing container" podUID="0dc4a32d-09c2-4e55-b847-737642b9b660" containerName="main" Oct 31 05:24:05.773955 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:24:05.773826 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0dc4a32d-09c2-4e55-b847-737642b9b660" containerName="main" Oct 31 05:24:05.773955 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:24:05.773884 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="0dc4a32d-09c2-4e55-b847-737642b9b660" containerName="main" Oct 31 05:24:05.776838 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:24:05.776821 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5534f0-7dfb7cd94f-xsv4c" Oct 31 05:24:05.796995 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:24:05.796969 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5534f0-7dfb7cd94f-xsv4c"] Oct 31 05:24:05.806241 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:24:05.806217 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-5534f0-serving-cert\"" Oct 31 05:24:05.822912 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:24:05.822878 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4418aac6-6f1d-42cf-b1a8-611c5d84c22b-metrics-certs\") pod \"maas-api-e2e-worker-single-main-5534f0-7dfb7cd94f-xsv4c\" (UID: \"4418aac6-6f1d-42cf-b1a8-611c5d84c22b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5534f0-7dfb7cd94f-xsv4c" Oct 31 05:24:05.823050 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:24:05.823016 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4418aac6-6f1d-42cf-b1a8-611c5d84c22b-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-5534f0-7dfb7cd94f-xsv4c\" (UID: \"4418aac6-6f1d-42cf-b1a8-611c5d84c22b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5534f0-7dfb7cd94f-xsv4c" Oct 31 05:24:05.823102 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:24:05.823050 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4418aac6-6f1d-42cf-b1a8-611c5d84c22b-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-5534f0-7dfb7cd94f-xsv4c\" (UID: \"4418aac6-6f1d-42cf-b1a8-611c5d84c22b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5534f0-7dfb7cd94f-xsv4c" Oct 31 05:24:05.823146 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:24:05.823107 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpgpt\" (UniqueName: \"kubernetes.io/projected/4418aac6-6f1d-42cf-b1a8-611c5d84c22b-kube-api-access-lpgpt\") pod \"maas-api-e2e-worker-single-main-5534f0-7dfb7cd94f-xsv4c\" (UID: \"4418aac6-6f1d-42cf-b1a8-611c5d84c22b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5534f0-7dfb7cd94f-xsv4c" Oct 31 05:24:05.924713 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:24:05.924673 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4418aac6-6f1d-42cf-b1a8-611c5d84c22b-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-5534f0-7dfb7cd94f-xsv4c\" (UID: \"4418aac6-6f1d-42cf-b1a8-611c5d84c22b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5534f0-7dfb7cd94f-xsv4c" Oct 31 05:24:05.924713 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:24:05.924722 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4418aac6-6f1d-42cf-b1a8-611c5d84c22b-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-5534f0-7dfb7cd94f-xsv4c\" (UID: \"4418aac6-6f1d-42cf-b1a8-611c5d84c22b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5534f0-7dfb7cd94f-xsv4c" Oct 31 05:24:05.924929 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:24:05.924742 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpgpt\" (UniqueName: \"kubernetes.io/projected/4418aac6-6f1d-42cf-b1a8-611c5d84c22b-kube-api-access-lpgpt\") pod \"maas-api-e2e-worker-single-main-5534f0-7dfb7cd94f-xsv4c\" (UID: \"4418aac6-6f1d-42cf-b1a8-611c5d84c22b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5534f0-7dfb7cd94f-xsv4c" Oct 31 05:24:05.924929 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:24:05.924863 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4418aac6-6f1d-42cf-b1a8-611c5d84c22b-metrics-certs\") pod \"maas-api-e2e-worker-single-main-5534f0-7dfb7cd94f-xsv4c\" (UID: \"4418aac6-6f1d-42cf-b1a8-611c5d84c22b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5534f0-7dfb7cd94f-xsv4c" Oct 31 05:24:05.925352 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:24:05.925318 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4418aac6-6f1d-42cf-b1a8-611c5d84c22b-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-5534f0-7dfb7cd94f-xsv4c\" (UID: \"4418aac6-6f1d-42cf-b1a8-611c5d84c22b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5534f0-7dfb7cd94f-xsv4c" Oct 31 05:24:05.927285 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:24:05.927266 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4418aac6-6f1d-42cf-b1a8-611c5d84c22b-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-5534f0-7dfb7cd94f-xsv4c\" (UID: \"4418aac6-6f1d-42cf-b1a8-611c5d84c22b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5534f0-7dfb7cd94f-xsv4c" Oct 31 05:24:05.927352 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:24:05.927297 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4418aac6-6f1d-42cf-b1a8-611c5d84c22b-metrics-certs\") pod \"maas-api-e2e-worker-single-main-5534f0-7dfb7cd94f-xsv4c\" (UID: \"4418aac6-6f1d-42cf-b1a8-611c5d84c22b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5534f0-7dfb7cd94f-xsv4c" Oct 31 05:24:05.949983 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:24:05.949952 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpgpt\" (UniqueName: \"kubernetes.io/projected/4418aac6-6f1d-42cf-b1a8-611c5d84c22b-kube-api-access-lpgpt\") pod \"maas-api-e2e-worker-single-main-5534f0-7dfb7cd94f-xsv4c\" (UID: \"4418aac6-6f1d-42cf-b1a8-611c5d84c22b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5534f0-7dfb7cd94f-xsv4c" Oct 31 05:24:06.090134 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:24:06.090012 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5534f0-7dfb7cd94f-xsv4c" Oct 31 05:24:06.253383 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:24:06.253346 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-5534f0-cfbf7q4drj"] Oct 31 05:24:06.256944 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:24:06.256926 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-5534f0-cfbf7q4drj" Oct 31 05:24:06.273985 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:24:06.273962 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-5534f0-cfbf7q4drj"] Oct 31 05:24:06.276435 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:24:06.276412 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-bf061496-dockercfg-rlk8f\"" Oct 31 05:24:06.325241 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:24:06.325217 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-5534f0\"" Oct 31 05:24:06.328657 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:24:06.328626 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6gjg\" (UniqueName: \"kubernetes.io/projected/6ea13985-d736-40aa-9f9d-1665379e20bf-kube-api-access-n6gjg\") pod \"payload-pre-processing-e2e-worker-single-main-5534f0-cfbf7q4drj\" (UID: \"6ea13985-d736-40aa-9f9d-1665379e20bf\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-5534f0-cfbf7q4drj" Oct 31 05:24:06.328820 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:24:06.328795 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ea13985-d736-40aa-9f9d-1665379e20bf-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-5534f0-cfbf7q4drj\" (UID: \"6ea13985-d736-40aa-9f9d-1665379e20bf\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-5534f0-cfbf7q4drj" Oct 31 05:24:06.428215 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:24:06.428190 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5534f0-7dfb7cd94f-xsv4c"] Oct 31 05:24:06.429521 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:24:06.429494 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ea13985-d736-40aa-9f9d-1665379e20bf-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-5534f0-cfbf7q4drj\" (UID: \"6ea13985-d736-40aa-9f9d-1665379e20bf\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-5534f0-cfbf7q4drj" Oct 31 05:24:06.429675 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:24:06.429562 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6gjg\" (UniqueName: \"kubernetes.io/projected/6ea13985-d736-40aa-9f9d-1665379e20bf-kube-api-access-n6gjg\") pod \"payload-pre-processing-e2e-worker-single-main-5534f0-cfbf7q4drj\" (UID: \"6ea13985-d736-40aa-9f9d-1665379e20bf\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-5534f0-cfbf7q4drj" Oct 31 05:24:06.430003 ip-10-0-134-155 kubenswrapper[4376]: W1031 05:24:06.429980 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4418aac6_6f1d_42cf_b1a8_611c5d84c22b.slice/crio-c723ad46cdb72f358d20dc2de8ab2053464ff20b9217acba1d174ac9b92e5fa4 WatchSource:0}: Error finding container c723ad46cdb72f358d20dc2de8ab2053464ff20b9217acba1d174ac9b92e5fa4: Status 404 returned error can't find the container with id c723ad46cdb72f358d20dc2de8ab2053464ff20b9217acba1d174ac9b92e5fa4 Oct 31 05:24:06.430411 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:24:06.430388 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ea13985-d736-40aa-9f9d-1665379e20bf-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-5534f0-cfbf7q4drj\" (UID: \"6ea13985-d736-40aa-9f9d-1665379e20bf\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-5534f0-cfbf7q4drj" Oct 31 05:24:06.431648 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:24:06.431621 4376 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 05:24:06.455297 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:24:06.455270 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6gjg\" (UniqueName: \"kubernetes.io/projected/6ea13985-d736-40aa-9f9d-1665379e20bf-kube-api-access-n6gjg\") pod \"payload-pre-processing-e2e-worker-single-main-5534f0-cfbf7q4drj\" (UID: \"6ea13985-d736-40aa-9f9d-1665379e20bf\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-5534f0-cfbf7q4drj" Oct 31 05:24:06.569537 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:24:06.569508 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-5534f0-cfbf7q4drj" Oct 31 05:24:06.718852 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:24:06.718770 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5534f0-7dfb7cd94f-xsv4c" event={"ID":"4418aac6-6f1d-42cf-b1a8-611c5d84c22b","Type":"ContainerStarted","Data":"c723ad46cdb72f358d20dc2de8ab2053464ff20b9217acba1d174ac9b92e5fa4"} Oct 31 05:24:06.873914 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:24:06.873880 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-5534f0-cfbf7q4drj"] Oct 31 05:24:06.880827 ip-10-0-134-155 kubenswrapper[4376]: W1031 05:24:06.880798 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ea13985_d736_40aa_9f9d_1665379e20bf.slice/crio-1c332406a937ae8ff480bd10b65f4c1a638d9d13326b5ab8d728623881de37f7 WatchSource:0}: Error finding container 1c332406a937ae8ff480bd10b65f4c1a638d9d13326b5ab8d728623881de37f7: Status 404 returned error can't find the container with id 1c332406a937ae8ff480bd10b65f4c1a638d9d13326b5ab8d728623881de37f7 Oct 31 05:24:07.724300 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:24:07.724256 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5534f0-7dfb7cd94f-xsv4c" event={"ID":"4418aac6-6f1d-42cf-b1a8-611c5d84c22b","Type":"ContainerStarted","Data":"983679147a9780691875ee69bdccf6afab9418ec9eb799a01e47e04c83e94801"} Oct 31 05:24:07.724534 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:24:07.724361 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5534f0-7dfb7cd94f-xsv4c" Oct 31 05:24:07.725851 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:24:07.725822 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-5534f0-cfbf7q4drj" event={"ID":"6ea13985-d736-40aa-9f9d-1665379e20bf","Type":"ContainerStarted","Data":"b16d36e3f4398ab2fa1558e364ee02fd7025f31e4e08bff1b1b8e4b603f2adfb"} Oct 31 05:24:07.725971 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:24:07.725871 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-5534f0-cfbf7q4drj" event={"ID":"6ea13985-d736-40aa-9f9d-1665379e20bf","Type":"ContainerStarted","Data":"1c332406a937ae8ff480bd10b65f4c1a638d9d13326b5ab8d728623881de37f7"} Oct 31 05:24:07.725971 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:24:07.725962 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-5534f0-cfbf7q4drj" Oct 31 05:24:07.774345 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:24:07.774278 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5534f0-7dfb7cd94f-xsv4c" podStartSLOduration=2.342174054 podStartE2EDuration="2.774261025s" totalImagesPullingTime="432.086971ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 05:24:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 05:24:07.773187764 +0000 UTC m=+4795.487522380" watchObservedRunningTime="2026-10-31 05:24:07.774261025 +0000 UTC m=+4795.488595641" Oct 31 05:24:07.816945 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:24:07.816888 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-5534f0-cfbf7q4drj" podStartSLOduration=1.816873406 podStartE2EDuration="1.816873406s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 05:24:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 05:24:07.816633708 +0000 UTC m=+4795.530968329" watchObservedRunningTime="2026-10-31 05:24:07.816873406 +0000 UTC m=+4795.531208023" Oct 31 05:24:09.413782 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:24:09.413742 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 05:24:11.000242 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:24:11.000206 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 05:24:13.736479 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:24:13.736451 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5534f0-7dfb7cd94f-xsv4c" Oct 31 05:24:18.734187 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:24:18.734159 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-5534f0-cfbf7q4drj" Oct 31 05:24:24.610494 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:24:24.610457 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-5534f0-cfbf7q4drj"] Oct 31 05:24:25.684673 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:24:25.684637 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5534f0-7dfb7cd94f-xsv4c"] Oct 31 05:24:25.685177 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:24:25.684873 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5534f0-7dfb7cd94f-xsv4c" podUID="4418aac6-6f1d-42cf-b1a8-611c5d84c22b" containerName="maas-api" containerID="cri-o://983679147a9780691875ee69bdccf6afab9418ec9eb799a01e47e04c83e94801" gracePeriod=30 Oct 31 05:24:25.809921 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:24:25.809888 4376 generic.go:356] "Generic (PLEG): container finished" podID="4418aac6-6f1d-42cf-b1a8-611c5d84c22b" containerID="983679147a9780691875ee69bdccf6afab9418ec9eb799a01e47e04c83e94801" exitCode=0 Oct 31 05:24:25.810079 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:24:25.809960 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5534f0-7dfb7cd94f-xsv4c" event={"ID":"4418aac6-6f1d-42cf-b1a8-611c5d84c22b","Type":"ContainerDied","Data":"983679147a9780691875ee69bdccf6afab9418ec9eb799a01e47e04c83e94801"} Oct 31 05:24:25.934582 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:24:25.934539 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5534f0-7dfb7cd94f-xsv4c" Oct 31 05:24:26.049990 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:24:26.049903 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4418aac6-6f1d-42cf-b1a8-611c5d84c22b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4418aac6-6f1d-42cf-b1a8-611c5d84c22b-metrics-certs\") pod \"4418aac6-6f1d-42cf-b1a8-611c5d84c22b\" (UID: \"4418aac6-6f1d-42cf-b1a8-611c5d84c22b\") " Oct 31 05:24:26.049990 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:24:26.049959 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4418aac6-6f1d-42cf-b1a8-611c5d84c22b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4418aac6-6f1d-42cf-b1a8-611c5d84c22b-maas-api-tls\") pod \"4418aac6-6f1d-42cf-b1a8-611c5d84c22b\" (UID: \"4418aac6-6f1d-42cf-b1a8-611c5d84c22b\") " Oct 31 05:24:26.050248 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:24:26.050001 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4418aac6-6f1d-42cf-b1a8-611c5d84c22b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4418aac6-6f1d-42cf-b1a8-611c5d84c22b-openshift-service-ca\") pod \"4418aac6-6f1d-42cf-b1a8-611c5d84c22b\" (UID: \"4418aac6-6f1d-42cf-b1a8-611c5d84c22b\") " Oct 31 05:24:26.050248 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:24:26.050116 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4418aac6-6f1d-42cf-b1a8-611c5d84c22b-kube-api-access-lpgpt\" (UniqueName: \"kubernetes.io/projected/4418aac6-6f1d-42cf-b1a8-611c5d84c22b-kube-api-access-lpgpt\") pod \"4418aac6-6f1d-42cf-b1a8-611c5d84c22b\" (UID: \"4418aac6-6f1d-42cf-b1a8-611c5d84c22b\") " Oct 31 05:24:26.050529 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:24:26.050499 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4418aac6-6f1d-42cf-b1a8-611c5d84c22b-openshift-service-ca" pod "4418aac6-6f1d-42cf-b1a8-611c5d84c22b" (UID: "4418aac6-6f1d-42cf-b1a8-611c5d84c22b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:24:26.052190 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:24:26.052159 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4418aac6-6f1d-42cf-b1a8-611c5d84c22b-maas-api-tls" pod "4418aac6-6f1d-42cf-b1a8-611c5d84c22b" (UID: "4418aac6-6f1d-42cf-b1a8-611c5d84c22b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 05:24:26.052302 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:24:26.052266 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4418aac6-6f1d-42cf-b1a8-611c5d84c22b-kube-api-access-lpgpt" pod "4418aac6-6f1d-42cf-b1a8-611c5d84c22b" (UID: "4418aac6-6f1d-42cf-b1a8-611c5d84c22b"). InnerVolumeSpecName "kube-api-access-lpgpt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:24:26.052405 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:24:26.052379 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4418aac6-6f1d-42cf-b1a8-611c5d84c22b-metrics-certs" pod "4418aac6-6f1d-42cf-b1a8-611c5d84c22b" (UID: "4418aac6-6f1d-42cf-b1a8-611c5d84c22b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 05:24:26.151499 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:24:26.151459 4376 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4418aac6-6f1d-42cf-b1a8-611c5d84c22b-metrics-certs\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:24:26.151499 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:24:26.151493 4376 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4418aac6-6f1d-42cf-b1a8-611c5d84c22b-maas-api-tls\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:24:26.151711 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:24:26.151506 4376 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4418aac6-6f1d-42cf-b1a8-611c5d84c22b-openshift-service-ca\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:24:26.151711 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:24:26.151520 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lpgpt\" (UniqueName: \"kubernetes.io/projected/4418aac6-6f1d-42cf-b1a8-611c5d84c22b-kube-api-access-lpgpt\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:24:26.815959 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:24:26.815870 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5534f0-7dfb7cd94f-xsv4c" event={"ID":"4418aac6-6f1d-42cf-b1a8-611c5d84c22b","Type":"ContainerDied","Data":"c723ad46cdb72f358d20dc2de8ab2053464ff20b9217acba1d174ac9b92e5fa4"} Oct 31 05:24:26.815959 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:24:26.815900 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5534f0-7dfb7cd94f-xsv4c" Oct 31 05:24:26.815959 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:24:26.815929 4376 scope.go:122] "RemoveContainer" containerID="983679147a9780691875ee69bdccf6afab9418ec9eb799a01e47e04c83e94801" Oct 31 05:24:26.873303 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:24:26.873270 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5534f0-7dfb7cd94f-xsv4c"] Oct 31 05:24:26.908823 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:24:26.908780 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5534f0-7dfb7cd94f-xsv4c"] Oct 31 05:24:28.441610 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:24:28.441572 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4418aac6-6f1d-42cf-b1a8-611c5d84c22b" path="/var/lib/kubelet/pods/4418aac6-6f1d-42cf-b1a8-611c5d84c22b/volumes" Oct 31 05:24:28.935954 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:24:28.935920 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 05:24:29.611545 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:24:29.611505 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-5534f0-cfbf7q4drj" podUID="6ea13985-d736-40aa-9f9d-1665379e20bf" containerName="payload-pre-processing" containerID="cri-o://b16d36e3f4398ab2fa1558e364ee02fd7025f31e4e08bff1b1b8e4b603f2adfb" gracePeriod=25 Oct 31 05:24:29.831555 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:24:29.831524 4376 generic.go:356] "Generic (PLEG): container finished" podID="6ea13985-d736-40aa-9f9d-1665379e20bf" containerID="b16d36e3f4398ab2fa1558e364ee02fd7025f31e4e08bff1b1b8e4b603f2adfb" exitCode=0 Oct 31 05:24:29.831733 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:24:29.831574 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-5534f0-cfbf7q4drj" event={"ID":"6ea13985-d736-40aa-9f9d-1665379e20bf","Type":"ContainerDied","Data":"b16d36e3f4398ab2fa1558e364ee02fd7025f31e4e08bff1b1b8e4b603f2adfb"} Oct 31 05:24:29.859592 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:24:29.859569 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-5534f0-cfbf7q4drj" Oct 31 05:24:29.990633 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:24:29.990600 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ea13985-d736-40aa-9f9d-1665379e20bf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ea13985-d736-40aa-9f9d-1665379e20bf-plugins-config-volume\") pod \"6ea13985-d736-40aa-9f9d-1665379e20bf\" (UID: \"6ea13985-d736-40aa-9f9d-1665379e20bf\") " Oct 31 05:24:29.990838 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:24:29.990747 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ea13985-d736-40aa-9f9d-1665379e20bf-kube-api-access-n6gjg\" (UniqueName: \"kubernetes.io/projected/6ea13985-d736-40aa-9f9d-1665379e20bf-kube-api-access-n6gjg\") pod \"6ea13985-d736-40aa-9f9d-1665379e20bf\" (UID: \"6ea13985-d736-40aa-9f9d-1665379e20bf\") " Oct 31 05:24:29.990958 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:24:29.990936 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ea13985-d736-40aa-9f9d-1665379e20bf-plugins-config-volume" pod "6ea13985-d736-40aa-9f9d-1665379e20bf" (UID: "6ea13985-d736-40aa-9f9d-1665379e20bf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:24:29.992834 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:24:29.992812 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ea13985-d736-40aa-9f9d-1665379e20bf-kube-api-access-n6gjg" pod "6ea13985-d736-40aa-9f9d-1665379e20bf" (UID: "6ea13985-d736-40aa-9f9d-1665379e20bf"). InnerVolumeSpecName "kube-api-access-n6gjg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:24:30.091606 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:24:30.091568 4376 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ea13985-d736-40aa-9f9d-1665379e20bf-plugins-config-volume\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:24:30.091606 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:24:30.091599 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n6gjg\" (UniqueName: \"kubernetes.io/projected/6ea13985-d736-40aa-9f9d-1665379e20bf-kube-api-access-n6gjg\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:24:30.837900 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:24:30.837811 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-5534f0-cfbf7q4drj" event={"ID":"6ea13985-d736-40aa-9f9d-1665379e20bf","Type":"ContainerDied","Data":"1c332406a937ae8ff480bd10b65f4c1a638d9d13326b5ab8d728623881de37f7"} Oct 31 05:24:30.837900 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:24:30.837878 4376 scope.go:122] "RemoveContainer" containerID="b16d36e3f4398ab2fa1558e364ee02fd7025f31e4e08bff1b1b8e4b603f2adfb" Oct 31 05:24:30.838421 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:24:30.837827 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-5534f0-cfbf7q4drj" Oct 31 05:24:30.896363 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:24:30.896313 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-5534f0-cfbf7q4drj"] Oct 31 05:24:30.910591 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:24:30.910557 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-5534f0-cfbf7q4drj"] Oct 31 05:24:32.441045 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:24:32.441004 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ea13985-d736-40aa-9f9d-1665379e20bf" path="/var/lib/kubelet/pods/6ea13985-d736-40aa-9f9d-1665379e20bf/volumes" Oct 31 05:24:34.511735 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:24:34.511703 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 05:24:40.999111 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:24:40.999067 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 05:24:50.050991 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:24:50.050948 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 05:25:06.764603 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:25:06.764556 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 05:25:10.998979 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:25:10.998943 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 05:25:18.832941 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:25:18.832903 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 05:25:28.035490 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:25:28.035453 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 05:25:38.339725 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:25:38.339688 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 05:25:41.000276 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:25:41.000242 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 05:25:42.946500 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:25:42.946469 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 05:25:58.444453 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:25:58.444368 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 05:26:10.998046 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:26:10.998007 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 05:26:15.406023 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:26:15.405990 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-n2mk7"] Oct 31 05:26:17.115288 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:26:17.115246 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-7d68ddf775-nv8tc"] Oct 31 05:26:17.115825 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:26:17.115651 4376 cpu_manager.go:414] "containerMap: removing container" podUID="6ea13985-d736-40aa-9f9d-1665379e20bf" containerName="payload-pre-processing" Oct 31 05:26:17.115825 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:26:17.115668 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ea13985-d736-40aa-9f9d-1665379e20bf" containerName="payload-pre-processing" Oct 31 05:26:17.115825 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:26:17.115703 4376 cpu_manager.go:414] "containerMap: removing container" podUID="4418aac6-6f1d-42cf-b1a8-611c5d84c22b" containerName="maas-api" Oct 31 05:26:17.115825 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:26:17.115711 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4418aac6-6f1d-42cf-b1a8-611c5d84c22b" containerName="maas-api" Oct 31 05:26:17.115825 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:26:17.115817 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="4418aac6-6f1d-42cf-b1a8-611c5d84c22b" containerName="maas-api" Oct 31 05:26:17.116064 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:26:17.115838 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ea13985-d736-40aa-9f9d-1665379e20bf" containerName="payload-pre-processing" Oct 31 05:26:17.119558 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:26:17.119535 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-7d68ddf775-nv8tc" Oct 31 05:26:17.136496 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:26:17.136467 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-7d68ddf775-nv8tc"] Oct 31 05:26:17.196985 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:26:17.196952 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ca3851f-64ee-4d94-9216-425ead298d3f-plugins-config-volume\") pod \"payload-processing-7d68ddf775-nv8tc\" (UID: \"4ca3851f-64ee-4d94-9216-425ead298d3f\") " pod="openshift-ingress/payload-processing-7d68ddf775-nv8tc" Oct 31 05:26:17.197158 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:26:17.197079 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6lx7\" (UniqueName: \"kubernetes.io/projected/4ca3851f-64ee-4d94-9216-425ead298d3f-kube-api-access-w6lx7\") pod \"payload-processing-7d68ddf775-nv8tc\" (UID: \"4ca3851f-64ee-4d94-9216-425ead298d3f\") " pod="openshift-ingress/payload-processing-7d68ddf775-nv8tc" Oct 31 05:26:17.298396 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:26:17.298360 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ca3851f-64ee-4d94-9216-425ead298d3f-plugins-config-volume\") pod \"payload-processing-7d68ddf775-nv8tc\" (UID: \"4ca3851f-64ee-4d94-9216-425ead298d3f\") " pod="openshift-ingress/payload-processing-7d68ddf775-nv8tc" Oct 31 05:26:17.298575 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:26:17.298452 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6lx7\" (UniqueName: \"kubernetes.io/projected/4ca3851f-64ee-4d94-9216-425ead298d3f-kube-api-access-w6lx7\") pod \"payload-processing-7d68ddf775-nv8tc\" (UID: \"4ca3851f-64ee-4d94-9216-425ead298d3f\") " pod="openshift-ingress/payload-processing-7d68ddf775-nv8tc" Oct 31 05:26:17.299063 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:26:17.299043 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ca3851f-64ee-4d94-9216-425ead298d3f-plugins-config-volume\") pod \"payload-processing-7d68ddf775-nv8tc\" (UID: \"4ca3851f-64ee-4d94-9216-425ead298d3f\") " pod="openshift-ingress/payload-processing-7d68ddf775-nv8tc" Oct 31 05:26:17.327272 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:26:17.327241 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6lx7\" (UniqueName: \"kubernetes.io/projected/4ca3851f-64ee-4d94-9216-425ead298d3f-kube-api-access-w6lx7\") pod \"payload-processing-7d68ddf775-nv8tc\" (UID: \"4ca3851f-64ee-4d94-9216-425ead298d3f\") " pod="openshift-ingress/payload-processing-7d68ddf775-nv8tc" Oct 31 05:26:17.432840 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:26:17.432751 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-7d68ddf775-nv8tc" Oct 31 05:26:17.732660 ip-10-0-134-155 kubenswrapper[4376]: W1031 05:26:17.732628 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ca3851f_64ee_4d94_9216_425ead298d3f.slice/crio-2f2a9aca31a231f7f42493c2823f642274d72a156d24d4bb80a8bae92c9a5cbc WatchSource:0}: Error finding container 2f2a9aca31a231f7f42493c2823f642274d72a156d24d4bb80a8bae92c9a5cbc: Status 404 returned error can't find the container with id 2f2a9aca31a231f7f42493c2823f642274d72a156d24d4bb80a8bae92c9a5cbc Oct 31 05:26:17.733085 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:26:17.733037 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-7d68ddf775-nv8tc"] Oct 31 05:26:18.458972 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:26:18.458936 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-7d68ddf775-nv8tc" event={"ID":"4ca3851f-64ee-4d94-9216-425ead298d3f","Type":"ContainerStarted","Data":"851eea68e2289a1245b6ad19f49b937e25971357c1b15f9e7af81bb2047fdbf8"} Oct 31 05:26:18.459394 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:26:18.458991 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-7d68ddf775-nv8tc" event={"ID":"4ca3851f-64ee-4d94-9216-425ead298d3f","Type":"ContainerStarted","Data":"2f2a9aca31a231f7f42493c2823f642274d72a156d24d4bb80a8bae92c9a5cbc"} Oct 31 05:26:18.459394 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:26:18.459075 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-7d68ddf775-nv8tc" Oct 31 05:26:18.504646 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:26:18.504602 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-7d68ddf775-nv8tc" podStartSLOduration=1.504590071 podStartE2EDuration="1.504590071s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 05:26:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 05:26:18.503410827 +0000 UTC m=+4926.217745464" watchObservedRunningTime="2026-10-31 05:26:18.504590071 +0000 UTC m=+4926.218924687" Oct 31 05:26:20.406536 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:26:20.406474 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-7b98bb678c-n2mk7" podUID="9700b83d-909b-46b4-98d4-c01c16aed865" containerName="payload-pre-processing" containerID="cri-o://80762b253d0de69f03f21af7886ecccde022c45bfcf55ea10854983986c8581c" gracePeriod=25 Oct 31 05:26:20.654520 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:26:20.654494 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-n2mk7" Oct 31 05:26:20.730280 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:26:20.730201 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9700b83d-909b-46b4-98d4-c01c16aed865-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9700b83d-909b-46b4-98d4-c01c16aed865-plugins-config-volume\") pod \"9700b83d-909b-46b4-98d4-c01c16aed865\" (UID: \"9700b83d-909b-46b4-98d4-c01c16aed865\") " Oct 31 05:26:20.730487 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:26:20.730367 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9700b83d-909b-46b4-98d4-c01c16aed865-kube-api-access-wtgwc\" (UniqueName: \"kubernetes.io/projected/9700b83d-909b-46b4-98d4-c01c16aed865-kube-api-access-wtgwc\") pod \"9700b83d-909b-46b4-98d4-c01c16aed865\" (UID: \"9700b83d-909b-46b4-98d4-c01c16aed865\") " Oct 31 05:26:20.730658 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:26:20.730634 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9700b83d-909b-46b4-98d4-c01c16aed865-plugins-config-volume" pod "9700b83d-909b-46b4-98d4-c01c16aed865" (UID: "9700b83d-909b-46b4-98d4-c01c16aed865"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:26:20.732460 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:26:20.732430 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9700b83d-909b-46b4-98d4-c01c16aed865-kube-api-access-wtgwc" pod "9700b83d-909b-46b4-98d4-c01c16aed865" (UID: "9700b83d-909b-46b4-98d4-c01c16aed865"). InnerVolumeSpecName "kube-api-access-wtgwc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:26:20.831497 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:26:20.831447 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtgwc\" (UniqueName: \"kubernetes.io/projected/9700b83d-909b-46b4-98d4-c01c16aed865-kube-api-access-wtgwc\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:26:20.831497 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:26:20.831490 4376 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9700b83d-909b-46b4-98d4-c01c16aed865-plugins-config-volume\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:26:21.472165 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:26:21.472129 4376 generic.go:356] "Generic (PLEG): container finished" podID="9700b83d-909b-46b4-98d4-c01c16aed865" containerID="80762b253d0de69f03f21af7886ecccde022c45bfcf55ea10854983986c8581c" exitCode=0 Oct 31 05:26:21.472703 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:26:21.472179 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-n2mk7" Oct 31 05:26:21.472703 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:26:21.472217 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-n2mk7" event={"ID":"9700b83d-909b-46b4-98d4-c01c16aed865","Type":"ContainerDied","Data":"80762b253d0de69f03f21af7886ecccde022c45bfcf55ea10854983986c8581c"} Oct 31 05:26:21.472703 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:26:21.472277 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-n2mk7" event={"ID":"9700b83d-909b-46b4-98d4-c01c16aed865","Type":"ContainerDied","Data":"fc0d20a3fd68873f8697cb6fd431fd01ed647c035666155f4adfc6d9a6a3e4fc"} Oct 31 05:26:21.472703 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:26:21.472301 4376 scope.go:122] "RemoveContainer" containerID="80762b253d0de69f03f21af7886ecccde022c45bfcf55ea10854983986c8581c" Oct 31 05:26:21.481693 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:26:21.481672 4376 scope.go:122] "RemoveContainer" containerID="80762b253d0de69f03f21af7886ecccde022c45bfcf55ea10854983986c8581c" Oct 31 05:26:21.481953 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:26:21.481932 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80762b253d0de69f03f21af7886ecccde022c45bfcf55ea10854983986c8581c\": container with ID starting with 80762b253d0de69f03f21af7886ecccde022c45bfcf55ea10854983986c8581c not found: ID does not exist" containerID="80762b253d0de69f03f21af7886ecccde022c45bfcf55ea10854983986c8581c" Oct 31 05:26:21.482066 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:26:21.481961 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80762b253d0de69f03f21af7886ecccde022c45bfcf55ea10854983986c8581c"} err="failed to get container status \"80762b253d0de69f03f21af7886ecccde022c45bfcf55ea10854983986c8581c\": rpc error: code = NotFound desc = could not find container \"80762b253d0de69f03f21af7886ecccde022c45bfcf55ea10854983986c8581c\": container with ID starting with 80762b253d0de69f03f21af7886ecccde022c45bfcf55ea10854983986c8581c not found: ID does not exist" Oct 31 05:26:21.530964 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:26:21.530932 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-n2mk7"] Oct 31 05:26:21.545847 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:26:21.545815 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-n2mk7"] Oct 31 05:26:22.442459 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:26:22.442424 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9700b83d-909b-46b4-98d4-c01c16aed865" path="/var/lib/kubelet/pods/9700b83d-909b-46b4-98d4-c01c16aed865/volumes" Oct 31 05:26:29.466051 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:26:29.466018 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-7d68ddf775-nv8tc" Oct 31 05:26:29.540944 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:26:29.540912 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-cpb5g"] Oct 31 05:26:31.375025 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:26:31.374990 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-6ddb9b8755-bwn8k"] Oct 31 05:26:31.375419 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:26:31.375384 4376 cpu_manager.go:414] "containerMap: removing container" podUID="9700b83d-909b-46b4-98d4-c01c16aed865" containerName="payload-pre-processing" Oct 31 05:26:31.375419 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:26:31.375395 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9700b83d-909b-46b4-98d4-c01c16aed865" containerName="payload-pre-processing" Oct 31 05:26:31.375510 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:26:31.375461 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="9700b83d-909b-46b4-98d4-c01c16aed865" containerName="payload-pre-processing" Oct 31 05:26:31.378632 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:26:31.378613 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-6ddb9b8755-bwn8k" Oct 31 05:26:31.395873 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:26:31.395848 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-6ddb9b8755-bwn8k"] Oct 31 05:26:31.435990 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:26:31.435954 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf2ff14c-f327-4ea4-aa03-92a0926aba6b-plugins-config-volume\") pod \"payload-processing-6ddb9b8755-bwn8k\" (UID: \"cf2ff14c-f327-4ea4-aa03-92a0926aba6b\") " pod="openshift-ingress/payload-processing-6ddb9b8755-bwn8k" Oct 31 05:26:31.436159 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:26:31.436031 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbhgv\" (UniqueName: \"kubernetes.io/projected/cf2ff14c-f327-4ea4-aa03-92a0926aba6b-kube-api-access-fbhgv\") pod \"payload-processing-6ddb9b8755-bwn8k\" (UID: \"cf2ff14c-f327-4ea4-aa03-92a0926aba6b\") " pod="openshift-ingress/payload-processing-6ddb9b8755-bwn8k" Oct 31 05:26:31.537262 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:26:31.537230 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbhgv\" (UniqueName: \"kubernetes.io/projected/cf2ff14c-f327-4ea4-aa03-92a0926aba6b-kube-api-access-fbhgv\") pod \"payload-processing-6ddb9b8755-bwn8k\" (UID: \"cf2ff14c-f327-4ea4-aa03-92a0926aba6b\") " pod="openshift-ingress/payload-processing-6ddb9b8755-bwn8k" Oct 31 05:26:31.537467 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:26:31.537386 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf2ff14c-f327-4ea4-aa03-92a0926aba6b-plugins-config-volume\") pod \"payload-processing-6ddb9b8755-bwn8k\" (UID: \"cf2ff14c-f327-4ea4-aa03-92a0926aba6b\") " pod="openshift-ingress/payload-processing-6ddb9b8755-bwn8k" Oct 31 05:26:31.538188 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:26:31.538159 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf2ff14c-f327-4ea4-aa03-92a0926aba6b-plugins-config-volume\") pod \"payload-processing-6ddb9b8755-bwn8k\" (UID: \"cf2ff14c-f327-4ea4-aa03-92a0926aba6b\") " pod="openshift-ingress/payload-processing-6ddb9b8755-bwn8k" Oct 31 05:26:31.564185 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:26:31.564153 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbhgv\" (UniqueName: \"kubernetes.io/projected/cf2ff14c-f327-4ea4-aa03-92a0926aba6b-kube-api-access-fbhgv\") pod \"payload-processing-6ddb9b8755-bwn8k\" (UID: \"cf2ff14c-f327-4ea4-aa03-92a0926aba6b\") " pod="openshift-ingress/payload-processing-6ddb9b8755-bwn8k" Oct 31 05:26:31.691591 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:26:31.691512 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-6ddb9b8755-bwn8k" Oct 31 05:26:31.988237 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:26:31.988208 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-6ddb9b8755-bwn8k"] Oct 31 05:26:31.988612 ip-10-0-134-155 kubenswrapper[4376]: W1031 05:26:31.988585 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf2ff14c_f327_4ea4_aa03_92a0926aba6b.slice/crio-e99aa5896e5515f1b214d2a8f9fe4796554e58acfa413b1a189d78931d49536a WatchSource:0}: Error finding container e99aa5896e5515f1b214d2a8f9fe4796554e58acfa413b1a189d78931d49536a: Status 404 returned error can't find the container with id e99aa5896e5515f1b214d2a8f9fe4796554e58acfa413b1a189d78931d49536a Oct 31 05:26:32.523587 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:26:32.523544 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-6ddb9b8755-bwn8k" event={"ID":"cf2ff14c-f327-4ea4-aa03-92a0926aba6b","Type":"ContainerStarted","Data":"d37ed089b4e4169308868d70bfb5818fd1f30fc65000ae4f192a75fa58715b93"} Oct 31 05:26:32.524043 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:26:32.523615 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-6ddb9b8755-bwn8k" event={"ID":"cf2ff14c-f327-4ea4-aa03-92a0926aba6b","Type":"ContainerStarted","Data":"e99aa5896e5515f1b214d2a8f9fe4796554e58acfa413b1a189d78931d49536a"} Oct 31 05:26:32.524043 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:26:32.523677 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-6ddb9b8755-bwn8k" Oct 31 05:26:32.569411 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:26:32.569357 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-6ddb9b8755-bwn8k" podStartSLOduration=1.569342538 podStartE2EDuration="1.569342538s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 05:26:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 05:26:32.56776951 +0000 UTC m=+4940.282104146" watchObservedRunningTime="2026-10-31 05:26:32.569342538 +0000 UTC m=+4940.283677152" Oct 31 05:26:34.541869 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:26:34.541796 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-55fbbbf9dc-cpb5g" podUID="83cfbbfb-a15c-4967-b20c-e102351c6b7b" containerName="payload-processing" containerID="cri-o://b577ebd21405364322a9ee69cce9b1e2fbe1ccd28f3dea8f3b5823010c860e76" gracePeriod=25 Oct 31 05:26:34.793977 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:26:34.793919 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-cpb5g" Oct 31 05:26:34.870997 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:26:34.870966 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83cfbbfb-a15c-4967-b20c-e102351c6b7b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83cfbbfb-a15c-4967-b20c-e102351c6b7b-plugins-config-volume\") pod \"83cfbbfb-a15c-4967-b20c-e102351c6b7b\" (UID: \"83cfbbfb-a15c-4967-b20c-e102351c6b7b\") " Oct 31 05:26:34.870997 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:26:34.871005 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83cfbbfb-a15c-4967-b20c-e102351c6b7b-kube-api-access-kkngk\" (UniqueName: \"kubernetes.io/projected/83cfbbfb-a15c-4967-b20c-e102351c6b7b-kube-api-access-kkngk\") pod \"83cfbbfb-a15c-4967-b20c-e102351c6b7b\" (UID: \"83cfbbfb-a15c-4967-b20c-e102351c6b7b\") " Oct 31 05:26:34.871385 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:26:34.871361 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83cfbbfb-a15c-4967-b20c-e102351c6b7b-plugins-config-volume" pod "83cfbbfb-a15c-4967-b20c-e102351c6b7b" (UID: "83cfbbfb-a15c-4967-b20c-e102351c6b7b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:26:34.873172 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:26:34.873148 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83cfbbfb-a15c-4967-b20c-e102351c6b7b-kube-api-access-kkngk" pod "83cfbbfb-a15c-4967-b20c-e102351c6b7b" (UID: "83cfbbfb-a15c-4967-b20c-e102351c6b7b"). InnerVolumeSpecName "kube-api-access-kkngk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:26:34.971960 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:26:34.971929 4376 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83cfbbfb-a15c-4967-b20c-e102351c6b7b-plugins-config-volume\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:26:34.971960 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:26:34.971957 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkngk\" (UniqueName: \"kubernetes.io/projected/83cfbbfb-a15c-4967-b20c-e102351c6b7b-kube-api-access-kkngk\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:26:35.539145 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:26:35.539104 4376 generic.go:356] "Generic (PLEG): container finished" podID="83cfbbfb-a15c-4967-b20c-e102351c6b7b" containerID="b577ebd21405364322a9ee69cce9b1e2fbe1ccd28f3dea8f3b5823010c860e76" exitCode=0 Oct 31 05:26:35.539349 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:26:35.539153 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-cpb5g" Oct 31 05:26:35.539349 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:26:35.539182 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-cpb5g" event={"ID":"83cfbbfb-a15c-4967-b20c-e102351c6b7b","Type":"ContainerDied","Data":"b577ebd21405364322a9ee69cce9b1e2fbe1ccd28f3dea8f3b5823010c860e76"} Oct 31 05:26:35.539349 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:26:35.539243 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-cpb5g" event={"ID":"83cfbbfb-a15c-4967-b20c-e102351c6b7b","Type":"ContainerDied","Data":"c961467db3ca252f01f1dfbe317bfa3f99bc5fec6ab5939c9729e1d0d0884e32"} Oct 31 05:26:35.539349 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:26:35.539269 4376 scope.go:122] "RemoveContainer" containerID="b577ebd21405364322a9ee69cce9b1e2fbe1ccd28f3dea8f3b5823010c860e76" Oct 31 05:26:35.548505 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:26:35.548279 4376 scope.go:122] "RemoveContainer" containerID="b577ebd21405364322a9ee69cce9b1e2fbe1ccd28f3dea8f3b5823010c860e76" Oct 31 05:26:35.548786 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:26:35.548598 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b577ebd21405364322a9ee69cce9b1e2fbe1ccd28f3dea8f3b5823010c860e76\": container with ID starting with b577ebd21405364322a9ee69cce9b1e2fbe1ccd28f3dea8f3b5823010c860e76 not found: ID does not exist" containerID="b577ebd21405364322a9ee69cce9b1e2fbe1ccd28f3dea8f3b5823010c860e76" Oct 31 05:26:35.548786 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:26:35.548634 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b577ebd21405364322a9ee69cce9b1e2fbe1ccd28f3dea8f3b5823010c860e76"} err="failed to get container status \"b577ebd21405364322a9ee69cce9b1e2fbe1ccd28f3dea8f3b5823010c860e76\": rpc error: code = NotFound desc = could not find container \"b577ebd21405364322a9ee69cce9b1e2fbe1ccd28f3dea8f3b5823010c860e76\": container with ID starting with b577ebd21405364322a9ee69cce9b1e2fbe1ccd28f3dea8f3b5823010c860e76 not found: ID does not exist" Oct 31 05:26:35.596688 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:26:35.596665 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-cpb5g"] Oct 31 05:26:35.611228 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:26:35.611201 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-cpb5g"] Oct 31 05:26:36.441209 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:26:36.441176 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83cfbbfb-a15c-4967-b20c-e102351c6b7b" path="/var/lib/kubelet/pods/83cfbbfb-a15c-4967-b20c-e102351c6b7b/volumes" Oct 31 05:26:40.998671 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:26:40.998637 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 05:26:43.531350 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:26:43.531302 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-6ddb9b8755-bwn8k" Oct 31 05:26:43.604202 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:26:43.604169 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-7d68ddf775-nv8tc"] Oct 31 05:26:46.920979 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:26:46.920937 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-zdvn7"] Oct 31 05:26:46.921536 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:26:46.921512 4376 cpu_manager.go:414] "containerMap: removing container" podUID="83cfbbfb-a15c-4967-b20c-e102351c6b7b" containerName="payload-processing" Oct 31 05:26:46.921625 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:26:46.921538 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83cfbbfb-a15c-4967-b20c-e102351c6b7b" containerName="payload-processing" Oct 31 05:26:46.921683 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:26:46.921657 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="83cfbbfb-a15c-4967-b20c-e102351c6b7b" containerName="payload-processing" Oct 31 05:26:46.925917 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:26:46.925894 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bcd7997-zdvn7" Oct 31 05:26:46.942019 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:26:46.941993 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-zdvn7"] Oct 31 05:26:46.954769 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:26:46.954729 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"e2e-rotation-external-model-api-key\"" Oct 31 05:26:46.954769 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:26:46.954744 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Oct 31 05:26:46.954769 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:26:46.954744 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Oct 31 05:26:46.955035 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:26:46.954773 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-cvfj5\"" Oct 31 05:26:46.955035 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:26:46.954784 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Oct 31 05:26:46.958556 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:26:46.958541 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Oct 31 05:26:47.100540 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:26:47.100502 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmzrc\" (UniqueName: \"kubernetes.io/projected/944abc2d-f8d8-48bf-853f-54d6c51e2311-kube-api-access-cmzrc\") pod \"payload-processing-external-model-5bcd7997-zdvn7\" (UID: \"944abc2d-f8d8-48bf-853f-54d6c51e2311\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-zdvn7" Oct 31 05:26:47.100806 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:26:47.100550 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/944abc2d-f8d8-48bf-853f-54d6c51e2311-plugins-config-volume\") pod \"payload-processing-external-model-5bcd7997-zdvn7\" (UID: \"944abc2d-f8d8-48bf-853f-54d6c51e2311\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-zdvn7" Oct 31 05:26:47.100806 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:26:47.100690 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/944abc2d-f8d8-48bf-853f-54d6c51e2311-provider-credentials\") pod \"payload-processing-external-model-5bcd7997-zdvn7\" (UID: \"944abc2d-f8d8-48bf-853f-54d6c51e2311\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-zdvn7" Oct 31 05:26:47.100995 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:26:47.100881 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/944abc2d-f8d8-48bf-853f-54d6c51e2311-routing-overlay-volume\") pod \"payload-processing-external-model-5bcd7997-zdvn7\" (UID: \"944abc2d-f8d8-48bf-853f-54d6c51e2311\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-zdvn7" Oct 31 05:26:47.201696 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:26:47.201608 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/944abc2d-f8d8-48bf-853f-54d6c51e2311-provider-credentials\") pod \"payload-processing-external-model-5bcd7997-zdvn7\" (UID: \"944abc2d-f8d8-48bf-853f-54d6c51e2311\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-zdvn7" Oct 31 05:26:47.201897 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:26:47.201697 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/944abc2d-f8d8-48bf-853f-54d6c51e2311-routing-overlay-volume\") pod \"payload-processing-external-model-5bcd7997-zdvn7\" (UID: \"944abc2d-f8d8-48bf-853f-54d6c51e2311\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-zdvn7" Oct 31 05:26:47.201897 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:26:47.201724 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmzrc\" (UniqueName: \"kubernetes.io/projected/944abc2d-f8d8-48bf-853f-54d6c51e2311-kube-api-access-cmzrc\") pod \"payload-processing-external-model-5bcd7997-zdvn7\" (UID: \"944abc2d-f8d8-48bf-853f-54d6c51e2311\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-zdvn7" Oct 31 05:26:47.201897 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:26:47.201752 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/944abc2d-f8d8-48bf-853f-54d6c51e2311-plugins-config-volume\") pod \"payload-processing-external-model-5bcd7997-zdvn7\" (UID: \"944abc2d-f8d8-48bf-853f-54d6c51e2311\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-zdvn7" Oct 31 05:26:47.202463 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:26:47.202437 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/944abc2d-f8d8-48bf-853f-54d6c51e2311-routing-overlay-volume\") pod \"payload-processing-external-model-5bcd7997-zdvn7\" (UID: \"944abc2d-f8d8-48bf-853f-54d6c51e2311\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-zdvn7" Oct 31 05:26:47.202596 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:26:47.202525 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/944abc2d-f8d8-48bf-853f-54d6c51e2311-plugins-config-volume\") pod \"payload-processing-external-model-5bcd7997-zdvn7\" (UID: \"944abc2d-f8d8-48bf-853f-54d6c51e2311\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-zdvn7" Oct 31 05:26:47.204176 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:26:47.204154 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/944abc2d-f8d8-48bf-853f-54d6c51e2311-provider-credentials\") pod \"payload-processing-external-model-5bcd7997-zdvn7\" (UID: \"944abc2d-f8d8-48bf-853f-54d6c51e2311\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-zdvn7" Oct 31 05:26:47.226961 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:26:47.226929 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmzrc\" (UniqueName: \"kubernetes.io/projected/944abc2d-f8d8-48bf-853f-54d6c51e2311-kube-api-access-cmzrc\") pod \"payload-processing-external-model-5bcd7997-zdvn7\" (UID: \"944abc2d-f8d8-48bf-853f-54d6c51e2311\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-zdvn7" Oct 31 05:26:47.238926 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:26:47.238894 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bcd7997-zdvn7" Oct 31 05:26:47.535784 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:26:47.535756 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-zdvn7"] Oct 31 05:26:47.594439 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:26:47.594398 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bcd7997-zdvn7" event={"ID":"944abc2d-f8d8-48bf-853f-54d6c51e2311","Type":"ContainerStarted","Data":"9de375d959a6251dd5e99558a3164addc86db3076c8dca3cf58f1548d9d3d931"} Oct 31 05:26:48.250153 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:26:48.250116 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 05:26:48.600140 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:26:48.600047 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bcd7997-zdvn7" event={"ID":"944abc2d-f8d8-48bf-853f-54d6c51e2311","Type":"ContainerStarted","Data":"6e00cfa85c9b62d7def25c1f1804bfc43b44306a9c9923789fdfade2e2a71533"} Oct 31 05:26:48.600297 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:26:48.600143 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-5bcd7997-zdvn7" Oct 31 05:26:48.605389 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:26:48.605360 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-7d68ddf775-nv8tc" podUID="4ca3851f-64ee-4d94-9216-425ead298d3f" containerName="payload-processing" containerID="cri-o://851eea68e2289a1245b6ad19f49b937e25971357c1b15f9e7af81bb2047fdbf8" gracePeriod=25 Oct 31 05:26:48.646456 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:26:48.646409 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-5bcd7997-zdvn7" podStartSLOduration=2.646395321 podStartE2EDuration="2.646395321s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 05:26:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 05:26:48.644741113 +0000 UTC m=+4956.359075729" watchObservedRunningTime="2026-10-31 05:26:48.646395321 +0000 UTC m=+4956.360729938" Oct 31 05:26:48.898239 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:26:48.898205 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-7d68ddf775-nv8tc" Oct 31 05:26:48.921428 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:26:48.921391 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ca3851f-64ee-4d94-9216-425ead298d3f-kube-api-access-w6lx7\" (UniqueName: \"kubernetes.io/projected/4ca3851f-64ee-4d94-9216-425ead298d3f-kube-api-access-w6lx7\") pod \"4ca3851f-64ee-4d94-9216-425ead298d3f\" (UID: \"4ca3851f-64ee-4d94-9216-425ead298d3f\") " Oct 31 05:26:48.923719 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:26:48.923686 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ca3851f-64ee-4d94-9216-425ead298d3f-kube-api-access-w6lx7" pod "4ca3851f-64ee-4d94-9216-425ead298d3f" (UID: "4ca3851f-64ee-4d94-9216-425ead298d3f"). InnerVolumeSpecName "kube-api-access-w6lx7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:26:49.022139 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:26:49.022091 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ca3851f-64ee-4d94-9216-425ead298d3f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ca3851f-64ee-4d94-9216-425ead298d3f-plugins-config-volume\") pod \"4ca3851f-64ee-4d94-9216-425ead298d3f\" (UID: \"4ca3851f-64ee-4d94-9216-425ead298d3f\") " Oct 31 05:26:49.022510 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:26:49.022490 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6lx7\" (UniqueName: \"kubernetes.io/projected/4ca3851f-64ee-4d94-9216-425ead298d3f-kube-api-access-w6lx7\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:26:49.022622 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:26:49.022591 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ca3851f-64ee-4d94-9216-425ead298d3f-plugins-config-volume" pod "4ca3851f-64ee-4d94-9216-425ead298d3f" (UID: "4ca3851f-64ee-4d94-9216-425ead298d3f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:26:49.124001 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:26:49.123910 4376 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ca3851f-64ee-4d94-9216-425ead298d3f-plugins-config-volume\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:26:49.633399 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:26:49.628702 4376 generic.go:356] "Generic (PLEG): container finished" podID="4ca3851f-64ee-4d94-9216-425ead298d3f" containerID="851eea68e2289a1245b6ad19f49b937e25971357c1b15f9e7af81bb2047fdbf8" exitCode=0 Oct 31 05:26:49.633399 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:26:49.629127 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-7d68ddf775-nv8tc" Oct 31 05:26:49.633399 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:26:49.629230 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-7d68ddf775-nv8tc" event={"ID":"4ca3851f-64ee-4d94-9216-425ead298d3f","Type":"ContainerDied","Data":"851eea68e2289a1245b6ad19f49b937e25971357c1b15f9e7af81bb2047fdbf8"} Oct 31 05:26:49.633399 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:26:49.629381 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-7d68ddf775-nv8tc" event={"ID":"4ca3851f-64ee-4d94-9216-425ead298d3f","Type":"ContainerDied","Data":"2f2a9aca31a231f7f42493c2823f642274d72a156d24d4bb80a8bae92c9a5cbc"} Oct 31 05:26:49.633399 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:26:49.629513 4376 scope.go:122] "RemoveContainer" containerID="851eea68e2289a1245b6ad19f49b937e25971357c1b15f9e7af81bb2047fdbf8" Oct 31 05:26:49.689572 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:26:49.677081 4376 scope.go:122] "RemoveContainer" containerID="851eea68e2289a1245b6ad19f49b937e25971357c1b15f9e7af81bb2047fdbf8" Oct 31 05:26:49.689572 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:26:49.689352 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"851eea68e2289a1245b6ad19f49b937e25971357c1b15f9e7af81bb2047fdbf8\": container with ID starting with 851eea68e2289a1245b6ad19f49b937e25971357c1b15f9e7af81bb2047fdbf8 not found: ID does not exist" containerID="851eea68e2289a1245b6ad19f49b937e25971357c1b15f9e7af81bb2047fdbf8" Oct 31 05:26:49.689572 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:26:49.689398 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"851eea68e2289a1245b6ad19f49b937e25971357c1b15f9e7af81bb2047fdbf8"} err="failed to get container status \"851eea68e2289a1245b6ad19f49b937e25971357c1b15f9e7af81bb2047fdbf8\": rpc error: code = NotFound desc = could not find container \"851eea68e2289a1245b6ad19f49b937e25971357c1b15f9e7af81bb2047fdbf8\": container with ID starting with 851eea68e2289a1245b6ad19f49b937e25971357c1b15f9e7af81bb2047fdbf8 not found: ID does not exist" Oct 31 05:26:49.765368 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:26:49.759672 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-7d68ddf775-nv8tc"] Oct 31 05:26:49.784675 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:26:49.784593 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-7d68ddf775-nv8tc"] Oct 31 05:26:50.470863 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:26:50.470810 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ca3851f-64ee-4d94-9216-425ead298d3f" path="/var/lib/kubelet/pods/4ca3851f-64ee-4d94-9216-425ead298d3f/volumes" Oct 31 05:26:59.633203 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:26:59.633172 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-5bcd7997-zdvn7" Oct 31 05:27:10.998518 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:27:10.998476 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 05:27:40.999005 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:27:40.998964 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 05:28:11.001474 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:28:11.001427 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 05:28:15.674247 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:28:15.674209 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-zdvn7"] Oct 31 05:28:17.255509 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:28:17.255470 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 05:28:20.675295 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:28:20.675253 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-5bcd7997-zdvn7" podUID="944abc2d-f8d8-48bf-853f-54d6c51e2311" containerName="payload-processing" containerID="cri-o://6e00cfa85c9b62d7def25c1f1804bfc43b44306a9c9923789fdfade2e2a71533" gracePeriod=25 Oct 31 05:28:20.931872 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:28:20.931802 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bcd7997-zdvn7" Oct 31 05:28:21.032493 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:28:21.032458 4376 generic.go:356] "Generic (PLEG): container finished" podID="944abc2d-f8d8-48bf-853f-54d6c51e2311" containerID="6e00cfa85c9b62d7def25c1f1804bfc43b44306a9c9923789fdfade2e2a71533" exitCode=0 Oct 31 05:28:21.032682 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:28:21.032513 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bcd7997-zdvn7" Oct 31 05:28:21.032682 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:28:21.032538 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bcd7997-zdvn7" event={"ID":"944abc2d-f8d8-48bf-853f-54d6c51e2311","Type":"ContainerDied","Data":"6e00cfa85c9b62d7def25c1f1804bfc43b44306a9c9923789fdfade2e2a71533"} Oct 31 05:28:21.032682 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:28:21.032593 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bcd7997-zdvn7" event={"ID":"944abc2d-f8d8-48bf-853f-54d6c51e2311","Type":"ContainerDied","Data":"9de375d959a6251dd5e99558a3164addc86db3076c8dca3cf58f1548d9d3d931"} Oct 31 05:28:21.032682 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:28:21.032617 4376 scope.go:122] "RemoveContainer" containerID="6e00cfa85c9b62d7def25c1f1804bfc43b44306a9c9923789fdfade2e2a71533" Oct 31 05:28:21.035953 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:28:21.035931 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/944abc2d-f8d8-48bf-853f-54d6c51e2311-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/944abc2d-f8d8-48bf-853f-54d6c51e2311-routing-overlay-volume\") pod \"944abc2d-f8d8-48bf-853f-54d6c51e2311\" (UID: \"944abc2d-f8d8-48bf-853f-54d6c51e2311\") " Oct 31 05:28:21.036057 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:28:21.036015 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/944abc2d-f8d8-48bf-853f-54d6c51e2311-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/944abc2d-f8d8-48bf-853f-54d6c51e2311-plugins-config-volume\") pod \"944abc2d-f8d8-48bf-853f-54d6c51e2311\" (UID: \"944abc2d-f8d8-48bf-853f-54d6c51e2311\") " Oct 31 05:28:21.036204 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:28:21.036185 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/944abc2d-f8d8-48bf-853f-54d6c51e2311-provider-credentials\" (UniqueName: \"kubernetes.io/projected/944abc2d-f8d8-48bf-853f-54d6c51e2311-provider-credentials\") pod \"944abc2d-f8d8-48bf-853f-54d6c51e2311\" (UID: \"944abc2d-f8d8-48bf-853f-54d6c51e2311\") " Oct 31 05:28:21.036281 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:28:21.036225 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/944abc2d-f8d8-48bf-853f-54d6c51e2311-kube-api-access-cmzrc\" (UniqueName: \"kubernetes.io/projected/944abc2d-f8d8-48bf-853f-54d6c51e2311-kube-api-access-cmzrc\") pod \"944abc2d-f8d8-48bf-853f-54d6c51e2311\" (UID: \"944abc2d-f8d8-48bf-853f-54d6c51e2311\") " Oct 31 05:28:21.036612 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:28:21.036366 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/944abc2d-f8d8-48bf-853f-54d6c51e2311-routing-overlay-volume" pod "944abc2d-f8d8-48bf-853f-54d6c51e2311" (UID: "944abc2d-f8d8-48bf-853f-54d6c51e2311"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:28:21.036612 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:28:21.036394 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/944abc2d-f8d8-48bf-853f-54d6c51e2311-plugins-config-volume" pod "944abc2d-f8d8-48bf-853f-54d6c51e2311" (UID: "944abc2d-f8d8-48bf-853f-54d6c51e2311"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:28:21.036612 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:28:21.036571 4376 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/944abc2d-f8d8-48bf-853f-54d6c51e2311-routing-overlay-volume\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:28:21.036612 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:28:21.036585 4376 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/944abc2d-f8d8-48bf-853f-54d6c51e2311-plugins-config-volume\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:28:21.038393 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:28:21.038306 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/944abc2d-f8d8-48bf-853f-54d6c51e2311-kube-api-access-cmzrc" pod "944abc2d-f8d8-48bf-853f-54d6c51e2311" (UID: "944abc2d-f8d8-48bf-853f-54d6c51e2311"). InnerVolumeSpecName "kube-api-access-cmzrc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:28:21.038393 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:28:21.038340 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/944abc2d-f8d8-48bf-853f-54d6c51e2311-provider-credentials" pod "944abc2d-f8d8-48bf-853f-54d6c51e2311" (UID: "944abc2d-f8d8-48bf-853f-54d6c51e2311"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:28:21.041231 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:28:21.041213 4376 scope.go:122] "RemoveContainer" containerID="6e00cfa85c9b62d7def25c1f1804bfc43b44306a9c9923789fdfade2e2a71533" Oct 31 05:28:21.041515 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:28:21.041496 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e00cfa85c9b62d7def25c1f1804bfc43b44306a9c9923789fdfade2e2a71533\": container with ID starting with 6e00cfa85c9b62d7def25c1f1804bfc43b44306a9c9923789fdfade2e2a71533 not found: ID does not exist" containerID="6e00cfa85c9b62d7def25c1f1804bfc43b44306a9c9923789fdfade2e2a71533" Oct 31 05:28:21.041585 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:28:21.041522 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e00cfa85c9b62d7def25c1f1804bfc43b44306a9c9923789fdfade2e2a71533"} err="failed to get container status \"6e00cfa85c9b62d7def25c1f1804bfc43b44306a9c9923789fdfade2e2a71533\": rpc error: code = NotFound desc = could not find container \"6e00cfa85c9b62d7def25c1f1804bfc43b44306a9c9923789fdfade2e2a71533\": container with ID starting with 6e00cfa85c9b62d7def25c1f1804bfc43b44306a9c9923789fdfade2e2a71533 not found: ID does not exist" Oct 31 05:28:21.137833 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:28:21.137800 4376 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/944abc2d-f8d8-48bf-853f-54d6c51e2311-provider-credentials\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:28:21.137833 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:28:21.137825 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmzrc\" (UniqueName: \"kubernetes.io/projected/944abc2d-f8d8-48bf-853f-54d6c51e2311-kube-api-access-cmzrc\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:28:21.381844 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:28:21.381812 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-zdvn7"] Oct 31 05:28:21.396233 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:28:21.396204 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-zdvn7"] Oct 31 05:28:22.441554 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:28:22.441521 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="944abc2d-f8d8-48bf-853f-54d6c51e2311" path="/var/lib/kubelet/pods/944abc2d-f8d8-48bf-853f-54d6c51e2311/volumes" Oct 31 05:28:24.539678 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:28:24.539649 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 05:28:40.924552 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:28:40.924506 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 05:28:40.998532 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:28:40.998496 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 05:28:45.725156 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:28:45.725123 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 05:29:01.157054 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:29:01.156977 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 05:29:10.999000 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:29:10.998951 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 05:29:16.947586 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:29:16.947546 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 05:29:40.629898 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:29:40.629860 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 05:29:41.017272 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:29:41.017196 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 05:29:51.118185 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:29:51.118150 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 05:29:55.038634 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:29:55.038597 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 05:30:11.005880 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:30:11.005841 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 05:30:12.853916 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:30:12.853885 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 05:30:18.535236 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:30:18.535203 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 05:30:23.135250 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:30:23.135212 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 05:30:34.845694 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:30:34.845660 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 05:30:39.747588 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:30:39.747554 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 05:30:40.999338 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:30:40.999284 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 05:30:48.330276 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:30:48.330234 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 05:30:54.132026 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:30:54.131993 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 05:31:10.999166 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:31:10.999123 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 05:31:11.938422 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:31:11.938388 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 05:31:14.920807 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:31:14.920766 4376 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-ptb2g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.244:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 05:31:14.921369 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:31:14.920827 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ptb2g" podUID="f3f2eb40-6a45-42f1-9f44-71784631f635" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.244:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 05:31:18.541370 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:31:18.541315 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 05:31:39.109855 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:31:39.109815 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 05:31:41.031104 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:31:41.031064 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 05:31:57.046952 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:31:57.046868 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 05:31:59.932496 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:31:59.932451 4376 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-ptb2g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.244:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 05:31:59.932959 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:31:59.932524 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ptb2g" podUID="f3f2eb40-6a45-42f1-9f44-71784631f635" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.244:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 05:32:07.420398 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:32:07.420356 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 05:32:11.003909 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:32:11.003868 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 05:32:27.761297 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:32:27.761259 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-9632aa-openshift-default-6b55b9f5bfbtp9d"] Oct 31 05:32:27.761880 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:32:27.761858 4376 cpu_manager.go:414] "containerMap: removing container" podUID="944abc2d-f8d8-48bf-853f-54d6c51e2311" containerName="payload-processing" Oct 31 05:32:27.761935 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:32:27.761884 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="944abc2d-f8d8-48bf-853f-54d6c51e2311" containerName="payload-processing" Oct 31 05:32:27.761935 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:32:27.761911 4376 cpu_manager.go:414] "containerMap: removing container" podUID="4ca3851f-64ee-4d94-9216-425ead298d3f" containerName="payload-processing" Oct 31 05:32:27.761935 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:32:27.761921 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ca3851f-64ee-4d94-9216-425ead298d3f" containerName="payload-processing" Oct 31 05:32:27.762254 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:32:27.762027 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="944abc2d-f8d8-48bf-853f-54d6c51e2311" containerName="payload-processing" Oct 31 05:32:27.762254 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:32:27.762040 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ca3851f-64ee-4d94-9216-425ead298d3f" containerName="payload-processing" Oct 31 05:32:27.764830 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:32:27.764809 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-9632aa-openshift-default-6b55b9f5bfbtp9d" Oct 31 05:32:27.786111 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:32:27.786079 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-9632aa-openshift-default-dockercfg-gqhfx\"" Oct 31 05:32:27.788287 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:32:27.788262 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c8f2987-a903-4d11-a43e-2e41de5cb4bd-istio-data\") pod \"e2e-worker-single-main-9632aa-openshift-default-6b55b9f5bfbtp9d\" (UID: \"2c8f2987-a903-4d11-a43e-2e41de5cb4bd\") " pod="openshift-ingress/e2e-worker-single-main-9632aa-openshift-default-6b55b9f5bfbtp9d" Oct 31 05:32:27.788444 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:32:27.788304 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c8f2987-a903-4d11-a43e-2e41de5cb4bd-istio-podinfo\") pod \"e2e-worker-single-main-9632aa-openshift-default-6b55b9f5bfbtp9d\" (UID: \"2c8f2987-a903-4d11-a43e-2e41de5cb4bd\") " pod="openshift-ingress/e2e-worker-single-main-9632aa-openshift-default-6b55b9f5bfbtp9d" Oct 31 05:32:27.788444 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:32:27.788350 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c8f2987-a903-4d11-a43e-2e41de5cb4bd-credential-socket\") pod \"e2e-worker-single-main-9632aa-openshift-default-6b55b9f5bfbtp9d\" (UID: \"2c8f2987-a903-4d11-a43e-2e41de5cb4bd\") " pod="openshift-ingress/e2e-worker-single-main-9632aa-openshift-default-6b55b9f5bfbtp9d" Oct 31 05:32:27.788444 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:32:27.788394 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c8f2987-a903-4d11-a43e-2e41de5cb4bd-istiod-ca-cert\") pod \"e2e-worker-single-main-9632aa-openshift-default-6b55b9f5bfbtp9d\" (UID: \"2c8f2987-a903-4d11-a43e-2e41de5cb4bd\") " pod="openshift-ingress/e2e-worker-single-main-9632aa-openshift-default-6b55b9f5bfbtp9d" Oct 31 05:32:27.788650 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:32:27.788446 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c8f2987-a903-4d11-a43e-2e41de5cb4bd-workload-socket\") pod \"e2e-worker-single-main-9632aa-openshift-default-6b55b9f5bfbtp9d\" (UID: \"2c8f2987-a903-4d11-a43e-2e41de5cb4bd\") " pod="openshift-ingress/e2e-worker-single-main-9632aa-openshift-default-6b55b9f5bfbtp9d" Oct 31 05:32:27.788650 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:32:27.788494 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c8f2987-a903-4d11-a43e-2e41de5cb4bd-workload-certs\") pod \"e2e-worker-single-main-9632aa-openshift-default-6b55b9f5bfbtp9d\" (UID: \"2c8f2987-a903-4d11-a43e-2e41de5cb4bd\") " pod="openshift-ingress/e2e-worker-single-main-9632aa-openshift-default-6b55b9f5bfbtp9d" Oct 31 05:32:27.788795 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:32:27.788725 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqg84\" (UniqueName: \"kubernetes.io/projected/2c8f2987-a903-4d11-a43e-2e41de5cb4bd-kube-api-access-xqg84\") pod \"e2e-worker-single-main-9632aa-openshift-default-6b55b9f5bfbtp9d\" (UID: \"2c8f2987-a903-4d11-a43e-2e41de5cb4bd\") " pod="openshift-ingress/e2e-worker-single-main-9632aa-openshift-default-6b55b9f5bfbtp9d" Oct 31 05:32:27.788795 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:32:27.788774 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c8f2987-a903-4d11-a43e-2e41de5cb4bd-istio-envoy\") pod \"e2e-worker-single-main-9632aa-openshift-default-6b55b9f5bfbtp9d\" (UID: \"2c8f2987-a903-4d11-a43e-2e41de5cb4bd\") " pod="openshift-ingress/e2e-worker-single-main-9632aa-openshift-default-6b55b9f5bfbtp9d" Oct 31 05:32:27.788911 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:32:27.788802 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c8f2987-a903-4d11-a43e-2e41de5cb4bd-istio-token\") pod \"e2e-worker-single-main-9632aa-openshift-default-6b55b9f5bfbtp9d\" (UID: \"2c8f2987-a903-4d11-a43e-2e41de5cb4bd\") " pod="openshift-ingress/e2e-worker-single-main-9632aa-openshift-default-6b55b9f5bfbtp9d" Oct 31 05:32:27.800573 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:32:27.800547 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-9632aa-openshift-default-6b55b9f5bfbtp9d"] Oct 31 05:32:27.889192 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:32:27.889164 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c8f2987-a903-4d11-a43e-2e41de5cb4bd-istiod-ca-cert\") pod \"e2e-worker-single-main-9632aa-openshift-default-6b55b9f5bfbtp9d\" (UID: \"2c8f2987-a903-4d11-a43e-2e41de5cb4bd\") " pod="openshift-ingress/e2e-worker-single-main-9632aa-openshift-default-6b55b9f5bfbtp9d" Oct 31 05:32:27.889384 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:32:27.889208 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c8f2987-a903-4d11-a43e-2e41de5cb4bd-workload-socket\") pod \"e2e-worker-single-main-9632aa-openshift-default-6b55b9f5bfbtp9d\" (UID: \"2c8f2987-a903-4d11-a43e-2e41de5cb4bd\") " pod="openshift-ingress/e2e-worker-single-main-9632aa-openshift-default-6b55b9f5bfbtp9d" Oct 31 05:32:27.889384 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:32:27.889238 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c8f2987-a903-4d11-a43e-2e41de5cb4bd-workload-certs\") pod \"e2e-worker-single-main-9632aa-openshift-default-6b55b9f5bfbtp9d\" (UID: \"2c8f2987-a903-4d11-a43e-2e41de5cb4bd\") " pod="openshift-ingress/e2e-worker-single-main-9632aa-openshift-default-6b55b9f5bfbtp9d" Oct 31 05:32:27.889384 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:32:27.889273 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqg84\" (UniqueName: \"kubernetes.io/projected/2c8f2987-a903-4d11-a43e-2e41de5cb4bd-kube-api-access-xqg84\") pod \"e2e-worker-single-main-9632aa-openshift-default-6b55b9f5bfbtp9d\" (UID: \"2c8f2987-a903-4d11-a43e-2e41de5cb4bd\") " pod="openshift-ingress/e2e-worker-single-main-9632aa-openshift-default-6b55b9f5bfbtp9d" Oct 31 05:32:27.889384 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:32:27.889296 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c8f2987-a903-4d11-a43e-2e41de5cb4bd-istio-envoy\") pod \"e2e-worker-single-main-9632aa-openshift-default-6b55b9f5bfbtp9d\" (UID: \"2c8f2987-a903-4d11-a43e-2e41de5cb4bd\") " pod="openshift-ingress/e2e-worker-single-main-9632aa-openshift-default-6b55b9f5bfbtp9d" Oct 31 05:32:27.889384 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:32:27.889320 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c8f2987-a903-4d11-a43e-2e41de5cb4bd-istio-token\") pod \"e2e-worker-single-main-9632aa-openshift-default-6b55b9f5bfbtp9d\" (UID: \"2c8f2987-a903-4d11-a43e-2e41de5cb4bd\") " pod="openshift-ingress/e2e-worker-single-main-9632aa-openshift-default-6b55b9f5bfbtp9d" Oct 31 05:32:27.889679 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:32:27.889507 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c8f2987-a903-4d11-a43e-2e41de5cb4bd-istio-data\") pod \"e2e-worker-single-main-9632aa-openshift-default-6b55b9f5bfbtp9d\" (UID: \"2c8f2987-a903-4d11-a43e-2e41de5cb4bd\") " pod="openshift-ingress/e2e-worker-single-main-9632aa-openshift-default-6b55b9f5bfbtp9d" Oct 31 05:32:27.889679 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:32:27.889560 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c8f2987-a903-4d11-a43e-2e41de5cb4bd-istio-podinfo\") pod \"e2e-worker-single-main-9632aa-openshift-default-6b55b9f5bfbtp9d\" (UID: \"2c8f2987-a903-4d11-a43e-2e41de5cb4bd\") " pod="openshift-ingress/e2e-worker-single-main-9632aa-openshift-default-6b55b9f5bfbtp9d" Oct 31 05:32:27.889679 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:32:27.889599 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c8f2987-a903-4d11-a43e-2e41de5cb4bd-credential-socket\") pod \"e2e-worker-single-main-9632aa-openshift-default-6b55b9f5bfbtp9d\" (UID: \"2c8f2987-a903-4d11-a43e-2e41de5cb4bd\") " pod="openshift-ingress/e2e-worker-single-main-9632aa-openshift-default-6b55b9f5bfbtp9d" Oct 31 05:32:27.889844 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:32:27.889743 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c8f2987-a903-4d11-a43e-2e41de5cb4bd-workload-socket\") pod \"e2e-worker-single-main-9632aa-openshift-default-6b55b9f5bfbtp9d\" (UID: \"2c8f2987-a903-4d11-a43e-2e41de5cb4bd\") " pod="openshift-ingress/e2e-worker-single-main-9632aa-openshift-default-6b55b9f5bfbtp9d" Oct 31 05:32:27.889844 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:32:27.889767 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c8f2987-a903-4d11-a43e-2e41de5cb4bd-workload-certs\") pod \"e2e-worker-single-main-9632aa-openshift-default-6b55b9f5bfbtp9d\" (UID: \"2c8f2987-a903-4d11-a43e-2e41de5cb4bd\") " pod="openshift-ingress/e2e-worker-single-main-9632aa-openshift-default-6b55b9f5bfbtp9d" Oct 31 05:32:27.889967 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:32:27.889849 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c8f2987-a903-4d11-a43e-2e41de5cb4bd-istio-data\") pod \"e2e-worker-single-main-9632aa-openshift-default-6b55b9f5bfbtp9d\" (UID: \"2c8f2987-a903-4d11-a43e-2e41de5cb4bd\") " pod="openshift-ingress/e2e-worker-single-main-9632aa-openshift-default-6b55b9f5bfbtp9d" Oct 31 05:32:27.890031 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:32:27.889981 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c8f2987-a903-4d11-a43e-2e41de5cb4bd-credential-socket\") pod \"e2e-worker-single-main-9632aa-openshift-default-6b55b9f5bfbtp9d\" (UID: \"2c8f2987-a903-4d11-a43e-2e41de5cb4bd\") " pod="openshift-ingress/e2e-worker-single-main-9632aa-openshift-default-6b55b9f5bfbtp9d" Oct 31 05:32:27.890324 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:32:27.890301 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c8f2987-a903-4d11-a43e-2e41de5cb4bd-istiod-ca-cert\") pod \"e2e-worker-single-main-9632aa-openshift-default-6b55b9f5bfbtp9d\" (UID: \"2c8f2987-a903-4d11-a43e-2e41de5cb4bd\") " pod="openshift-ingress/e2e-worker-single-main-9632aa-openshift-default-6b55b9f5bfbtp9d" Oct 31 05:32:27.891633 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:32:27.891612 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c8f2987-a903-4d11-a43e-2e41de5cb4bd-istio-envoy\") pod \"e2e-worker-single-main-9632aa-openshift-default-6b55b9f5bfbtp9d\" (UID: \"2c8f2987-a903-4d11-a43e-2e41de5cb4bd\") " pod="openshift-ingress/e2e-worker-single-main-9632aa-openshift-default-6b55b9f5bfbtp9d" Oct 31 05:32:27.891960 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:32:27.891938 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c8f2987-a903-4d11-a43e-2e41de5cb4bd-istio-podinfo\") pod \"e2e-worker-single-main-9632aa-openshift-default-6b55b9f5bfbtp9d\" (UID: \"2c8f2987-a903-4d11-a43e-2e41de5cb4bd\") " pod="openshift-ingress/e2e-worker-single-main-9632aa-openshift-default-6b55b9f5bfbtp9d" Oct 31 05:32:27.914524 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:32:27.914503 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqg84\" (UniqueName: \"kubernetes.io/projected/2c8f2987-a903-4d11-a43e-2e41de5cb4bd-kube-api-access-xqg84\") pod \"e2e-worker-single-main-9632aa-openshift-default-6b55b9f5bfbtp9d\" (UID: \"2c8f2987-a903-4d11-a43e-2e41de5cb4bd\") " pod="openshift-ingress/e2e-worker-single-main-9632aa-openshift-default-6b55b9f5bfbtp9d" Oct 31 05:32:27.914612 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:32:27.914550 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c8f2987-a903-4d11-a43e-2e41de5cb4bd-istio-token\") pod \"e2e-worker-single-main-9632aa-openshift-default-6b55b9f5bfbtp9d\" (UID: \"2c8f2987-a903-4d11-a43e-2e41de5cb4bd\") " pod="openshift-ingress/e2e-worker-single-main-9632aa-openshift-default-6b55b9f5bfbtp9d" Oct 31 05:32:28.078238 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:32:28.078159 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-9632aa-openshift-default-6b55b9f5bfbtp9d" Oct 31 05:32:28.391024 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:32:28.390986 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-9632aa-openshift-default-6b55b9f5bfbtp9d"] Oct 31 05:32:28.392760 ip-10-0-134-155 kubenswrapper[4376]: W1031 05:32:28.392731 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c8f2987_a903_4d11_a43e_2e41de5cb4bd.slice/crio-df01ae8af7ca8e33f9f3cd9c57856f510b7c5fdce56b72be029dc7207528d73b WatchSource:0}: Error finding container df01ae8af7ca8e33f9f3cd9c57856f510b7c5fdce56b72be029dc7207528d73b: Status 404 returned error can't find the container with id df01ae8af7ca8e33f9f3cd9c57856f510b7c5fdce56b72be029dc7207528d73b Oct 31 05:32:28.394773 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:32:28.394740 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 05:32:28.394839 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:32:28.394807 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 05:32:28.394878 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:32:28.394838 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 05:32:29.336906 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:32:29.336863 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-9632aa-openshift-default-6b55b9f5bfbtp9d" event={"ID":"2c8f2987-a903-4d11-a43e-2e41de5cb4bd","Type":"ContainerStarted","Data":"7df32700907c8677532c73b5fc67b972e2365b9f99821f2d41c1bff82c2303ac"} Oct 31 05:32:29.337307 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:32:29.336925 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-9632aa-openshift-default-6b55b9f5bfbtp9d" event={"ID":"2c8f2987-a903-4d11-a43e-2e41de5cb4bd","Type":"ContainerStarted","Data":"df01ae8af7ca8e33f9f3cd9c57856f510b7c5fdce56b72be029dc7207528d73b"} Oct 31 05:32:29.415358 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:32:29.415291 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-9632aa-openshift-default-6b55b9f5bfbtp9d" podStartSLOduration=2.415278049 podStartE2EDuration="2.415278049s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 05:32:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 05:32:29.400590811 +0000 UTC m=+5297.114925447" watchObservedRunningTime="2026-10-31 05:32:29.415278049 +0000 UTC m=+5297.129612666" Oct 31 05:32:30.079239 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:32:30.079205 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-9632aa-openshift-default-6b55b9f5bfbtp9d" Oct 31 05:32:30.084154 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:32:30.084130 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-9632aa-openshift-default-6b55b9f5bfbtp9d" Oct 31 05:32:30.341911 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:32:30.341834 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-9632aa-openshift-default-6b55b9f5bfbtp9d" Oct 31 05:32:30.342859 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:32:30.342841 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-9632aa-openshift-default-6b55b9f5bfbtp9d" Oct 31 05:32:33.256351 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:32:33.256294 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9632aa-77765f8fdb-bj5vr"] Oct 31 05:32:33.260202 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:32:33.260178 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9632aa-77765f8fdb-bj5vr" Oct 31 05:32:33.279934 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:32:33.279897 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-9632aa-serving-cert\"" Oct 31 05:32:33.297390 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:32:33.297365 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9632aa-77765f8fdb-bj5vr"] Oct 31 05:32:33.345757 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:32:33.345722 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ecd0b08f-c55a-4c61-96d7-0ce0f7522952-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-9632aa-77765f8fdb-bj5vr\" (UID: \"ecd0b08f-c55a-4c61-96d7-0ce0f7522952\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9632aa-77765f8fdb-bj5vr" Oct 31 05:32:33.345911 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:32:33.345848 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzs9c\" (UniqueName: \"kubernetes.io/projected/ecd0b08f-c55a-4c61-96d7-0ce0f7522952-kube-api-access-dzs9c\") pod \"maas-api-e2e-worker-single-main-9632aa-77765f8fdb-bj5vr\" (UID: \"ecd0b08f-c55a-4c61-96d7-0ce0f7522952\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9632aa-77765f8fdb-bj5vr" Oct 31 05:32:33.345911 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:32:33.345890 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ecd0b08f-c55a-4c61-96d7-0ce0f7522952-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-9632aa-77765f8fdb-bj5vr\" (UID: \"ecd0b08f-c55a-4c61-96d7-0ce0f7522952\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9632aa-77765f8fdb-bj5vr" Oct 31 05:32:33.345996 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:32:33.345916 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ecd0b08f-c55a-4c61-96d7-0ce0f7522952-metrics-certs\") pod \"maas-api-e2e-worker-single-main-9632aa-77765f8fdb-bj5vr\" (UID: \"ecd0b08f-c55a-4c61-96d7-0ce0f7522952\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9632aa-77765f8fdb-bj5vr" Oct 31 05:32:33.446701 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:32:33.446669 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ecd0b08f-c55a-4c61-96d7-0ce0f7522952-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-9632aa-77765f8fdb-bj5vr\" (UID: \"ecd0b08f-c55a-4c61-96d7-0ce0f7522952\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9632aa-77765f8fdb-bj5vr" Oct 31 05:32:33.446862 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:32:33.446729 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzs9c\" (UniqueName: \"kubernetes.io/projected/ecd0b08f-c55a-4c61-96d7-0ce0f7522952-kube-api-access-dzs9c\") pod \"maas-api-e2e-worker-single-main-9632aa-77765f8fdb-bj5vr\" (UID: \"ecd0b08f-c55a-4c61-96d7-0ce0f7522952\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9632aa-77765f8fdb-bj5vr" Oct 31 05:32:33.446862 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:32:33.446764 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ecd0b08f-c55a-4c61-96d7-0ce0f7522952-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-9632aa-77765f8fdb-bj5vr\" (UID: \"ecd0b08f-c55a-4c61-96d7-0ce0f7522952\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9632aa-77765f8fdb-bj5vr" Oct 31 05:32:33.446862 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:32:33.446794 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ecd0b08f-c55a-4c61-96d7-0ce0f7522952-metrics-certs\") pod \"maas-api-e2e-worker-single-main-9632aa-77765f8fdb-bj5vr\" (UID: \"ecd0b08f-c55a-4c61-96d7-0ce0f7522952\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9632aa-77765f8fdb-bj5vr" Oct 31 05:32:33.447499 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:32:33.447468 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ecd0b08f-c55a-4c61-96d7-0ce0f7522952-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-9632aa-77765f8fdb-bj5vr\" (UID: \"ecd0b08f-c55a-4c61-96d7-0ce0f7522952\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9632aa-77765f8fdb-bj5vr" Oct 31 05:32:33.449141 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:32:33.449114 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ecd0b08f-c55a-4c61-96d7-0ce0f7522952-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-9632aa-77765f8fdb-bj5vr\" (UID: \"ecd0b08f-c55a-4c61-96d7-0ce0f7522952\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9632aa-77765f8fdb-bj5vr" Oct 31 05:32:33.449258 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:32:33.449153 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ecd0b08f-c55a-4c61-96d7-0ce0f7522952-metrics-certs\") pod \"maas-api-e2e-worker-single-main-9632aa-77765f8fdb-bj5vr\" (UID: \"ecd0b08f-c55a-4c61-96d7-0ce0f7522952\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9632aa-77765f8fdb-bj5vr" Oct 31 05:32:33.501666 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:32:33.501632 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzs9c\" (UniqueName: \"kubernetes.io/projected/ecd0b08f-c55a-4c61-96d7-0ce0f7522952-kube-api-access-dzs9c\") pod \"maas-api-e2e-worker-single-main-9632aa-77765f8fdb-bj5vr\" (UID: \"ecd0b08f-c55a-4c61-96d7-0ce0f7522952\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9632aa-77765f8fdb-bj5vr" Oct 31 05:32:33.574311 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:32:33.574242 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9632aa-77765f8fdb-bj5vr" Oct 31 05:32:33.876981 ip-10-0-134-155 kubenswrapper[4376]: W1031 05:32:33.876942 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecd0b08f_c55a_4c61_96d7_0ce0f7522952.slice/crio-f55121cd88b39799e356097344eadee4fa56e9960199f696c15f0b3b63b6ab5f WatchSource:0}: Error finding container f55121cd88b39799e356097344eadee4fa56e9960199f696c15f0b3b63b6ab5f: Status 404 returned error can't find the container with id f55121cd88b39799e356097344eadee4fa56e9960199f696c15f0b3b63b6ab5f Oct 31 05:32:33.877256 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:32:33.877235 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9632aa-77765f8fdb-bj5vr"] Oct 31 05:32:33.878174 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:32:33.878157 4376 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 05:32:34.360792 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:32:34.360718 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9632aa-77765f8fdb-bj5vr" event={"ID":"ecd0b08f-c55a-4c61-96d7-0ce0f7522952","Type":"ContainerStarted","Data":"f55121cd88b39799e356097344eadee4fa56e9960199f696c15f0b3b63b6ab5f"} Oct 31 05:32:35.366119 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:32:35.366080 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9632aa-77765f8fdb-bj5vr" event={"ID":"ecd0b08f-c55a-4c61-96d7-0ce0f7522952","Type":"ContainerStarted","Data":"8f4097939f86886c83f224dbb09de7ef4365374224ba063da8ffb7d8ca9d394b"} Oct 31 05:32:35.366526 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:32:35.366216 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9632aa-77765f8fdb-bj5vr" Oct 31 05:32:35.412512 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:32:35.412460 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9632aa-77765f8fdb-bj5vr" podStartSLOduration=1.9148245419999999 podStartE2EDuration="2.412445429s" totalImagesPullingTime="497.620887ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 05:32:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 05:32:35.412099146 +0000 UTC m=+5303.126433763" watchObservedRunningTime="2026-10-31 05:32:35.412445429 +0000 UTC m=+5303.126780046" Oct 31 05:32:37.146574 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:32:37.146536 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 05:32:40.999378 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:32:40.999338 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 05:32:41.376489 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:32:41.376459 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9632aa-77765f8fdb-bj5vr" Oct 31 05:32:46.384449 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:32:46.384414 4376 patch_prober.go:29] interesting pod/e2e-worker-single-main-9632aa-openshift-default-6b55b9f5bfbtp9d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.252:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 05:32:46.384895 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:32:46.384468 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-9632aa-openshift-default-6b55b9f5bfbtp9d" podUID="2c8f2987-a903-4d11-a43e-2e41de5cb4bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.252:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 05:32:47.418802 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:32:47.418766 4376 generic.go:356] "Generic (PLEG): container finished" podID="2c8f2987-a903-4d11-a43e-2e41de5cb4bd" containerID="7df32700907c8677532c73b5fc67b972e2365b9f99821f2d41c1bff82c2303ac" exitCode=0 Oct 31 05:32:47.419298 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:32:47.418840 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-9632aa-openshift-default-6b55b9f5bfbtp9d" event={"ID":"2c8f2987-a903-4d11-a43e-2e41de5cb4bd","Type":"ContainerDied","Data":"7df32700907c8677532c73b5fc67b972e2365b9f99821f2d41c1bff82c2303ac"} Oct 31 05:32:47.419298 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:32:47.419222 4376 scope.go:122] "RemoveContainer" containerID="7df32700907c8677532c73b5fc67b972e2365b9f99821f2d41c1bff82c2303ac" Oct 31 05:32:47.420358 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:32:47.420312 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 05:32:47.420463 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:32:47.420413 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 05:32:47.420538 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:32:47.420461 4376 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 05:32:48.426644 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:32:48.426616 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-9632aa-openshift-default-6b55b9f5bfbtp9d" event={"ID":"2c8f2987-a903-4d11-a43e-2e41de5cb4bd","Type":"ContainerStarted","Data":"4932a3f3fc18724711eb17edf5d659ed55db8a66a2c8b9bc3ff482544ef6d61c"} Oct 31 05:32:49.078885 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:32:49.078848 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-9632aa-openshift-default-6b55b9f5bfbtp9d" Oct 31 05:32:49.079069 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:32:49.079026 4376 patch_prober.go:29] interesting pod/e2e-worker-single-main-9632aa-openshift-default-6b55b9f5bfbtp9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.252:15021/healthz/ready\": dial tcp 10.133.0.252:15021: connect: connection refused" start-of-body= Oct 31 05:32:49.079121 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:32:49.079085 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-9632aa-openshift-default-6b55b9f5bfbtp9d" podUID="2c8f2987-a903-4d11-a43e-2e41de5cb4bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.252:15021/healthz/ready\": dial tcp 10.133.0.252:15021: connect: connection refused" Oct 31 05:32:50.078635 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:32:50.078605 4376 patch_prober.go:29] interesting pod/e2e-worker-single-main-9632aa-openshift-default-6b55b9f5bfbtp9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.252:15021/healthz/ready\": dial tcp 10.133.0.252:15021: connect: connection refused" start-of-body= Oct 31 05:32:50.079023 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:32:50.078655 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-9632aa-openshift-default-6b55b9f5bfbtp9d" podUID="2c8f2987-a903-4d11-a43e-2e41de5cb4bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.252:15021/healthz/ready\": dial tcp 10.133.0.252:15021: connect: connection refused" Oct 31 05:32:51.078923 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:32:51.078890 4376 patch_prober.go:29] interesting pod/e2e-worker-single-main-9632aa-openshift-default-6b55b9f5bfbtp9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.252:15021/healthz/ready\": dial tcp 10.133.0.252:15021: connect: connection refused" start-of-body= Oct 31 05:32:51.079351 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:32:51.078938 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-9632aa-openshift-default-6b55b9f5bfbtp9d" podUID="2c8f2987-a903-4d11-a43e-2e41de5cb4bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.252:15021/healthz/ready\": dial tcp 10.133.0.252:15021: connect: connection refused" Oct 31 05:32:51.664861 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:32:51.662488 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 05:32:52.078906 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:32:52.078831 4376 patch_prober.go:29] interesting pod/e2e-worker-single-main-9632aa-openshift-default-6b55b9f5bfbtp9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.252:15021/healthz/ready\": dial tcp 10.133.0.252:15021: connect: connection refused" start-of-body= Oct 31 05:32:52.078906 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:32:52.078882 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-9632aa-openshift-default-6b55b9f5bfbtp9d" podUID="2c8f2987-a903-4d11-a43e-2e41de5cb4bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.252:15021/healthz/ready\": dial tcp 10.133.0.252:15021: connect: connection refused" Oct 31 05:32:53.078898 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:32:53.078855 4376 patch_prober.go:29] interesting pod/e2e-worker-single-main-9632aa-openshift-default-6b55b9f5bfbtp9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.252:15021/healthz/ready\": dial tcp 10.133.0.252:15021: connect: connection refused" start-of-body= Oct 31 05:32:53.079081 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:32:53.078917 4376 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-9632aa-openshift-default-6b55b9f5bfbtp9d" podUID="2c8f2987-a903-4d11-a43e-2e41de5cb4bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.252:15021/healthz/ready\": dial tcp 10.133.0.252:15021: connect: connection refused" Oct 31 05:32:54.082141 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:32:54.082107 4376 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-9632aa-openshift-default-6b55b9f5bfbtp9d" Oct 31 05:32:54.082557 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:32:54.082537 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-9632aa-openshift-default-6b55b9f5bfbtp9d" Oct 31 05:32:54.083100 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:32:54.083083 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-9632aa-openshift-default-6b55b9f5bfbtp9d" Oct 31 05:33:07.341165 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:33:07.341126 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 05:33:10.998867 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:33:10.998827 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 05:33:33.745220 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:33:33.745178 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 05:33:40.060934 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:33:40.060900 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9632aa-77765f8fdb-bj5vr"] Oct 31 05:33:40.061516 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:33:40.061115 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9632aa-77765f8fdb-bj5vr" podUID="ecd0b08f-c55a-4c61-96d7-0ce0f7522952" containerName="maas-api" containerID="cri-o://8f4097939f86886c83f224dbb09de7ef4365374224ba063da8ffb7d8ca9d394b" gracePeriod=30 Oct 31 05:33:40.557469 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:33:40.557444 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9632aa-77765f8fdb-bj5vr" Oct 31 05:33:40.674406 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:33:40.674374 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ecd0b08f-c55a-4c61-96d7-0ce0f7522952-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ecd0b08f-c55a-4c61-96d7-0ce0f7522952-maas-api-tls\") pod \"ecd0b08f-c55a-4c61-96d7-0ce0f7522952\" (UID: \"ecd0b08f-c55a-4c61-96d7-0ce0f7522952\") " Oct 31 05:33:40.674639 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:33:40.674433 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecd0b08f-c55a-4c61-96d7-0ce0f7522952-kube-api-access-dzs9c\" (UniqueName: \"kubernetes.io/projected/ecd0b08f-c55a-4c61-96d7-0ce0f7522952-kube-api-access-dzs9c\") pod \"ecd0b08f-c55a-4c61-96d7-0ce0f7522952\" (UID: \"ecd0b08f-c55a-4c61-96d7-0ce0f7522952\") " Oct 31 05:33:40.674639 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:33:40.674475 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ecd0b08f-c55a-4c61-96d7-0ce0f7522952-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ecd0b08f-c55a-4c61-96d7-0ce0f7522952-openshift-service-ca\") pod \"ecd0b08f-c55a-4c61-96d7-0ce0f7522952\" (UID: \"ecd0b08f-c55a-4c61-96d7-0ce0f7522952\") " Oct 31 05:33:40.674639 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:33:40.674549 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ecd0b08f-c55a-4c61-96d7-0ce0f7522952-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ecd0b08f-c55a-4c61-96d7-0ce0f7522952-metrics-certs\") pod \"ecd0b08f-c55a-4c61-96d7-0ce0f7522952\" (UID: \"ecd0b08f-c55a-4c61-96d7-0ce0f7522952\") " Oct 31 05:33:40.674867 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:33:40.674839 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecd0b08f-c55a-4c61-96d7-0ce0f7522952-openshift-service-ca" pod "ecd0b08f-c55a-4c61-96d7-0ce0f7522952" (UID: "ecd0b08f-c55a-4c61-96d7-0ce0f7522952"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:33:40.674976 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:33:40.674958 4376 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ecd0b08f-c55a-4c61-96d7-0ce0f7522952-openshift-service-ca\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:33:40.677274 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:33:40.677248 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecd0b08f-c55a-4c61-96d7-0ce0f7522952-kube-api-access-dzs9c" pod "ecd0b08f-c55a-4c61-96d7-0ce0f7522952" (UID: "ecd0b08f-c55a-4c61-96d7-0ce0f7522952"). InnerVolumeSpecName "kube-api-access-dzs9c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:33:40.677274 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:33:40.677261 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ecd0b08f-c55a-4c61-96d7-0ce0f7522952-metrics-certs" pod "ecd0b08f-c55a-4c61-96d7-0ce0f7522952" (UID: "ecd0b08f-c55a-4c61-96d7-0ce0f7522952"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 05:33:40.677940 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:33:40.677920 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ecd0b08f-c55a-4c61-96d7-0ce0f7522952-maas-api-tls" pod "ecd0b08f-c55a-4c61-96d7-0ce0f7522952" (UID: "ecd0b08f-c55a-4c61-96d7-0ce0f7522952"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 05:33:40.679617 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:33:40.679594 4376 generic.go:356] "Generic (PLEG): container finished" podID="ecd0b08f-c55a-4c61-96d7-0ce0f7522952" containerID="8f4097939f86886c83f224dbb09de7ef4365374224ba063da8ffb7d8ca9d394b" exitCode=0 Oct 31 05:33:40.679745 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:33:40.679654 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9632aa-77765f8fdb-bj5vr" Oct 31 05:33:40.679745 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:33:40.679673 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9632aa-77765f8fdb-bj5vr" event={"ID":"ecd0b08f-c55a-4c61-96d7-0ce0f7522952","Type":"ContainerDied","Data":"8f4097939f86886c83f224dbb09de7ef4365374224ba063da8ffb7d8ca9d394b"} Oct 31 05:33:40.679745 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:33:40.679736 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9632aa-77765f8fdb-bj5vr" event={"ID":"ecd0b08f-c55a-4c61-96d7-0ce0f7522952","Type":"ContainerDied","Data":"f55121cd88b39799e356097344eadee4fa56e9960199f696c15f0b3b63b6ab5f"} Oct 31 05:33:40.679866 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:33:40.679760 4376 scope.go:122] "RemoveContainer" containerID="8f4097939f86886c83f224dbb09de7ef4365374224ba063da8ffb7d8ca9d394b" Oct 31 05:33:40.701057 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:33:40.701039 4376 scope.go:122] "RemoveContainer" containerID="8f4097939f86886c83f224dbb09de7ef4365374224ba063da8ffb7d8ca9d394b" Oct 31 05:33:40.701354 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:33:40.701316 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f4097939f86886c83f224dbb09de7ef4365374224ba063da8ffb7d8ca9d394b\": container with ID starting with 8f4097939f86886c83f224dbb09de7ef4365374224ba063da8ffb7d8ca9d394b not found: ID does not exist" containerID="8f4097939f86886c83f224dbb09de7ef4365374224ba063da8ffb7d8ca9d394b" Oct 31 05:33:40.701463 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:33:40.701361 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f4097939f86886c83f224dbb09de7ef4365374224ba063da8ffb7d8ca9d394b"} err="failed to get container status \"8f4097939f86886c83f224dbb09de7ef4365374224ba063da8ffb7d8ca9d394b\": rpc error: code = NotFound desc = could not find container \"8f4097939f86886c83f224dbb09de7ef4365374224ba063da8ffb7d8ca9d394b\": container with ID starting with 8f4097939f86886c83f224dbb09de7ef4365374224ba063da8ffb7d8ca9d394b not found: ID does not exist" Oct 31 05:33:40.738876 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:33:40.738842 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9632aa-77765f8fdb-bj5vr"] Oct 31 05:33:40.744606 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:33:40.744579 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9632aa-77765f8fdb-bj5vr"] Oct 31 05:33:40.776138 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:33:40.776106 4376 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ecd0b08f-c55a-4c61-96d7-0ce0f7522952-maas-api-tls\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:33:40.776138 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:33:40.776132 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dzs9c\" (UniqueName: \"kubernetes.io/projected/ecd0b08f-c55a-4c61-96d7-0ce0f7522952-kube-api-access-dzs9c\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:33:40.776138 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:33:40.776144 4376 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ecd0b08f-c55a-4c61-96d7-0ce0f7522952-metrics-certs\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:33:41.004954 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:33:41.004866 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 05:33:42.441638 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:33:42.441605 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ecd0b08f-c55a-4c61-96d7-0ce0f7522952" path="/var/lib/kubelet/pods/ecd0b08f-c55a-4c61-96d7-0ce0f7522952/volumes" Oct 31 05:33:42.571423 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:33:42.571387 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 05:34:11.001686 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:34:11.001645 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 05:34:12.062088 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:34:12.062053 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9632aa-77765f8fdb-8tgst"] Oct 31 05:34:12.062501 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:34:12.062425 4376 cpu_manager.go:414] "containerMap: removing container" podUID="ecd0b08f-c55a-4c61-96d7-0ce0f7522952" containerName="maas-api" Oct 31 05:34:12.062501 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:34:12.062437 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecd0b08f-c55a-4c61-96d7-0ce0f7522952" containerName="maas-api" Oct 31 05:34:12.062577 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:34:12.062517 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecd0b08f-c55a-4c61-96d7-0ce0f7522952" containerName="maas-api" Oct 31 05:34:12.064845 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:34:12.064823 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9632aa-77765f8fdb-8tgst" Oct 31 05:34:12.097108 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:34:12.097069 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-9632aa-serving-cert\"" Oct 31 05:34:12.099015 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:34:12.098984 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9632aa-77765f8fdb-8tgst"] Oct 31 05:34:12.177049 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:34:12.177010 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa47e13c-f842-405b-a989-046436f77080-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-9632aa-77765f8fdb-8tgst\" (UID: \"aa47e13c-f842-405b-a989-046436f77080\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9632aa-77765f8fdb-8tgst" Oct 31 05:34:12.177049 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:34:12.177050 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79x99\" (UniqueName: \"kubernetes.io/projected/aa47e13c-f842-405b-a989-046436f77080-kube-api-access-79x99\") pod \"maas-api-e2e-worker-single-main-9632aa-77765f8fdb-8tgst\" (UID: \"aa47e13c-f842-405b-a989-046436f77080\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9632aa-77765f8fdb-8tgst" Oct 31 05:34:12.177291 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:34:12.177088 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa47e13c-f842-405b-a989-046436f77080-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-9632aa-77765f8fdb-8tgst\" (UID: \"aa47e13c-f842-405b-a989-046436f77080\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9632aa-77765f8fdb-8tgst" Oct 31 05:34:12.177291 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:34:12.177182 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa47e13c-f842-405b-a989-046436f77080-metrics-certs\") pod \"maas-api-e2e-worker-single-main-9632aa-77765f8fdb-8tgst\" (UID: \"aa47e13c-f842-405b-a989-046436f77080\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9632aa-77765f8fdb-8tgst" Oct 31 05:34:12.277780 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:34:12.277742 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa47e13c-f842-405b-a989-046436f77080-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-9632aa-77765f8fdb-8tgst\" (UID: \"aa47e13c-f842-405b-a989-046436f77080\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9632aa-77765f8fdb-8tgst" Oct 31 05:34:12.277780 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:34:12.277782 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79x99\" (UniqueName: \"kubernetes.io/projected/aa47e13c-f842-405b-a989-046436f77080-kube-api-access-79x99\") pod \"maas-api-e2e-worker-single-main-9632aa-77765f8fdb-8tgst\" (UID: \"aa47e13c-f842-405b-a989-046436f77080\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9632aa-77765f8fdb-8tgst" Oct 31 05:34:12.278019 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:34:12.277821 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa47e13c-f842-405b-a989-046436f77080-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-9632aa-77765f8fdb-8tgst\" (UID: \"aa47e13c-f842-405b-a989-046436f77080\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9632aa-77765f8fdb-8tgst" Oct 31 05:34:12.278019 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:34:12.277848 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa47e13c-f842-405b-a989-046436f77080-metrics-certs\") pod \"maas-api-e2e-worker-single-main-9632aa-77765f8fdb-8tgst\" (UID: \"aa47e13c-f842-405b-a989-046436f77080\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9632aa-77765f8fdb-8tgst" Oct 31 05:34:12.278019 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:34:12.277917 4376 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9632aa-serving-cert: secret "maas-api-e2e-worker-single-main-9632aa-serving-cert" not found Oct 31 05:34:12.278019 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:34:12.278014 4376 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aa47e13c-f842-405b-a989-046436f77080-maas-api-tls podName:aa47e13c-f842-405b-a989-046436f77080 nodeName:}" failed. No retries permitted until 2026-10-31 05:34:12.777992267 +0000 UTC m=+5400.492326861 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/aa47e13c-f842-405b-a989-046436f77080-maas-api-tls") pod "maas-api-e2e-worker-single-main-9632aa-77765f8fdb-8tgst" (UID: "aa47e13c-f842-405b-a989-046436f77080") : secret "maas-api-e2e-worker-single-main-9632aa-serving-cert" not found Oct 31 05:34:12.278589 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:34:12.278569 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa47e13c-f842-405b-a989-046436f77080-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-9632aa-77765f8fdb-8tgst\" (UID: \"aa47e13c-f842-405b-a989-046436f77080\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9632aa-77765f8fdb-8tgst" Oct 31 05:34:12.280235 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:34:12.280213 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa47e13c-f842-405b-a989-046436f77080-metrics-certs\") pod \"maas-api-e2e-worker-single-main-9632aa-77765f8fdb-8tgst\" (UID: \"aa47e13c-f842-405b-a989-046436f77080\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9632aa-77765f8fdb-8tgst" Oct 31 05:34:12.306624 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:34:12.306593 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79x99\" (UniqueName: \"kubernetes.io/projected/aa47e13c-f842-405b-a989-046436f77080-kube-api-access-79x99\") pod \"maas-api-e2e-worker-single-main-9632aa-77765f8fdb-8tgst\" (UID: \"aa47e13c-f842-405b-a989-046436f77080\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9632aa-77765f8fdb-8tgst" Oct 31 05:34:12.796821 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:34:12.796782 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa47e13c-f842-405b-a989-046436f77080-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-9632aa-77765f8fdb-8tgst\" (UID: \"aa47e13c-f842-405b-a989-046436f77080\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9632aa-77765f8fdb-8tgst" Oct 31 05:34:12.799183 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:34:12.799161 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa47e13c-f842-405b-a989-046436f77080-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-9632aa-77765f8fdb-8tgst\" (UID: \"aa47e13c-f842-405b-a989-046436f77080\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9632aa-77765f8fdb-8tgst" Oct 31 05:34:12.976133 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:34:12.976098 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9632aa-77765f8fdb-8tgst" Oct 31 05:34:13.278368 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:34:13.278324 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9632aa-77765f8fdb-8tgst"] Oct 31 05:34:13.278796 ip-10-0-134-155 kubenswrapper[4376]: W1031 05:34:13.278763 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa47e13c_f842_405b_a989_046436f77080.slice/crio-b9d9eb766a70eeca2eb51370afadd4593afbfae45b46c97d8b70f679981141d7 WatchSource:0}: Error finding container b9d9eb766a70eeca2eb51370afadd4593afbfae45b46c97d8b70f679981141d7: Status 404 returned error can't find the container with id b9d9eb766a70eeca2eb51370afadd4593afbfae45b46c97d8b70f679981141d7 Oct 31 05:34:13.826267 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:34:13.826228 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9632aa-77765f8fdb-8tgst" event={"ID":"aa47e13c-f842-405b-a989-046436f77080","Type":"ContainerStarted","Data":"97e72553120d64c908839c3f678e7f60dd4be4156edd0affe466ca72a957b39a"} Oct 31 05:34:13.826461 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:34:13.826297 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9632aa-77765f8fdb-8tgst" Oct 31 05:34:13.826461 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:34:13.826317 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9632aa-77765f8fdb-8tgst" event={"ID":"aa47e13c-f842-405b-a989-046436f77080","Type":"ContainerStarted","Data":"b9d9eb766a70eeca2eb51370afadd4593afbfae45b46c97d8b70f679981141d7"} Oct 31 05:34:13.878366 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:34:13.878282 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9632aa-77765f8fdb-8tgst" podStartSLOduration=1.485905848 podStartE2EDuration="1.878262106s" totalImagesPullingTime="392.356258ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 05:34:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 05:34:13.874837904 +0000 UTC m=+5401.589172521" watchObservedRunningTime="2026-10-31 05:34:13.878262106 +0000 UTC m=+5401.592596724" Oct 31 05:34:16.254152 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:34:16.254114 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 05:34:19.837262 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:34:19.837232 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9632aa-77765f8fdb-8tgst" Oct 31 05:34:31.883735 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:34:31.883695 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9632aa-77765f8fdb-8tgst"] Oct 31 05:34:31.884271 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:34:31.883947 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9632aa-77765f8fdb-8tgst" podUID="aa47e13c-f842-405b-a989-046436f77080" containerName="maas-api" containerID="cri-o://97e72553120d64c908839c3f678e7f60dd4be4156edd0affe466ca72a957b39a" gracePeriod=30 Oct 31 05:34:32.168312 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:34:32.168284 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9632aa-77765f8fdb-8tgst" Oct 31 05:34:32.351354 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:34:32.345469 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa47e13c-f842-405b-a989-046436f77080-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa47e13c-f842-405b-a989-046436f77080-openshift-service-ca\") pod \"aa47e13c-f842-405b-a989-046436f77080\" (UID: \"aa47e13c-f842-405b-a989-046436f77080\") " Oct 31 05:34:32.351354 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:34:32.345524 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa47e13c-f842-405b-a989-046436f77080-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa47e13c-f842-405b-a989-046436f77080-maas-api-tls\") pod \"aa47e13c-f842-405b-a989-046436f77080\" (UID: \"aa47e13c-f842-405b-a989-046436f77080\") " Oct 31 05:34:32.351354 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:34:32.345598 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa47e13c-f842-405b-a989-046436f77080-metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa47e13c-f842-405b-a989-046436f77080-metrics-certs\") pod \"aa47e13c-f842-405b-a989-046436f77080\" (UID: \"aa47e13c-f842-405b-a989-046436f77080\") " Oct 31 05:34:32.351354 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:34:32.345626 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa47e13c-f842-405b-a989-046436f77080-kube-api-access-79x99\" (UniqueName: \"kubernetes.io/projected/aa47e13c-f842-405b-a989-046436f77080-kube-api-access-79x99\") pod \"aa47e13c-f842-405b-a989-046436f77080\" (UID: \"aa47e13c-f842-405b-a989-046436f77080\") " Oct 31 05:34:32.355974 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:34:32.355929 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa47e13c-f842-405b-a989-046436f77080-openshift-service-ca" pod "aa47e13c-f842-405b-a989-046436f77080" (UID: "aa47e13c-f842-405b-a989-046436f77080"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:34:32.373520 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:34:32.373483 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa47e13c-f842-405b-a989-046436f77080-maas-api-tls" pod "aa47e13c-f842-405b-a989-046436f77080" (UID: "aa47e13c-f842-405b-a989-046436f77080"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 05:34:32.373701 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:34:32.373506 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa47e13c-f842-405b-a989-046436f77080-kube-api-access-79x99" pod "aa47e13c-f842-405b-a989-046436f77080" (UID: "aa47e13c-f842-405b-a989-046436f77080"). InnerVolumeSpecName "kube-api-access-79x99". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:34:32.373701 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:34:32.373507 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa47e13c-f842-405b-a989-046436f77080-metrics-certs" pod "aa47e13c-f842-405b-a989-046436f77080" (UID: "aa47e13c-f842-405b-a989-046436f77080"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 05:34:32.446703 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:34:32.446620 4376 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa47e13c-f842-405b-a989-046436f77080-metrics-certs\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:34:32.446703 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:34:32.446649 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79x99\" (UniqueName: \"kubernetes.io/projected/aa47e13c-f842-405b-a989-046436f77080-kube-api-access-79x99\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:34:32.446703 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:34:32.446663 4376 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa47e13c-f842-405b-a989-046436f77080-openshift-service-ca\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:34:32.446703 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:34:32.446675 4376 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa47e13c-f842-405b-a989-046436f77080-maas-api-tls\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:34:32.919678 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:34:32.919642 4376 generic.go:356] "Generic (PLEG): container finished" podID="aa47e13c-f842-405b-a989-046436f77080" containerID="97e72553120d64c908839c3f678e7f60dd4be4156edd0affe466ca72a957b39a" exitCode=0 Oct 31 05:34:32.920139 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:34:32.919692 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9632aa-77765f8fdb-8tgst" event={"ID":"aa47e13c-f842-405b-a989-046436f77080","Type":"ContainerDied","Data":"97e72553120d64c908839c3f678e7f60dd4be4156edd0affe466ca72a957b39a"} Oct 31 05:34:32.920139 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:34:32.919699 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9632aa-77765f8fdb-8tgst" Oct 31 05:34:32.920139 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:34:32.919732 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9632aa-77765f8fdb-8tgst" event={"ID":"aa47e13c-f842-405b-a989-046436f77080","Type":"ContainerDied","Data":"b9d9eb766a70eeca2eb51370afadd4593afbfae45b46c97d8b70f679981141d7"} Oct 31 05:34:32.920139 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:34:32.919753 4376 scope.go:122] "RemoveContainer" containerID="97e72553120d64c908839c3f678e7f60dd4be4156edd0affe466ca72a957b39a" Oct 31 05:34:32.928312 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:34:32.928288 4376 scope.go:122] "RemoveContainer" containerID="97e72553120d64c908839c3f678e7f60dd4be4156edd0affe466ca72a957b39a" Oct 31 05:34:32.928622 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:34:32.928601 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97e72553120d64c908839c3f678e7f60dd4be4156edd0affe466ca72a957b39a\": container with ID starting with 97e72553120d64c908839c3f678e7f60dd4be4156edd0affe466ca72a957b39a not found: ID does not exist" containerID="97e72553120d64c908839c3f678e7f60dd4be4156edd0affe466ca72a957b39a" Oct 31 05:34:32.928694 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:34:32.928631 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97e72553120d64c908839c3f678e7f60dd4be4156edd0affe466ca72a957b39a"} err="failed to get container status \"97e72553120d64c908839c3f678e7f60dd4be4156edd0affe466ca72a957b39a\": rpc error: code = NotFound desc = could not find container \"97e72553120d64c908839c3f678e7f60dd4be4156edd0affe466ca72a957b39a\": container with ID starting with 97e72553120d64c908839c3f678e7f60dd4be4156edd0affe466ca72a957b39a not found: ID does not exist" Oct 31 05:34:32.980074 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:34:32.980035 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9632aa-77765f8fdb-8tgst"] Oct 31 05:34:32.994500 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:34:32.994463 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9632aa-77765f8fdb-8tgst"] Oct 31 05:34:32.996996 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:34:32.996970 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-9632aa-openshift-default-6b55b9f5bfbtp9d"] Oct 31 05:34:32.997283 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:34:32.997233 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-9632aa-openshift-default-6b55b9f5bfbtp9d" podUID="2c8f2987-a903-4d11-a43e-2e41de5cb4bd" containerName="istio-proxy" containerID="cri-o://4932a3f3fc18724711eb17edf5d659ed55db8a66a2c8b9bc3ff482544ef6d61c" gracePeriod=30 Oct 31 05:34:34.441282 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:34:34.441250 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa47e13c-f842-405b-a989-046436f77080" path="/var/lib/kubelet/pods/aa47e13c-f842-405b-a989-046436f77080/volumes" Oct 31 05:34:35.137882 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:34:35.137843 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 05:34:38.243093 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:34:38.243070 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-9632aa-openshift-default-6b55b9f5bfbtp9d" Oct 31 05:34:38.404234 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:34:38.404198 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2c8f2987-a903-4d11-a43e-2e41de5cb4bd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c8f2987-a903-4d11-a43e-2e41de5cb4bd-istio-podinfo\") pod \"2c8f2987-a903-4d11-a43e-2e41de5cb4bd\" (UID: \"2c8f2987-a903-4d11-a43e-2e41de5cb4bd\") " Oct 31 05:34:38.404471 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:34:38.404251 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c8f2987-a903-4d11-a43e-2e41de5cb4bd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c8f2987-a903-4d11-a43e-2e41de5cb4bd-istio-envoy\") pod \"2c8f2987-a903-4d11-a43e-2e41de5cb4bd\" (UID: \"2c8f2987-a903-4d11-a43e-2e41de5cb4bd\") " Oct 31 05:34:38.404552 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:34:38.404468 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c8f2987-a903-4d11-a43e-2e41de5cb4bd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c8f2987-a903-4d11-a43e-2e41de5cb4bd-istiod-ca-cert\") pod \"2c8f2987-a903-4d11-a43e-2e41de5cb4bd\" (UID: \"2c8f2987-a903-4d11-a43e-2e41de5cb4bd\") " Oct 31 05:34:38.404552 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:34:38.404517 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c8f2987-a903-4d11-a43e-2e41de5cb4bd-istio-token\" (UniqueName: \"kubernetes.io/projected/2c8f2987-a903-4d11-a43e-2e41de5cb4bd-istio-token\") pod \"2c8f2987-a903-4d11-a43e-2e41de5cb4bd\" (UID: \"2c8f2987-a903-4d11-a43e-2e41de5cb4bd\") " Oct 31 05:34:38.404682 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:34:38.404649 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c8f2987-a903-4d11-a43e-2e41de5cb4bd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c8f2987-a903-4d11-a43e-2e41de5cb4bd-workload-socket\") pod \"2c8f2987-a903-4d11-a43e-2e41de5cb4bd\" (UID: \"2c8f2987-a903-4d11-a43e-2e41de5cb4bd\") " Oct 31 05:34:38.404792 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:34:38.404767 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c8f2987-a903-4d11-a43e-2e41de5cb4bd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c8f2987-a903-4d11-a43e-2e41de5cb4bd-credential-socket\") pod \"2c8f2987-a903-4d11-a43e-2e41de5cb4bd\" (UID: \"2c8f2987-a903-4d11-a43e-2e41de5cb4bd\") " Oct 31 05:34:38.404889 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:34:38.404804 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c8f2987-a903-4d11-a43e-2e41de5cb4bd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c8f2987-a903-4d11-a43e-2e41de5cb4bd-istio-data\") pod \"2c8f2987-a903-4d11-a43e-2e41de5cb4bd\" (UID: \"2c8f2987-a903-4d11-a43e-2e41de5cb4bd\") " Oct 31 05:34:38.404889 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:34:38.404837 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c8f2987-a903-4d11-a43e-2e41de5cb4bd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c8f2987-a903-4d11-a43e-2e41de5cb4bd-workload-certs\") pod \"2c8f2987-a903-4d11-a43e-2e41de5cb4bd\" (UID: \"2c8f2987-a903-4d11-a43e-2e41de5cb4bd\") " Oct 31 05:34:38.404889 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:34:38.404851 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c8f2987-a903-4d11-a43e-2e41de5cb4bd-workload-socket" pod "2c8f2987-a903-4d11-a43e-2e41de5cb4bd" (UID: "2c8f2987-a903-4d11-a43e-2e41de5cb4bd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:34:38.404889 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:34:38.404868 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c8f2987-a903-4d11-a43e-2e41de5cb4bd-istiod-ca-cert" pod "2c8f2987-a903-4d11-a43e-2e41de5cb4bd" (UID: "2c8f2987-a903-4d11-a43e-2e41de5cb4bd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:34:38.405107 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:34:38.404886 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c8f2987-a903-4d11-a43e-2e41de5cb4bd-kube-api-access-xqg84\" (UniqueName: \"kubernetes.io/projected/2c8f2987-a903-4d11-a43e-2e41de5cb4bd-kube-api-access-xqg84\") pod \"2c8f2987-a903-4d11-a43e-2e41de5cb4bd\" (UID: \"2c8f2987-a903-4d11-a43e-2e41de5cb4bd\") " Oct 31 05:34:38.405172 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:34:38.405108 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c8f2987-a903-4d11-a43e-2e41de5cb4bd-workload-certs" pod "2c8f2987-a903-4d11-a43e-2e41de5cb4bd" (UID: "2c8f2987-a903-4d11-a43e-2e41de5cb4bd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:34:38.405172 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:34:38.405144 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c8f2987-a903-4d11-a43e-2e41de5cb4bd-istio-data" pod "2c8f2987-a903-4d11-a43e-2e41de5cb4bd" (UID: "2c8f2987-a903-4d11-a43e-2e41de5cb4bd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:34:38.405310 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:34:38.405281 4376 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c8f2987-a903-4d11-a43e-2e41de5cb4bd-workload-socket\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:34:38.405310 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:34:38.405311 4376 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c8f2987-a903-4d11-a43e-2e41de5cb4bd-istio-data\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:34:38.405572 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:34:38.405343 4376 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c8f2987-a903-4d11-a43e-2e41de5cb4bd-workload-certs\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:34:38.405572 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:34:38.405035 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c8f2987-a903-4d11-a43e-2e41de5cb4bd-credential-socket" pod "2c8f2987-a903-4d11-a43e-2e41de5cb4bd" (UID: "2c8f2987-a903-4d11-a43e-2e41de5cb4bd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:34:38.405572 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:34:38.405356 4376 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c8f2987-a903-4d11-a43e-2e41de5cb4bd-istiod-ca-cert\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:34:38.406571 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:34:38.406549 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c8f2987-a903-4d11-a43e-2e41de5cb4bd-istio-envoy" pod "2c8f2987-a903-4d11-a43e-2e41de5cb4bd" (UID: "2c8f2987-a903-4d11-a43e-2e41de5cb4bd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:34:38.406738 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:34:38.406710 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c8f2987-a903-4d11-a43e-2e41de5cb4bd-istio-token" pod "2c8f2987-a903-4d11-a43e-2e41de5cb4bd" (UID: "2c8f2987-a903-4d11-a43e-2e41de5cb4bd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:34:38.406827 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:34:38.406779 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2c8f2987-a903-4d11-a43e-2e41de5cb4bd-istio-podinfo" pod "2c8f2987-a903-4d11-a43e-2e41de5cb4bd" (UID: "2c8f2987-a903-4d11-a43e-2e41de5cb4bd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 05:34:38.407076 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:34:38.407062 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c8f2987-a903-4d11-a43e-2e41de5cb4bd-kube-api-access-xqg84" pod "2c8f2987-a903-4d11-a43e-2e41de5cb4bd" (UID: "2c8f2987-a903-4d11-a43e-2e41de5cb4bd"). InnerVolumeSpecName "kube-api-access-xqg84". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:34:38.506500 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:34:38.506467 4376 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c8f2987-a903-4d11-a43e-2e41de5cb4bd-istio-token\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:34:38.506500 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:34:38.506495 4376 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c8f2987-a903-4d11-a43e-2e41de5cb4bd-credential-socket\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:34:38.506500 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:34:38.506505 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xqg84\" (UniqueName: \"kubernetes.io/projected/2c8f2987-a903-4d11-a43e-2e41de5cb4bd-kube-api-access-xqg84\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:34:38.506729 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:34:38.506514 4376 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c8f2987-a903-4d11-a43e-2e41de5cb4bd-istio-podinfo\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:34:38.506729 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:34:38.506524 4376 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c8f2987-a903-4d11-a43e-2e41de5cb4bd-istio-envoy\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:34:38.950514 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:34:38.950480 4376 generic.go:356] "Generic (PLEG): container finished" podID="2c8f2987-a903-4d11-a43e-2e41de5cb4bd" containerID="4932a3f3fc18724711eb17edf5d659ed55db8a66a2c8b9bc3ff482544ef6d61c" exitCode=0 Oct 31 05:34:38.950691 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:34:38.950528 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-9632aa-openshift-default-6b55b9f5bfbtp9d" Oct 31 05:34:38.950691 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:34:38.950530 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-9632aa-openshift-default-6b55b9f5bfbtp9d" event={"ID":"2c8f2987-a903-4d11-a43e-2e41de5cb4bd","Type":"ContainerDied","Data":"4932a3f3fc18724711eb17edf5d659ed55db8a66a2c8b9bc3ff482544ef6d61c"} Oct 31 05:34:38.950691 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:34:38.950661 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-9632aa-openshift-default-6b55b9f5bfbtp9d" event={"ID":"2c8f2987-a903-4d11-a43e-2e41de5cb4bd","Type":"ContainerDied","Data":"df01ae8af7ca8e33f9f3cd9c57856f510b7c5fdce56b72be029dc7207528d73b"} Oct 31 05:34:38.950792 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:34:38.950695 4376 scope.go:122] "RemoveContainer" containerID="4932a3f3fc18724711eb17edf5d659ed55db8a66a2c8b9bc3ff482544ef6d61c" Oct 31 05:34:38.959197 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:34:38.959177 4376 scope.go:122] "RemoveContainer" containerID="7df32700907c8677532c73b5fc67b972e2365b9f99821f2d41c1bff82c2303ac" Oct 31 05:34:38.967140 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:34:38.967114 4376 scope.go:122] "RemoveContainer" containerID="4932a3f3fc18724711eb17edf5d659ed55db8a66a2c8b9bc3ff482544ef6d61c" Oct 31 05:34:38.967446 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:34:38.967425 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4932a3f3fc18724711eb17edf5d659ed55db8a66a2c8b9bc3ff482544ef6d61c\": container with ID starting with 4932a3f3fc18724711eb17edf5d659ed55db8a66a2c8b9bc3ff482544ef6d61c not found: ID does not exist" containerID="4932a3f3fc18724711eb17edf5d659ed55db8a66a2c8b9bc3ff482544ef6d61c" Oct 31 05:34:38.967523 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:34:38.967454 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4932a3f3fc18724711eb17edf5d659ed55db8a66a2c8b9bc3ff482544ef6d61c"} err="failed to get container status \"4932a3f3fc18724711eb17edf5d659ed55db8a66a2c8b9bc3ff482544ef6d61c\": rpc error: code = NotFound desc = could not find container \"4932a3f3fc18724711eb17edf5d659ed55db8a66a2c8b9bc3ff482544ef6d61c\": container with ID starting with 4932a3f3fc18724711eb17edf5d659ed55db8a66a2c8b9bc3ff482544ef6d61c not found: ID does not exist" Oct 31 05:34:38.967523 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:34:38.967473 4376 scope.go:122] "RemoveContainer" containerID="7df32700907c8677532c73b5fc67b972e2365b9f99821f2d41c1bff82c2303ac" Oct 31 05:34:38.967717 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:34:38.967700 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7df32700907c8677532c73b5fc67b972e2365b9f99821f2d41c1bff82c2303ac\": container with ID starting with 7df32700907c8677532c73b5fc67b972e2365b9f99821f2d41c1bff82c2303ac not found: ID does not exist" containerID="7df32700907c8677532c73b5fc67b972e2365b9f99821f2d41c1bff82c2303ac" Oct 31 05:34:38.967763 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:34:38.967723 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7df32700907c8677532c73b5fc67b972e2365b9f99821f2d41c1bff82c2303ac"} err="failed to get container status \"7df32700907c8677532c73b5fc67b972e2365b9f99821f2d41c1bff82c2303ac\": rpc error: code = NotFound desc = could not find container \"7df32700907c8677532c73b5fc67b972e2365b9f99821f2d41c1bff82c2303ac\": container with ID starting with 7df32700907c8677532c73b5fc67b972e2365b9f99821f2d41c1bff82c2303ac not found: ID does not exist" Oct 31 05:34:39.024107 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:34:39.024076 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-9632aa-openshift-default-6b55b9f5bfbtp9d"] Oct 31 05:34:39.038493 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:34:39.038462 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-9632aa-openshift-default-6b55b9f5bfbtp9d"] Oct 31 05:34:40.441658 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:34:40.441627 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c8f2987-a903-4d11-a43e-2e41de5cb4bd" path="/var/lib/kubelet/pods/2c8f2987-a903-4d11-a43e-2e41de5cb4bd/volumes" Oct 31 05:34:40.998386 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:34:40.998350 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 05:35:10.998904 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:35:10.998824 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 05:35:40.999897 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:35:40.999850 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 05:36:10.999171 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:36:10.999125 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 05:36:14.138739 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:36:14.138707 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 05:36:29.924829 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:36:29.924739 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 05:36:35.304035 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:36:35.303990 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 05:36:39.458424 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:36:39.458390 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 05:36:40.999411 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:36:40.999376 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 05:36:48.145543 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:36:48.145508 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-76f7dfc58b-5x7gn"] Oct 31 05:36:48.146584 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:36:48.146556 4376 cpu_manager.go:414] "containerMap: removing container" podUID="aa47e13c-f842-405b-a989-046436f77080" containerName="maas-api" Oct 31 05:36:48.146584 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:36:48.146579 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa47e13c-f842-405b-a989-046436f77080" containerName="maas-api" Oct 31 05:36:48.146787 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:36:48.146616 4376 cpu_manager.go:414] "containerMap: removing container" podUID="2c8f2987-a903-4d11-a43e-2e41de5cb4bd" containerName="istio-proxy" Oct 31 05:36:48.146787 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:36:48.146623 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c8f2987-a903-4d11-a43e-2e41de5cb4bd" containerName="istio-proxy" Oct 31 05:36:48.146787 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:36:48.146635 4376 cpu_manager.go:414] "containerMap: removing container" podUID="2c8f2987-a903-4d11-a43e-2e41de5cb4bd" containerName="istio-proxy" Oct 31 05:36:48.146787 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:36:48.146642 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c8f2987-a903-4d11-a43e-2e41de5cb4bd" containerName="istio-proxy" Oct 31 05:36:48.146787 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:36:48.146730 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c8f2987-a903-4d11-a43e-2e41de5cb4bd" containerName="istio-proxy" Oct 31 05:36:48.146787 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:36:48.146743 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c8f2987-a903-4d11-a43e-2e41de5cb4bd" containerName="istio-proxy" Oct 31 05:36:48.146787 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:36:48.146756 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa47e13c-f842-405b-a989-046436f77080" containerName="maas-api" Oct 31 05:36:48.150041 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:36:48.150014 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-76f7dfc58b-5x7gn" Oct 31 05:36:48.165445 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:36:48.165407 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-76f7dfc58b-5x7gn"] Oct 31 05:36:48.245879 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:36:48.245841 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93ae7f76-53a9-4500-9274-12f984bc1d97-metrics-certs\") pod \"maas-api-76f7dfc58b-5x7gn\" (UID: \"93ae7f76-53a9-4500-9274-12f984bc1d97\") " pod="odh-ai-gateway-infra/maas-api-76f7dfc58b-5x7gn" Oct 31 05:36:48.246071 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:36:48.246004 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93ae7f76-53a9-4500-9274-12f984bc1d97-maas-api-tls\") pod \"maas-api-76f7dfc58b-5x7gn\" (UID: \"93ae7f76-53a9-4500-9274-12f984bc1d97\") " pod="odh-ai-gateway-infra/maas-api-76f7dfc58b-5x7gn" Oct 31 05:36:48.246127 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:36:48.246105 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93ae7f76-53a9-4500-9274-12f984bc1d97-openshift-service-ca\") pod \"maas-api-76f7dfc58b-5x7gn\" (UID: \"93ae7f76-53a9-4500-9274-12f984bc1d97\") " pod="odh-ai-gateway-infra/maas-api-76f7dfc58b-5x7gn" Oct 31 05:36:48.246222 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:36:48.246208 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkp72\" (UniqueName: \"kubernetes.io/projected/93ae7f76-53a9-4500-9274-12f984bc1d97-kube-api-access-kkp72\") pod \"maas-api-76f7dfc58b-5x7gn\" (UID: \"93ae7f76-53a9-4500-9274-12f984bc1d97\") " pod="odh-ai-gateway-infra/maas-api-76f7dfc58b-5x7gn" Oct 31 05:36:48.347233 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:36:48.347197 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93ae7f76-53a9-4500-9274-12f984bc1d97-metrics-certs\") pod \"maas-api-76f7dfc58b-5x7gn\" (UID: \"93ae7f76-53a9-4500-9274-12f984bc1d97\") " pod="odh-ai-gateway-infra/maas-api-76f7dfc58b-5x7gn" Oct 31 05:36:48.347440 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:36:48.347282 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93ae7f76-53a9-4500-9274-12f984bc1d97-maas-api-tls\") pod \"maas-api-76f7dfc58b-5x7gn\" (UID: \"93ae7f76-53a9-4500-9274-12f984bc1d97\") " pod="odh-ai-gateway-infra/maas-api-76f7dfc58b-5x7gn" Oct 31 05:36:48.347440 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:36:48.347350 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93ae7f76-53a9-4500-9274-12f984bc1d97-openshift-service-ca\") pod \"maas-api-76f7dfc58b-5x7gn\" (UID: \"93ae7f76-53a9-4500-9274-12f984bc1d97\") " pod="odh-ai-gateway-infra/maas-api-76f7dfc58b-5x7gn" Oct 31 05:36:48.347440 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:36:48.347421 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkp72\" (UniqueName: \"kubernetes.io/projected/93ae7f76-53a9-4500-9274-12f984bc1d97-kube-api-access-kkp72\") pod \"maas-api-76f7dfc58b-5x7gn\" (UID: \"93ae7f76-53a9-4500-9274-12f984bc1d97\") " pod="odh-ai-gateway-infra/maas-api-76f7dfc58b-5x7gn" Oct 31 05:36:48.348025 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:36:48.347996 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93ae7f76-53a9-4500-9274-12f984bc1d97-openshift-service-ca\") pod \"maas-api-76f7dfc58b-5x7gn\" (UID: \"93ae7f76-53a9-4500-9274-12f984bc1d97\") " pod="odh-ai-gateway-infra/maas-api-76f7dfc58b-5x7gn" Oct 31 05:36:48.349919 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:36:48.349890 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93ae7f76-53a9-4500-9274-12f984bc1d97-metrics-certs\") pod \"maas-api-76f7dfc58b-5x7gn\" (UID: \"93ae7f76-53a9-4500-9274-12f984bc1d97\") " pod="odh-ai-gateway-infra/maas-api-76f7dfc58b-5x7gn" Oct 31 05:36:48.350014 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:36:48.349890 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93ae7f76-53a9-4500-9274-12f984bc1d97-maas-api-tls\") pod \"maas-api-76f7dfc58b-5x7gn\" (UID: \"93ae7f76-53a9-4500-9274-12f984bc1d97\") " pod="odh-ai-gateway-infra/maas-api-76f7dfc58b-5x7gn" Oct 31 05:36:48.372599 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:36:48.372571 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkp72\" (UniqueName: \"kubernetes.io/projected/93ae7f76-53a9-4500-9274-12f984bc1d97-kube-api-access-kkp72\") pod \"maas-api-76f7dfc58b-5x7gn\" (UID: \"93ae7f76-53a9-4500-9274-12f984bc1d97\") " pod="odh-ai-gateway-infra/maas-api-76f7dfc58b-5x7gn" Oct 31 05:36:48.465469 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:36:48.465388 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-76f7dfc58b-5x7gn" Oct 31 05:36:48.764091 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:36:48.764060 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-76f7dfc58b-5x7gn"] Oct 31 05:36:48.764246 ip-10-0-134-155 kubenswrapper[4376]: W1031 05:36:48.764217 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93ae7f76_53a9_4500_9274_12f984bc1d97.slice/crio-81c030d96c0608747c217d6a18fc3a10a4e7eadb1246f9cc582a4081e5f446fd WatchSource:0}: Error finding container 81c030d96c0608747c217d6a18fc3a10a4e7eadb1246f9cc582a4081e5f446fd: Status 404 returned error can't find the container with id 81c030d96c0608747c217d6a18fc3a10a4e7eadb1246f9cc582a4081e5f446fd Oct 31 05:36:49.579599 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:36:49.579504 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-76f7dfc58b-5x7gn" event={"ID":"93ae7f76-53a9-4500-9274-12f984bc1d97","Type":"ContainerStarted","Data":"38d4e90104422646748f49bc5387ecadcbf0ecfe0ff7e8da14483434255a6e4c"} Oct 31 05:36:49.579599 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:36:49.579560 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-76f7dfc58b-5x7gn" event={"ID":"93ae7f76-53a9-4500-9274-12f984bc1d97","Type":"ContainerStarted","Data":"81c030d96c0608747c217d6a18fc3a10a4e7eadb1246f9cc582a4081e5f446fd"} Oct 31 05:36:49.580125 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:36:49.579597 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-76f7dfc58b-5x7gn" Oct 31 05:36:49.626584 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:36:49.626527 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-76f7dfc58b-5x7gn" podStartSLOduration=1.256485646 podStartE2EDuration="1.626513348s" totalImagesPullingTime="370.027702ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 05:36:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 05:36:49.625113514 +0000 UTC m=+5557.339448143" watchObservedRunningTime="2026-10-31 05:36:49.626513348 +0000 UTC m=+5557.340847964" Oct 31 05:36:55.589570 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:36:55.589540 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-76f7dfc58b-5x7gn" Oct 31 05:36:59.057202 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:36:59.057168 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 05:37:03.822055 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:37:03.822015 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-b94596b94-xgfr2"] Oct 31 05:37:03.822735 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:37:03.822651 4376 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-b94596b94-xgfr2" podUID="047e9b6c-64dc-4c2f-b1a9-fc21e50df6b5" containerName="maas-api" containerID="cri-o://cfa18122b28717dc9adec7c976e041e1bd14b72e444248031c5b4667381d6d91" gracePeriod=30 Oct 31 05:37:04.132275 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:37:04.132227 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-b94596b94-xgfr2" podUID="047e9b6c-64dc-4c2f-b1a9-fc21e50df6b5" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.27:8443/health\": dial tcp 10.133.0.27:8443: connect: connection refused" Oct 31 05:37:05.265222 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:37:05.264887 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-b94596b94-xgfr2" Oct 31 05:37:05.410673 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:37:05.410638 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/047e9b6c-64dc-4c2f-b1a9-fc21e50df6b5-kube-api-access-w5r9l\" (UniqueName: \"kubernetes.io/projected/047e9b6c-64dc-4c2f-b1a9-fc21e50df6b5-kube-api-access-w5r9l\") pod \"047e9b6c-64dc-4c2f-b1a9-fc21e50df6b5\" (UID: \"047e9b6c-64dc-4c2f-b1a9-fc21e50df6b5\") " Oct 31 05:37:05.410890 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:37:05.410685 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/047e9b6c-64dc-4c2f-b1a9-fc21e50df6b5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/047e9b6c-64dc-4c2f-b1a9-fc21e50df6b5-openshift-service-ca\") pod \"047e9b6c-64dc-4c2f-b1a9-fc21e50df6b5\" (UID: \"047e9b6c-64dc-4c2f-b1a9-fc21e50df6b5\") " Oct 31 05:37:05.410890 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:37:05.410719 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/047e9b6c-64dc-4c2f-b1a9-fc21e50df6b5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/047e9b6c-64dc-4c2f-b1a9-fc21e50df6b5-maas-api-tls\") pod \"047e9b6c-64dc-4c2f-b1a9-fc21e50df6b5\" (UID: \"047e9b6c-64dc-4c2f-b1a9-fc21e50df6b5\") " Oct 31 05:37:05.410890 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:37:05.410758 4376 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/047e9b6c-64dc-4c2f-b1a9-fc21e50df6b5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/047e9b6c-64dc-4c2f-b1a9-fc21e50df6b5-metrics-certs\") pod \"047e9b6c-64dc-4c2f-b1a9-fc21e50df6b5\" (UID: \"047e9b6c-64dc-4c2f-b1a9-fc21e50df6b5\") " Oct 31 05:37:05.411141 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:37:05.411112 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/047e9b6c-64dc-4c2f-b1a9-fc21e50df6b5-openshift-service-ca" pod "047e9b6c-64dc-4c2f-b1a9-fc21e50df6b5" (UID: "047e9b6c-64dc-4c2f-b1a9-fc21e50df6b5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:37:05.413014 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:37:05.412987 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/047e9b6c-64dc-4c2f-b1a9-fc21e50df6b5-kube-api-access-w5r9l" pod "047e9b6c-64dc-4c2f-b1a9-fc21e50df6b5" (UID: "047e9b6c-64dc-4c2f-b1a9-fc21e50df6b5"). InnerVolumeSpecName "kube-api-access-w5r9l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:37:05.413127 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:37:05.413012 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/047e9b6c-64dc-4c2f-b1a9-fc21e50df6b5-metrics-certs" pod "047e9b6c-64dc-4c2f-b1a9-fc21e50df6b5" (UID: "047e9b6c-64dc-4c2f-b1a9-fc21e50df6b5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 05:37:05.413127 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:37:05.413034 4376 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/047e9b6c-64dc-4c2f-b1a9-fc21e50df6b5-maas-api-tls" pod "047e9b6c-64dc-4c2f-b1a9-fc21e50df6b5" (UID: "047e9b6c-64dc-4c2f-b1a9-fc21e50df6b5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 05:37:05.512031 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:37:05.511931 4376 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5r9l\" (UniqueName: \"kubernetes.io/projected/047e9b6c-64dc-4c2f-b1a9-fc21e50df6b5-kube-api-access-w5r9l\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:37:05.512031 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:37:05.511966 4376 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/047e9b6c-64dc-4c2f-b1a9-fc21e50df6b5-openshift-service-ca\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:37:05.512031 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:37:05.511983 4376 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/047e9b6c-64dc-4c2f-b1a9-fc21e50df6b5-maas-api-tls\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:37:05.512031 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:37:05.511994 4376 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/047e9b6c-64dc-4c2f-b1a9-fc21e50df6b5-metrics-certs\") on node \"ip-10-0-134-155.ec2.internal\" DevicePath \"\"" Oct 31 05:37:05.688695 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:37:05.688657 4376 generic.go:356] "Generic (PLEG): container finished" podID="047e9b6c-64dc-4c2f-b1a9-fc21e50df6b5" containerID="cfa18122b28717dc9adec7c976e041e1bd14b72e444248031c5b4667381d6d91" exitCode=0 Oct 31 05:37:05.688887 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:37:05.688703 4376 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-b94596b94-xgfr2" Oct 31 05:37:05.688887 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:37:05.688742 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-b94596b94-xgfr2" event={"ID":"047e9b6c-64dc-4c2f-b1a9-fc21e50df6b5","Type":"ContainerDied","Data":"cfa18122b28717dc9adec7c976e041e1bd14b72e444248031c5b4667381d6d91"} Oct 31 05:37:05.688887 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:37:05.688802 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-b94596b94-xgfr2" event={"ID":"047e9b6c-64dc-4c2f-b1a9-fc21e50df6b5","Type":"ContainerDied","Data":"041698f2bc9d485a6e0d42b1dab40809fa6017858809a4d0cf6bef8b113013af"} Oct 31 05:37:05.688887 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:37:05.688826 4376 scope.go:122] "RemoveContainer" containerID="cfa18122b28717dc9adec7c976e041e1bd14b72e444248031c5b4667381d6d91" Oct 31 05:37:05.698060 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:37:05.698044 4376 scope.go:122] "RemoveContainer" containerID="cfa18122b28717dc9adec7c976e041e1bd14b72e444248031c5b4667381d6d91" Oct 31 05:37:05.698348 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:37:05.698313 4376 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cfa18122b28717dc9adec7c976e041e1bd14b72e444248031c5b4667381d6d91\": container with ID starting with cfa18122b28717dc9adec7c976e041e1bd14b72e444248031c5b4667381d6d91 not found: ID does not exist" containerID="cfa18122b28717dc9adec7c976e041e1bd14b72e444248031c5b4667381d6d91" Oct 31 05:37:05.698428 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:37:05.698352 4376 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfa18122b28717dc9adec7c976e041e1bd14b72e444248031c5b4667381d6d91"} err="failed to get container status \"cfa18122b28717dc9adec7c976e041e1bd14b72e444248031c5b4667381d6d91\": rpc error: code = NotFound desc = could not find container \"cfa18122b28717dc9adec7c976e041e1bd14b72e444248031c5b4667381d6d91\": container with ID starting with cfa18122b28717dc9adec7c976e041e1bd14b72e444248031c5b4667381d6d91 not found: ID does not exist" Oct 31 05:37:05.747551 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:37:05.747517 4376 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-b94596b94-xgfr2"] Oct 31 05:37:05.761678 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:37:05.761648 4376 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-b94596b94-xgfr2"] Oct 31 05:37:06.442456 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:37:06.442421 4376 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="047e9b6c-64dc-4c2f-b1a9-fc21e50df6b5" path="/var/lib/kubelet/pods/047e9b6c-64dc-4c2f-b1a9-fc21e50df6b5/volumes" Oct 31 05:37:10.999743 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:37:10.999700 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 05:37:28.551196 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:37:28.551157 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 05:37:33.613975 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:37:33.613938 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 05:37:40.998386 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:37:40.998342 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 05:37:50.545689 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:37:50.545651 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 05:37:54.930592 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:37:54.930506 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 05:38:02.064548 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:38:02.064511 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 05:38:10.998780 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:38:10.998743 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 05:38:21.468134 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:38:21.468090 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 05:38:31.926974 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:38:31.926941 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 05:38:36.452436 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:38:36.452402 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 05:38:41.006800 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:38:41.006755 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 05:38:45.439466 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:38:45.439422 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 05:39:03.913376 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:39:03.913321 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 05:39:10.998242 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:39:10.998190 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 05:39:14.146341 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:39:14.146307 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 05:39:19.526515 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:39:19.526480 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 05:39:29.942113 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:39:29.942025 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 05:39:40.998923 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:39:40.998883 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 05:39:47.349551 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:39:47.349516 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 05:40:10.714900 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:40:10.714852 4376 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-nv4mk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.211:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 05:40:10.715442 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:40:10.714922 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-nv4mk" podUID="3fd946a5-6c8c-4341-ae85-80f8cf13673f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.211:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 05:40:11.027912 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:40:11.027814 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 05:40:41.035138 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:40:41.035099 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 05:41:11.002052 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:41:11.001952 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 05:41:40.999417 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:41:40.999379 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 05:41:41.248073 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:41:41.248033 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54z7t"] Oct 31 05:41:43.400498 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:41:43.400449 4376 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-l4jlq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.240:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 05:41:43.400950 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:41:43.400533 4376 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-l4jlq" podUID="d3e2da41-2d03-4b11-8ef0-40e3848109d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.240:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 05:42:03.867586 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:42:03.867547 4376 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-mjffb_c3ae653d-3b01-495e-bc64-3466a1f9f14f/manager/0.log" Oct 31 05:42:04.376472 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:42:04.376440 4376 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-85c4f7957c-jtbfr_74c20194-9a91-40b7-b8b9-3d502c5d90be/manager/0.log" Oct 31 05:42:05.038398 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:42:05.038364 4376 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-zzvpx_c8c856e4-7a53-4077-8f5e-4faf46ec9d2a/proxy/0.log" Oct 31 05:42:06.015888 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:42:06.015859 4376 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-76f7dfc58b-5x7gn_93ae7f76-53a9-4500-9274-12f984bc1d97/maas-api/0.log" Oct 31 05:42:10.999320 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:42:10.999284 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 05:42:13.457908 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:42:13.457875 4376 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-54z7t_d9f625f9-0628-476d-8235-9afc0a939cc1/limitador/0.log" Oct 31 05:42:14.969527 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:42:14.969496 4376 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-7ltxr_df76adb1-33a3-4ddc-8192-0e1b3697626b/discovery/0.log" Oct 31 05:42:15.402446 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:42:15.402414 4376 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-58666cb9df-chk6f_f22948af-f85d-4360-afde-86b2a8c542d2/kube-auth-proxy/0.log" Oct 31 05:42:16.283709 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:42:16.283680 4376 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77bfb58f8c-l4jlq_d3e2da41-2d03-4b11-8ef0-40e3848109d3/istio-proxy/0.log" Oct 31 05:42:16.745021 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:42:16.744990 4376 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77bfb58f8c-nv4mk_3fd946a5-6c8c-4341-ae85-80f8cf13673f/istio-proxy/0.log" Oct 31 05:42:16.971780 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:42:16.971753 4376 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77bfb58f8c-ptb2g_f3f2eb40-6a45-42f1-9f44-71784631f635/istio-proxy/0.log" Oct 31 05:42:18.191837 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:42:18.191808 4376 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-6ddb9b8755-bwn8k_cf2ff14c-f327-4ea4-aa03-92a0926aba6b/payload-processing/0.log" Oct 31 05:42:19.474376 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:42:19.474351 4376 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-phl42_3d152b49-e42e-4584-a7ee-7b540322fd96/main/0.log" Oct 31 05:42:20.068706 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:42:20.068673 4376 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-gpzs9_91164053-d830-4b60-a4ef-ed2877b7c27c/main/0.log" Oct 31 05:42:20.466005 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:42:20.465974 4376 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-v7km4_839587e6-d175-4bfd-b531-0f1bc94b30b6/main/0.log" Oct 31 05:42:20.886491 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:42:20.886464 4376 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-d7npm_174171e7-100c-4b84-becd-0176672f1805/main/0.log" Oct 31 05:42:32.399433 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:42:32.399355 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-n76t8/must-gather-94wf8"] Oct 31 05:42:32.399810 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:42:32.399692 4376 cpu_manager.go:414] "containerMap: removing container" podUID="047e9b6c-64dc-4c2f-b1a9-fc21e50df6b5" containerName="maas-api" Oct 31 05:42:32.399810 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:42:32.399702 4376 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="047e9b6c-64dc-4c2f-b1a9-fc21e50df6b5" containerName="maas-api" Oct 31 05:42:32.399810 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:42:32.399779 4376 memory_manager.go:365] "RemoveStaleState removing state" podUID="047e9b6c-64dc-4c2f-b1a9-fc21e50df6b5" containerName="maas-api" Oct 31 05:42:32.402899 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:42:32.402875 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-n76t8/must-gather-94wf8" Oct 31 05:42:32.422047 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:42:32.422016 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-n76t8/must-gather-94wf8"] Oct 31 05:42:32.434225 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:42:32.434198 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-n76t8\"/\"kube-root-ca.crt\"" Oct 31 05:42:32.434396 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:42:32.434378 4376 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-n76t8\"/\"default-dockercfg-f2kxb\"" Oct 31 05:42:32.434451 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:42:32.434381 4376 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-n76t8\"/\"openshift-service-ca.crt\"" Oct 31 05:42:32.498278 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:42:32.498239 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/57a3389a-5023-4f1b-a2b9-bf1a824d1de1-must-gather-output\") pod \"must-gather-94wf8\" (UID: \"57a3389a-5023-4f1b-a2b9-bf1a824d1de1\") " pod="openshift-must-gather-n76t8/must-gather-94wf8" Oct 31 05:42:32.498480 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:42:32.498293 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grrg5\" (UniqueName: \"kubernetes.io/projected/57a3389a-5023-4f1b-a2b9-bf1a824d1de1-kube-api-access-grrg5\") pod \"must-gather-94wf8\" (UID: \"57a3389a-5023-4f1b-a2b9-bf1a824d1de1\") " pod="openshift-must-gather-n76t8/must-gather-94wf8" Oct 31 05:42:32.599094 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:42:32.599047 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/57a3389a-5023-4f1b-a2b9-bf1a824d1de1-must-gather-output\") pod \"must-gather-94wf8\" (UID: \"57a3389a-5023-4f1b-a2b9-bf1a824d1de1\") " pod="openshift-must-gather-n76t8/must-gather-94wf8" Oct 31 05:42:32.599094 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:42:32.599104 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grrg5\" (UniqueName: \"kubernetes.io/projected/57a3389a-5023-4f1b-a2b9-bf1a824d1de1-kube-api-access-grrg5\") pod \"must-gather-94wf8\" (UID: \"57a3389a-5023-4f1b-a2b9-bf1a824d1de1\") " pod="openshift-must-gather-n76t8/must-gather-94wf8" Oct 31 05:42:32.599511 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:42:32.599486 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/57a3389a-5023-4f1b-a2b9-bf1a824d1de1-must-gather-output\") pod \"must-gather-94wf8\" (UID: \"57a3389a-5023-4f1b-a2b9-bf1a824d1de1\") " pod="openshift-must-gather-n76t8/must-gather-94wf8" Oct 31 05:42:32.639806 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:42:32.639775 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-grrg5\" (UniqueName: \"kubernetes.io/projected/57a3389a-5023-4f1b-a2b9-bf1a824d1de1-kube-api-access-grrg5\") pod \"must-gather-94wf8\" (UID: \"57a3389a-5023-4f1b-a2b9-bf1a824d1de1\") " pod="openshift-must-gather-n76t8/must-gather-94wf8" Oct 31 05:42:32.716175 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:42:32.716073 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-n76t8/must-gather-94wf8" Oct 31 05:42:33.013099 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:42:33.013064 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-n76t8/must-gather-94wf8"] Oct 31 05:42:33.013283 ip-10-0-134-155 kubenswrapper[4376]: W1031 05:42:33.013255 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod57a3389a_5023_4f1b_a2b9_bf1a824d1de1.slice/crio-ceacef006a2fa925c2cbe8a71d734bc3af053b23ba8de44a971776f6106906da WatchSource:0}: Error finding container ceacef006a2fa925c2cbe8a71d734bc3af053b23ba8de44a971776f6106906da: Status 404 returned error can't find the container with id ceacef006a2fa925c2cbe8a71d734bc3af053b23ba8de44a971776f6106906da Oct 31 05:42:33.015300 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:42:33.015281 4376 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 05:42:33.355172 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:42:33.355079 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-n76t8/must-gather-94wf8" event={"ID":"57a3389a-5023-4f1b-a2b9-bf1a824d1de1","Type":"ContainerStarted","Data":"ceacef006a2fa925c2cbe8a71d734bc3af053b23ba8de44a971776f6106906da"} Oct 31 05:42:34.362249 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:42:34.362142 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-n76t8/must-gather-94wf8" event={"ID":"57a3389a-5023-4f1b-a2b9-bf1a824d1de1","Type":"ContainerStarted","Data":"6d32e6816b33ba5c8450c599b5a09cda2d5a2822c08cbae96082cd324eb8c729"} Oct 31 05:42:34.362249 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:42:34.362199 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-n76t8/must-gather-94wf8" event={"ID":"57a3389a-5023-4f1b-a2b9-bf1a824d1de1","Type":"ContainerStarted","Data":"429bcbe278bf0d1d945bd6f2275fb13e736573abf7a2024d472f5c1485be9bd6"} Oct 31 05:42:34.406060 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:42:34.406001 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-n76t8/must-gather-94wf8" podStartSLOduration=1.7176111920000001 podStartE2EDuration="2.405986199s" totalImagesPullingTime="688.375007ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 05:42:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 05:42:34.40531291 +0000 UTC m=+5902.119647527" watchObservedRunningTime="2026-10-31 05:42:34.405986199 +0000 UTC m=+5902.120320816" Oct 31 05:42:36.981112 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:42:36.981076 4376 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-tlcrp_bc9d2733-89fa-4079-9dc1-01983df773cb/global-pull-secret-syncer/0.log" Oct 31 05:42:37.851952 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:42:37.851865 4376 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-ntbm9_f12d4ae1-e673-4a9e-bd43-92d6e9d5a023/konnectivity-agent/0.log" Oct 31 05:42:38.087050 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:42:38.087021 4376 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-134-155.ec2.internal_db7fb1e6e2bd3665becb16d5fafc3d7e/haproxy/0.log" Oct 31 05:42:41.035131 ip-10-0-134-155 kubenswrapper[4376]: E1031 05:42:41.035077 4376 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fdtl5" containerName="extractor" Oct 31 05:42:41.804937 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:42:41.804905 4376 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-mjffb_c3ae653d-3b01-495e-bc64-3466a1f9f14f/manager/0.log" Oct 31 05:42:42.049596 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:42:42.049275 4376 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-mjffb_c3ae653d-3b01-495e-bc64-3466a1f9f14f/manager/0.log" Oct 31 05:42:43.603350 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:42:43.601004 4376 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-85c4f7957c-jtbfr_74c20194-9a91-40b7-b8b9-3d502c5d90be/manager/0.log" Oct 31 05:42:43.894346 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:42:43.891937 4376 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-85c4f7957c-jtbfr_74c20194-9a91-40b7-b8b9-3d502c5d90be/manager/0.log" Oct 31 05:42:44.752588 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:42:44.751882 4376 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-54z7t_d9f625f9-0628-476d-8235-9afc0a939cc1/limitador/0.log" Oct 31 05:42:46.751736 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:42:46.751697 4376 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-zzvpx_c8c856e4-7a53-4077-8f5e-4faf46ec9d2a/proxy/0.log" Oct 31 05:42:47.013352 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:42:47.013249 4376 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-zzvpx_c8c856e4-7a53-4077-8f5e-4faf46ec9d2a/proxy/0.log" Oct 31 05:42:47.408969 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:42:47.408934 4376 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-n76t8/perf-node-gather-daemonset-wmhlw"] Oct 31 05:42:47.414222 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:42:47.414188 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-n76t8/perf-node-gather-daemonset-wmhlw" Oct 31 05:42:47.433464 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:42:47.433423 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-n76t8/perf-node-gather-daemonset-wmhlw"] Oct 31 05:42:47.482045 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:42:47.482001 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrd6v\" (UniqueName: \"kubernetes.io/projected/70c517e8-950f-40a5-a5b8-6ea3febd98f5-kube-api-access-qrd6v\") pod \"perf-node-gather-daemonset-wmhlw\" (UID: \"70c517e8-950f-40a5-a5b8-6ea3febd98f5\") " pod="openshift-must-gather-n76t8/perf-node-gather-daemonset-wmhlw" Oct 31 05:42:47.482228 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:42:47.482058 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/70c517e8-950f-40a5-a5b8-6ea3febd98f5-podres\") pod \"perf-node-gather-daemonset-wmhlw\" (UID: \"70c517e8-950f-40a5-a5b8-6ea3febd98f5\") " pod="openshift-must-gather-n76t8/perf-node-gather-daemonset-wmhlw" Oct 31 05:42:47.482228 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:42:47.482112 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/70c517e8-950f-40a5-a5b8-6ea3febd98f5-proc\") pod \"perf-node-gather-daemonset-wmhlw\" (UID: \"70c517e8-950f-40a5-a5b8-6ea3febd98f5\") " pod="openshift-must-gather-n76t8/perf-node-gather-daemonset-wmhlw" Oct 31 05:42:47.482228 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:42:47.482153 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/70c517e8-950f-40a5-a5b8-6ea3febd98f5-sys\") pod \"perf-node-gather-daemonset-wmhlw\" (UID: \"70c517e8-950f-40a5-a5b8-6ea3febd98f5\") " pod="openshift-must-gather-n76t8/perf-node-gather-daemonset-wmhlw" Oct 31 05:42:47.482371 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:42:47.482258 4376 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/70c517e8-950f-40a5-a5b8-6ea3febd98f5-lib-modules\") pod \"perf-node-gather-daemonset-wmhlw\" (UID: \"70c517e8-950f-40a5-a5b8-6ea3febd98f5\") " pod="openshift-must-gather-n76t8/perf-node-gather-daemonset-wmhlw" Oct 31 05:42:47.582949 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:42:47.582916 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/70c517e8-950f-40a5-a5b8-6ea3febd98f5-sys\") pod \"perf-node-gather-daemonset-wmhlw\" (UID: \"70c517e8-950f-40a5-a5b8-6ea3febd98f5\") " pod="openshift-must-gather-n76t8/perf-node-gather-daemonset-wmhlw" Oct 31 05:42:47.583500 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:42:47.583470 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/70c517e8-950f-40a5-a5b8-6ea3febd98f5-lib-modules\") pod \"perf-node-gather-daemonset-wmhlw\" (UID: \"70c517e8-950f-40a5-a5b8-6ea3febd98f5\") " pod="openshift-must-gather-n76t8/perf-node-gather-daemonset-wmhlw" Oct 31 05:42:47.583730 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:42:47.583710 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrd6v\" (UniqueName: \"kubernetes.io/projected/70c517e8-950f-40a5-a5b8-6ea3febd98f5-kube-api-access-qrd6v\") pod \"perf-node-gather-daemonset-wmhlw\" (UID: \"70c517e8-950f-40a5-a5b8-6ea3febd98f5\") " pod="openshift-must-gather-n76t8/perf-node-gather-daemonset-wmhlw" Oct 31 05:42:47.583893 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:42:47.583869 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/70c517e8-950f-40a5-a5b8-6ea3febd98f5-podres\") pod \"perf-node-gather-daemonset-wmhlw\" (UID: \"70c517e8-950f-40a5-a5b8-6ea3febd98f5\") " pod="openshift-must-gather-n76t8/perf-node-gather-daemonset-wmhlw" Oct 31 05:42:47.583979 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:42:47.583274 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/70c517e8-950f-40a5-a5b8-6ea3febd98f5-sys\") pod \"perf-node-gather-daemonset-wmhlw\" (UID: \"70c517e8-950f-40a5-a5b8-6ea3febd98f5\") " pod="openshift-must-gather-n76t8/perf-node-gather-daemonset-wmhlw" Oct 31 05:42:47.583979 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:42:47.583725 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/70c517e8-950f-40a5-a5b8-6ea3febd98f5-lib-modules\") pod \"perf-node-gather-daemonset-wmhlw\" (UID: \"70c517e8-950f-40a5-a5b8-6ea3febd98f5\") " pod="openshift-must-gather-n76t8/perf-node-gather-daemonset-wmhlw" Oct 31 05:42:47.584085 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:42:47.584011 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/70c517e8-950f-40a5-a5b8-6ea3febd98f5-podres\") pod \"perf-node-gather-daemonset-wmhlw\" (UID: \"70c517e8-950f-40a5-a5b8-6ea3febd98f5\") " pod="openshift-must-gather-n76t8/perf-node-gather-daemonset-wmhlw" Oct 31 05:42:47.584085 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:42:47.584063 4376 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/70c517e8-950f-40a5-a5b8-6ea3febd98f5-proc\") pod \"perf-node-gather-daemonset-wmhlw\" (UID: \"70c517e8-950f-40a5-a5b8-6ea3febd98f5\") " pod="openshift-must-gather-n76t8/perf-node-gather-daemonset-wmhlw" Oct 31 05:42:47.584191 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:42:47.584133 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/70c517e8-950f-40a5-a5b8-6ea3febd98f5-proc\") pod \"perf-node-gather-daemonset-wmhlw\" (UID: \"70c517e8-950f-40a5-a5b8-6ea3febd98f5\") " pod="openshift-must-gather-n76t8/perf-node-gather-daemonset-wmhlw" Oct 31 05:42:47.608657 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:42:47.608623 4376 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrd6v\" (UniqueName: \"kubernetes.io/projected/70c517e8-950f-40a5-a5b8-6ea3febd98f5-kube-api-access-qrd6v\") pod \"perf-node-gather-daemonset-wmhlw\" (UID: \"70c517e8-950f-40a5-a5b8-6ea3febd98f5\") " pod="openshift-must-gather-n76t8/perf-node-gather-daemonset-wmhlw" Oct 31 05:42:47.730340 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:42:47.730250 4376 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-n76t8/perf-node-gather-daemonset-wmhlw" Oct 31 05:42:48.032153 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:42:48.032083 4376 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-n76t8/perf-node-gather-daemonset-wmhlw"] Oct 31 05:42:48.032841 ip-10-0-134-155 kubenswrapper[4376]: W1031 05:42:48.032805 4376 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod70c517e8_950f_40a5_a5b8_6ea3febd98f5.slice/crio-cde6caad4daa9b770fcfe00e64124dc6099346c9e4f770e61a29e633b0ea37a1 WatchSource:0}: Error finding container cde6caad4daa9b770fcfe00e64124dc6099346c9e4f770e61a29e633b0ea37a1: Status 404 returned error can't find the container with id cde6caad4daa9b770fcfe00e64124dc6099346c9e4f770e61a29e633b0ea37a1 Oct 31 05:42:48.059909 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:42:48.059861 4376 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-fg4ft_7b35c4cc-25db-4bf8-8b1b-37b8a45b46cf/node-exporter/0.log" Oct 31 05:42:48.180235 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:42:48.180201 4376 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-fg4ft_7b35c4cc-25db-4bf8-8b1b-37b8a45b46cf/kube-rbac-proxy/0.log" Oct 31 05:42:48.309424 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:42:48.304529 4376 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-fg4ft_7b35c4cc-25db-4bf8-8b1b-37b8a45b46cf/init-textfile/0.log" Oct 31 05:42:48.377278 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:42:48.376784 4376 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-mjffb_c3ae653d-3b01-495e-bc64-3466a1f9f14f/manager/0.log" Oct 31 05:42:48.500823 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:42:48.500787 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-n76t8/perf-node-gather-daemonset-wmhlw" event={"ID":"70c517e8-950f-40a5-a5b8-6ea3febd98f5","Type":"ContainerStarted","Data":"927b0b1de4efc6f005ed222ad38f7e1a84a9bd9fd6b11a8e3b6380313af803b6"} Oct 31 05:42:48.501058 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:42:48.500851 4376 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-n76t8/perf-node-gather-daemonset-wmhlw" event={"ID":"70c517e8-950f-40a5-a5b8-6ea3febd98f5","Type":"ContainerStarted","Data":"cde6caad4daa9b770fcfe00e64124dc6099346c9e4f770e61a29e633b0ea37a1"} Oct 31 05:42:48.501058 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:42:48.500991 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-n76t8/perf-node-gather-daemonset-wmhlw" Oct 31 05:42:48.546637 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:42:48.546581 4376 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-n76t8/perf-node-gather-daemonset-wmhlw" podStartSLOduration=1.546566213 podStartE2EDuration="1.546566213s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 05:42:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 05:42:48.545852237 +0000 UTC m=+5916.260186854" watchObservedRunningTime="2026-10-31 05:42:48.546566213 +0000 UTC m=+5916.260900829" Oct 31 05:42:48.742563 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:42:48.742520 4376 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-mjffb_c3ae653d-3b01-495e-bc64-3466a1f9f14f/manager/0.log" Oct 31 05:42:50.165857 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:42:50.165823 4376 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-85c4f7957c-jtbfr_74c20194-9a91-40b7-b8b9-3d502c5d90be/manager/0.log" Oct 31 05:42:50.388051 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:42:50.388019 4376 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-dn78l_55a51982-5071-4b52-9ce3-a6eadc5505cc/prometheus-operator/0.log" Oct 31 05:42:50.506352 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:42:50.506245 4376 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-dn78l_55a51982-5071-4b52-9ce3-a6eadc5505cc/kube-rbac-proxy/0.log" Oct 31 05:42:50.508260 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:42:50.508228 4376 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-85c4f7957c-jtbfr_74c20194-9a91-40b7-b8b9-3d502c5d90be/manager/0.log" Oct 31 05:42:50.630168 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:42:50.630126 4376 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-tj4g2_a6c57686-4e0f-43ed-8ff0-843f3d9b3310/prometheus-operator-admission-webhook/0.log" Oct 31 05:42:53.476989 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:42:53.476954 4376 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-zzvpx_c8c856e4-7a53-4077-8f5e-4faf46ec9d2a/proxy/0.log" Oct 31 05:42:53.804147 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:42:53.804039 4376 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-zzvpx_c8c856e4-7a53-4077-8f5e-4faf46ec9d2a/proxy/0.log" Oct 31 05:42:54.519726 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:42:54.519696 4376 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-n76t8/perf-node-gather-daemonset-wmhlw" Oct 31 05:42:55.126206 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:42:55.126175 4376 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-7b87c4b57-nxzpf_92740329-2888-4011-b817-1c0d74fd1c21/console/0.log" Oct 31 05:42:55.158228 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:42:55.158192 4376 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-mjffb_c3ae653d-3b01-495e-bc64-3466a1f9f14f/manager/0.log" Oct 31 05:42:55.265981 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:42:55.265947 4376 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6dd789c8b7-74d6m_43d6c532-a6b9-4f34-8d70-8bab1a8c3fd9/download-server/0.log" Oct 31 05:42:55.506502 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:42:55.506423 4376 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-mjffb_c3ae653d-3b01-495e-bc64-3466a1f9f14f/manager/0.log" Oct 31 05:42:57.275972 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:42:57.275935 4376 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-85c4f7957c-jtbfr_74c20194-9a91-40b7-b8b9-3d502c5d90be/manager/0.log" Oct 31 05:42:57.510705 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:42:57.510668 4376 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-85c4f7957c-jtbfr_74c20194-9a91-40b7-b8b9-3d502c5d90be/manager/0.log" Oct 31 05:42:58.812226 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:42:58.812191 4376 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-rv7hd_3ae04333-9598-482c-ae38-8de983d7a48f/dns/0.log" Oct 31 05:42:58.929686 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:42:58.929652 4376 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-rv7hd_3ae04333-9598-482c-ae38-8de983d7a48f/kube-rbac-proxy/0.log" Oct 31 05:42:59.534593 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:42:59.534560 4376 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-77z7p_1f2543cc-b528-41be-92e0-9d654405a723/dns-node-resolver/0.log" Oct 31 05:43:00.695292 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:43:00.695251 4376 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-zzvpx_c8c856e4-7a53-4077-8f5e-4faf46ec9d2a/proxy/0.log" Oct 31 05:43:00.873675 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:43:00.873641 4376 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-zzvpx_c8c856e4-7a53-4077-8f5e-4faf46ec9d2a/proxy/0.log" Oct 31 05:43:01.049270 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:43:01.049194 4376 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-799cd67bd8-fwz76_3e67fa28-1a34-48bd-be16-373044df6f41/registry/0.log" Oct 31 05:43:01.271427 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:43:01.271390 4376 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-xmbxc_a90157bb-46c0-4afa-82b1-9f6819518744/node-ca/0.log" Oct 31 05:43:01.461182 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:43:01.461133 4376 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-7ltxr_df76adb1-33a3-4ddc-8192-0e1b3697626b/discovery/0.log" Oct 31 05:43:01.561231 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:43:01.561198 4376 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-58666cb9df-chk6f_f22948af-f85d-4360-afde-86b2a8c542d2/kube-auth-proxy/0.log" Oct 31 05:43:01.829633 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:43:01.829541 4376 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77bfb58f8c-l4jlq_d3e2da41-2d03-4b11-8ef0-40e3848109d3/istio-proxy/0.log" Oct 31 05:43:01.959715 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:43:01.959675 4376 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77bfb58f8c-nv4mk_3fd946a5-6c8c-4341-ae85-80f8cf13673f/istio-proxy/0.log" Oct 31 05:43:02.028172 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:43:02.028146 4376 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77bfb58f8c-ptb2g_f3f2eb40-6a45-42f1-9f44-71784631f635/istio-proxy/0.log" Oct 31 05:43:02.422893 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:43:02.422862 4376 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-6ddb9b8755-bwn8k_cf2ff14c-f327-4ea4-aa03-92a0926aba6b/payload-processing/0.log" Oct 31 05:43:04.096757 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:43:04.096708 4376 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-7ltxr_df76adb1-33a3-4ddc-8192-0e1b3697626b/discovery/0.log" Oct 31 05:43:04.199117 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:43:04.199076 4376 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-58666cb9df-chk6f_f22948af-f85d-4360-afde-86b2a8c542d2/kube-auth-proxy/0.log" Oct 31 05:43:04.230003 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:43:04.229972 4376 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-7ltxr_df76adb1-33a3-4ddc-8192-0e1b3697626b/discovery/0.log" Oct 31 05:43:04.327963 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:43:04.327909 4376 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-58666cb9df-chk6f_f22948af-f85d-4360-afde-86b2a8c542d2/kube-auth-proxy/0.log" Oct 31 05:43:04.465249 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:43:04.465215 4376 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77bfb58f8c-l4jlq_d3e2da41-2d03-4b11-8ef0-40e3848109d3/istio-proxy/0.log" Oct 31 05:43:04.596068 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:43:04.596039 4376 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77bfb58f8c-l4jlq_d3e2da41-2d03-4b11-8ef0-40e3848109d3/istio-proxy/0.log" Oct 31 05:43:04.604085 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:43:04.604051 4376 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77bfb58f8c-nv4mk_3fd946a5-6c8c-4341-ae85-80f8cf13673f/istio-proxy/0.log" Oct 31 05:43:04.680947 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:43:04.680912 4376 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77bfb58f8c-ptb2g_f3f2eb40-6a45-42f1-9f44-71784631f635/istio-proxy/0.log" Oct 31 05:43:04.723967 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:43:04.723889 4376 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77bfb58f8c-nv4mk_3fd946a5-6c8c-4341-ae85-80f8cf13673f/istio-proxy/0.log" Oct 31 05:43:04.789516 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:43:04.789475 4376 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77bfb58f8c-ptb2g_f3f2eb40-6a45-42f1-9f44-71784631f635/istio-proxy/0.log" Oct 31 05:43:04.814532 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:43:04.814502 4376 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-7ltxr_df76adb1-33a3-4ddc-8192-0e1b3697626b/discovery/0.log" Oct 31 05:43:05.081283 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:43:05.081200 4376 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-6ddb9b8755-bwn8k_cf2ff14c-f327-4ea4-aa03-92a0926aba6b/payload-processing/0.log" Oct 31 05:43:05.239853 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:43:05.239821 4376 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-6ddb9b8755-bwn8k_cf2ff14c-f327-4ea4-aa03-92a0926aba6b/payload-processing/0.log" Oct 31 05:43:05.474691 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:43:05.474646 4376 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-58666cb9df-chk6f_f22948af-f85d-4360-afde-86b2a8c542d2/kube-auth-proxy/0.log" Oct 31 05:43:05.975972 ip-10-0-134-155 kubenswrapper[4376]: I1031 05:43:05.975935 4376 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77bfb58f8c-l4jlq_d3e2da41-2d03-4b11-8ef0-40e3848109d3/istio-proxy/0.log"